AWS Compute Blog

Unlocking full-stack observability for AWS Outposts

Aishwariya Khiani — Thu, 29 Oct 2026 18:23:18 +0000

Your team just deployed AWS Outposts to run latency-sensitive workloads on-premises, and now you need to monitor infrastructure that spans both your data center and an AWS Region. Some services run locally on the Outposts, and others depend on Regional endpoints. A network blip between the two can cause silent failures that traditional monitoring tools miss entirely.

AWS Outposts extends native AWS services to on-premises environments, but this hybrid topology introduces monitoring challenges that don’t exist in pure-cloud or pure-on-premises setups. Control planes are distributed, critical operations depend on AWS Region connectivity, and fixed, finite capacity requires proactive capacity planning.

In this post, you will learn how to build end-to-end observability for Outposts using AWS Systems Manager, Amazon CloudWatch, and Amazon EventBridge, covering four monitoring dimensions:

Network – Connectivity status, BGP sessions, and Regional service dependency tracking.

For dependency tracking, watch each Regional service’s own CloudWatch metrics together with the service-link ConnectedStatus: that pairing tells a dependency failure apart from a connectivity drop. For example, AWS Directory Service (DirectoryReachability, AuthenticationFailures), Amazon Relational Database Service (Amazon RDS) (ReplicaLag, DatabaseConnections), and so on.

Storage – Amazon Simple Storage Service (Amazon S3) on Outposts capacity, Amazon Elastic Block Store (Amazon EBS) volume utilization, and replication health.
Compute – Instance-level metrics, micro-architectural performance, and enhanced networking.
Containers – Amazon Elastic Kubernetes Service (Amazon EKS) on Outposts with Prometheus-based Kubernetes monitoring.

By the end, you will have a monitoring architecture that gives you unified visibility across your hybrid environment and catches issues before your users do.

Building a monitoring foundation with AWS Systems Manager

The following sections detail how to configure each component of the Systems Manager foundation for your Outposts environment.

Centralized management architecture

Systems Manager provides the foundation for Outposts monitoring with centralized management across hybrid environments. Configure it to span your cloud and on-premises infrastructure so you get unified visibility, automated operations, and consistent policy enforcement across your distributed AWS resources.

SSM Agent deployment

The Systems Manager agent (SSM Agent) is the basis for monitoring. Use the following commands to install the SSM Agent on your Outpost servers.

# OS-agnostic SSM Agent installation via user data
# For Amazon Linux/RHEL/CentOS
sudo yum update -y
sudo yum install -y amazon-ssm-agent
sudo systemctl enable amazon-ssm-agent
sudo systemctl start amazon-ssm-agent

# For Ubuntu/Debian
sudo apt-get update -y
sudo apt-get install -y amazon-ssm-agent
sudo systemctl enable amazon-ssm-agent
sudo systemctl start amazon-ssm-agent

# For Windows (PowerShell)
Invoke-WebRequest -Uri "https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/windows_amd64/AmazonSSMAgentSetup.exe" -OutFile "AmazonSSMAgentSetup.exe"
Start-Process -FilePath "AmazonSSMAgentSetup.exe" -ArgumentList "/quiet" -Wait

OpsCenter integration

Systems Manager OpsCenter provides incident management capabilities that extend beyond basic alerting:

Automated OpsItem creation: Configure Amazon EventBridge to automatically create operational work items when specific conditions are met.
Runbook integration: Link automated remediation runbooks directly to OpsItems for immediate response.
Cross-service correlation: Aggregate related issues across multiple AWS services into single operational views.
Historical analysis: Use OpsCenter’s data retention for trend analysis and continuous improvement.

Patch management strategy

Implement a patch management approach that addresses the unique requirements of Outposts:

{
  "PatchGroups": [
    {
      "Name": "OutpostsCriticalSystems",
      "MaintenanceWindow": "Sunday 02:00-04:00 UTC",
      "PatchBaseline": "Custom-Security-Critical"
    },
    {
      "Name": "OutpostsApplicationServers",
      "MaintenanceWindow": "Sunday 04:00-06:00 UTC",
      "PatchBaseline": "Custom-Application-Stable"
    }
  ]
}

Systems Manager Patch Manager requires the SSM Agent to communicate with Regional endpoints through the service link. If the service link is degraded, patching operations will fail.

Personal Health Dashboard integration

The integration between AWS Personal Health Dashboard (PHD) and OpsCenter creates a proactive monitoring workflow. When an Outposts health event occurs, this rule automatically creates an OpsItem in Systems Manager, giving your operations team a single view for triage.

Automated event processing

Configure Amazon EventBridge rules to automatically process PHD events:

{
  "Rules": [
    {
      "Name": "OutpostsPHDIntegration",
      "EventPattern": {
        "source": ["aws.health"],
        "detail-type": ["AWS Health Event"],
        "detail": {
          "service": ["OUTPOSTS"],
          "eventTypeCategory": ["issue", "scheduledChange"]
        }
      },
      "Targets": [
        {
          "Id": "CreateOpsItem",
          "Arn": "arn:aws:ssm:region:account:opsitem",
          "RoleArn": "arn:aws:iam::account:role/EventBridge-OpsCenter-Role"
        }
      ]
    }
  ]
}

Monitoring dimensions

This section covers the four key monitoring dimensions for Outposts, with specific metrics, configurations, and alerting strategies for each.

1. Network monitoring and notification

Network monitoring for Outposts requires approaches that account for the hybrid nature of the deployment:

Multi-layer connectivity monitoring

Implement monitoring across multiple network layers:

Physical layer: Monitor fiber-optic connections, network equipment health, and environmental factors.
- ConnectedStatus.

Figure 1: ConnectedStatus metric for the Outposts service link

Network layer: Track routing, Border Gateway Protocol (BGP) sessions, and inter-rack connectivity.
- VIF metrics.
  - VifBgpSessionState.

Figure 2: VifBgpSessionState metric for the Outposts virtual interface

VifConnectionStatus.

Figure 3: VifConnectionStatus metric for the Outposts virtual interface

IfTrafficIn.
IfTrafficOut.

Figure 4: IfTrafficIn and IfTrafficOut metrics for Outposts network traffic

CapacityExceptions.

Figure 5: CapacityExceptions metric for the Outposts link

LagStatus:.

Figure 6: LagStatus metric for the Outposts link aggregation group

Regional service dependency mapping

Create detailed dependency maps for critical Regional services:

Amazon Elastic File System (Amazon EFS): Monitor mount target availability and performance across Availability Zones.
AWS Directory Service: Track authentication latency and success rates.
Amazon RDS: Monitor cross-Region replication lag and connection pooling.
AWS Lambda: Track cold start times and execution duration for Region-dependent functions.

2. Storage monitoring and optimization

Outposts provides CloudWatch metrics for both Amazon S3 and Amazon EBS storage resources. Monitor these metrics to track capacity utilization and plan for scaling.

S3 metrics:

The S3Outposts namespace includes the following metrics for Amazon S3 on Outposts buckets. You can monitor the total number of S3 on Outposts bytes provisioned, the total available bytes for objects, and the total size of all objects for a given bucket. Bucket or account-related metrics exist for all direct S3 usage.

OutpostTotalBytes: The total provisioned capacity in bytes for an Outpost.

Figure 7: OutpostTotalBytes metric for S3 on Outposts provisioned capacity

OutpostFreeBytes: The count of available bytes on an Outpost to store customer data. BucketUsedBytes: The total size of all objects for the given bucket.

Figure 8: OutpostFreeBytes metric for available S3 on Outposts capacity

Other key metrics to monitor include the following:

AccountUsedBytes: The total size of all objects for the specified Outposts account.
BytesPendingReplication: The total number of bytes of objects that are pending replication for a given replication rule. For more information about how to enable replication metrics, see Creating replication rules between Outposts.
OperationsPendingReplication: The total number of operations that are pending replication for a given replication rule.

Amazon EBS performance deep dive

The following Amazon EBS metrics are available for Outposts volumes:

EBSVolumeTypeCapacityUtilization: The percentage of Amazon EBS volume type capacity in use.

Figure 9: EBSVolumeTypeCapacityUtilization metric for Outposts Amazon EBS volumes

EBSVolumeTypeCapacityAvailability: The percentage of Amazon EBS volume type capacity available.

Figure 10: EBSVolumeTypeCapacityAvailability metric for Outposts Amazon EBS volumes

EBSVolumeTypeCapacityUtilizationGB: The number of gigabytes in use for the Amazon EBS volume type.

Figure 11: EBSVolumeTypeCapacityUtilizationGB metric for Outposts Amazon EBS volumes

EBSVolumeTypeCapacityAvailabilityGB: The number of gigabytes of available capacity for the Amazon EBS volume type.

Figure 12: EBSVolumeTypeCapacityAvailabilityGB metric for Outposts Amazon EBS volumes

Note: Second-generation AWS Outposts racks support both gp2 and gp3 Amazon EBS volume types, whereas first-generation racks support gp2 only.

3. Compute and capacity monitoring

Outposts have a fixed capacity that doesn’t auto scale, so compute and capacity metrics are essential for planning. The following instance and capacity metrics are published to the AWS/Outposts namespace.

Instance metrics.

UsedInstanceType_Count: The number of instance types that are currently in use, including any instance types used by managed services such as Amazon RDS or Application Load Balancer. This metric doesn’t include capacity for any Dedicated Hosts configured on the Outpost.

Figure 13: UsedInstanceType_Count metric for Outposts instance usage

InstanceTypeCapacityUtilization: The percentage of instance capacity in use. This metric does not include capacity for any Dedicated Hosts configured on the Outpost.

Figure 14: InstanceTypeCapacityUtilization metric for Outposts instances

InstanceTypeCapacityAvailability: The percentage of instance capacity available. This metric doesn’t include capacity for any Dedicated Hosts configured on the Outpost.

Figure 15: InstanceTypeCapacityAvailability metric for Outposts instances

InstanceFamilyCapacityAvailability: The percentage of instance capacity available. This metric does not include capacity for any Dedicated Hosts configured on the Outpost.

Figure 16: InstanceFamilyCapacityAvailability metric for Outposts instances

InstanceFamilyCapacityUtilization: The percentage of instance capacity in use. This metric doesn’t include capacity for any Dedicated Hosts configured on the Outpost.

Figure 17: InstanceFamilyCapacityUtilization metric for Outposts instances

Monitoring EKS containers on AWS Outposts with Amazon CloudWatch

AWS Outposts extends AWS infrastructure to your on-premises environment. When running containerized workloads using Amazon EKS (extended clusters) on Outposts, monitoring is critical because unlike the Region, Outposts has fixed capacity that doesn’t auto-scale. Without proper monitoring, pods can fail to schedule, workloads can get OOMKilled, and disk pressure can cause evictions. All of this happens without warning.

This section walks through setting up CloudWatch Container Insights for EKS on Outposts, the key metrics to track, and how to build a unified monitoring dashboard.

Prerequisites

An AWS Outposts rack deployed and connected to your parent Region.
An EKS cluster with worker nodes running on the Outpost.
kubectl configured to access your cluster.
AWS Command Line Interface (AWS CLI) installed and configured.

Step 1: Launch self-managed worker nodes with Amazon Linux 2023 (AL2023) and nodeadm

Outpost subnets don’t support EKS managed node groups, so worker nodes come up as a self-managed node group launched from an EC2 launch template.

The AMI must be AL2023. Amazon EKS stopped publishing AL2 AMIs on November 26, 2025, and those images no longer receive security patches or bug fixes, as noted in the Amazon EKS AMI deprecation FAQs.

AL2023 also changes how a node initializes. It uses nodeadm rather than the AL2 bootstrap.sh script, so the user data becomes a YAML NodeConfig document instead of a shell script. AL2023 expects the cluster endpoint, certificate authority, and service CIDR to be passed in explicitly where AL2 discovered them at boot. Missing them is a common reason nodes launch but don’t join the cluster.

aws eks describe-cluster returns all three:

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  cluster:
    name: <cluster-name>
    apiServerEndpoint: <from: aws eks describe-cluster>
    certificateAuthority: <from: aws eks describe-cluster>
    cidr: <service CIDR from: aws eks describe-cluster>

Step 2: Install the CloudWatch Observability add-on

On an extended cluster, use the CloudWatch Observability EKS add-on to enable Container Insights. The add-on deploys the CloudWatch agent and Fluent Bit as DaemonSets on your Outpost worker nodes, and it collects container, pod, node, and cluster-level metrics automatically.

aws eks create-addon \
  --cluster-name your-cluster-name \
  --addon-name amazon-cloudwatch-observability

You can also do this from the EKS Console under your cluster’s Add-ons tab.

Step 3: Verify the agent is running

Confirm that the CloudWatch agent pods are running on each node:

kubectl get pods -n amazon-cloudwatch

Expected output:

NAME                                  READY   STATUS    RESTARTS   AGE
cloudwatch-agent-xxxxx                1/1     Running   0          2m
cloudwatch-agent-yyyyy                1/1     Running   0          2m
fluent-bit-zzzzz                      1/1     Running   0          2m

You should see one cloudwatch-agent pod per worker node.

Step 4: Verify metrics in CloudWatch

After 5–10 minutes, metrics start flowing into CloudWatch.

Open the CloudWatch Console.
Navigate to Metrics → All metrics.
Look for the ContainerInsights namespace.
You will see metrics grouped by Cluster, Node, Pod, Service, and Namespace.

Key metrics for EKS containers on Outposts

Here are the metrics that matter most, organized by level.

Cluster level

	Metric	Description	Unit
1	`cluster_node_count`	Total worker nodes in the cluster	Count
2	`cluster_failed_node_count`	Nodes in a failed state	Count

Example: cluster_node_count

Figure 18: cluster_node_count metric in CloudWatch Container Insights

Node level

	Metric	Description	Unit
1	`node_cpu_utilization`	% CPU used on each node	Percent
2	`node_cpu_reserved_capacity`	% CPU reserved for system components (kubelet, kube-proxy)	Percent
3	`node_memory_utilization`	% memory used on each node	Percent
5	`node_filesystem_utilization`	% disk used on each node	Percent
6	`node_network_total_bytes`	Network throughput (rx + tx)	Bytes/sec
7	`node_number_of_running_pods`	Pod count per node	Count
8	`node_number_of_running_containers`	Container count per node	Count

Example:

Figure 19: Node-level Container Insights metrics for EKS worker nodes

Pod level

	Metric	Description	Unit
1	`pod_cpu_utilization`	% CPU used by pods	Percent
2	`pod_cpu_utilization_over_pod_limit`	CPU usage relative to pod limit	Percent
3	`pod_memory_utilization`	% memory used by pods	Percent
4	`pod_memory_utilization_over_pod_limit`	Memory usage relative to pod limit	Percent
5	`pod_network_rx_bytes`	Bytes received per second	Bytes/sec
6	`pod_network_tx_bytes`	Bytes transmitted per second	Bytes/sec
7	`pod_number_of_container_restarts`	Container restart count (stability indicator)	Count

Example:

Figure 20: Pod-level Container Insights metrics for EKS workloads

Service level

	Metric	Description	Unit
1	`service_number_of_running_pods`	Running pods per service	Count
2	`namespace_number_of_running_pods`	Running pods per namespace	Count

Figure 21: Service-level Container Insights metrics for EKS on Outposts

Step 5: Build a unified dashboard

Bring everything together in a single CloudWatch dashboard.

Go to CloudWatch Console → Dashboards → Create dashboard.
Name it Outposts-EKS-Monitoring
Add widgets in this order:

Row 1 — Outposts infrastructure

InstanceFamilyCapacityUtilization (line chart)
EBSVolumeTypeCapacityUtilization (line chart)
UsedInstanceType_Count (number widget)

Row 2 — Node health

node_cpu_utilization (line chart)
node_memory_utilization (line chart)
node_filesystem_utilization (line chart)

Row 3 — Pod/container workloads

pod_cpu_utilization (line chart)
pod_memory_utilization (line chart)
pod_number_of_container_restarts (number widget)

Why this matters on Outposts

In the Region, if your containers need more resources, the infrastructure scales automatically. On Outposts, you have a fixed pool of compute, memory, and storage.

Without monitoring:

Pods fail to schedule when node capacity is exhausted.
Workloads get OOMKilled without warning.
Disk pressure causes pod evictions.
You have no visibility into how close you are to capacity limits.

By combining Container Insights metrics with Outposts infrastructure metrics in a single dashboard, you get a complete picture, from application-level container health down to physical infrastructure utilization. With it, you can plan capacity proactively before issues affect your workloads.

Useful references

Conclusion

Monitoring AWS Outposts requires visibility across four dimensions that don’t exist in pure-cloud deployments: Region connectivity health, local storage capacity, compute performance on fixed hardware, and container orchestration across hybrid boundaries. The gap between standard cloud monitoring and what Outposts demands is where incidents hide.

In this post, we walked through building that visibility layer using Systems Manager as the management foundation, Amazon CloudWatch for metrics and alarms, and Amazon EventBridge for automated incident response through OpsCenter. Together, these services give you a unified monitoring architecture that spans your data center and the parent Region.

Next steps

To get started with the monitoring approach described in this post:

Deploy the Systems Manager agent to all Outpost instances and verify Region connectivity through the Fleet Manager console.
Set up CloudWatch alarms for the critical Outpost metrics. Start with ConnectedStatus, OutpostFreeBytes, and CapacityExceptions before expanding to application-level metrics.
Configure the Amazon EventBridge-to-OpsCenter integration shown in this post to automate incident creation for Outposts health events.
Build a CloudWatch dashboard that combines network, storage, and compute metrics into a single operational view for your Outpost.

For more information, see the following resources:

About the authors

Observing AWS Lambda durable functions with the Durable Execution Telemetry Exporter

Pooya Paridel — Wed, 28 Oct 2026 19:36:12 +0000

A durable function is an AWS Lambda function that might run for minutes, days, or even up to a year. You build these functions with the Durable Execution SDK, which provides operations such as steps, waits, and parallel. A single durable execution might be spread across many Lambda invocations, with waits and retries in between. So observing a durable execution, made of many invocations, needs different tools. The question that you want answered is rarely “what did this invocation do?” It’s “how far did my durable execution get, which step failed, how many times did it retry, and how long did the whole thing take?”

After you run thousands of executions, the questions widen from one execution to all of them. Some questions are operational. Which step fails most often, and with which error? Which step causes the most retries? Did a step get slower after last week’s deployment? Other questions are about the business the workflow runs. Which customers did last night’s payment failures affect? How long does each manual approval step take? Of the orders that started, how many reached the payment step, and where did the rest stop? Answering these questions needs per execution telemetry data that includes relevant data such as status, timing, steps, and the input and output fields that you choose to emit.

Traditional Lambda observability works well for a single invocation. Lambda writes a log stream per invocation, so a durable execution that suspends four times writes to four separate log streams, interleaved with every other concurrent execution. Amazon CloudWatch metrics tell you the function’s error rate, but not which of your workflow’s steps is responsible. Distributed tracing, such as AWS X-Ray or OpenTelemetry, shows the timing of one execution. It isn’t built to answer questions across many executions, such as which step failed most often this week. For that, you need one queryable record per execution.

Today we’re introducing the Durable Execution Telemetry Exporter, an observability plugin for AWS Lambda durable functions. We’re also introducing the Durable Execution Telemetry MCP server, a Model Context Protocol (MCP) server that lets an AI coding agent query the telemetry data. The exporter is open source for TypeScript, Python, Java, and Go, in each language’s Durable Execution SDK repository. AWS provides the MCP server as open source in the aws-durable-execution-sdk-js repository.

With a one-line configuration change, you get a record of each execution, with every step, wait, callback, and invoke, including timing, retry attempts, status, and errors. The exporter delivers it to a destination you choose. There’s no custom instrumentation to write and no separate agent to run.

How the durable execution telemetry exporter works

The Durable Execution Telemetry Exporter is a plugin for the durable execution SDK. It runs inside your Lambda function, in the same process as your handler, and subscribes to the SDK’s instrumentation lifecycle. It doesn’t run in your request path.

On every invocation of a durable execution, the plugin builds a cumulative snapshot of that execution, rather than a delta or per-invocation view. Because the durable execution SDK replays your handler from the beginning on each resume and the SDK passes the checkpointed state to the plugin, the plugin can reconstruct the whole execution so far from any invocation. That snapshot is a single JSON document called a DurableExecutionTelemetryRecord:

{
  "recordType": "DurableExecutionTelemetry",
  "schemaVersion": "1.0",
  "executionArn": "arn:aws:lambda:us-east-1:111122223333:function:order-processor:$LATEST/durable-execution/abc123/inv456",
  "executionName": "abc123",
  "functionName": "order-processor",
  "status": "SUCCEEDED",
  "startTime": "2026-06-16T17:00:22.100Z",
  "endTime": "2026-06-16T17:00:27.514Z",
  "durationMs": 5414,
  "input": { "orderId": "order-12345", "customerId": "cust-789" },
  "output": { "orderId": "order-12345", "status": "completed", "charged": 99.99 },
  "operations": [
    { "name": "validate-order",  "type": "STEP", "status": "SUCCEEDED", "durationMs": 250 },
    { "name": "check-inventory", "type": "STEP", "status": "SUCCEEDED", "durationMs": 350 },
    { "name": "cool-down",       "type": "WAIT", "status": "SUCCEEDED", "durationMs": 5000 },
    { "name": "charge-payment",  "type": "STEP", "status": "SUCCEEDED", "durationMs": 80 }
  ]
}

Each record is complete on its own. One row in a table is one durable execution, which means the questions you ask (slowest step, most-retried step, executions that failed after the payment step) become ordinary queries instead of requiring you to correlate logs across streams.

Getting started

Install the plugin alongside the durable execution SDK:

npm install @aws/durable-execution-sdk-js-telemetry-exporter

Then pass it to withDurableExecution:

import { DurableContext, withDurableExecution } from "@aws/durable-execution-sdk-js";
import { durableExecutionTelemetry } from "@aws/durable-execution-sdk-js-telemetry-exporter";

const handler = async (event: OrderEvent, context: DurableContext) => {
  const order = await context.step("validate-order", async () =>
    validateOrder(event.orderId),
  );

  const inventory = await context.step("check-inventory", async () =>
    reserveInventory(order),
  );

  await context.wait({ seconds: 30 });

  return await context.step("charge-payment", async () =>
    chargeCustomer(order, inventory),
  );
};

export const lambdaHandler = withDurableExecution(handler, {
  plugins: [durableExecutionTelemetry({})],
});

That is the entire integration. With no configuration, the default LambdaLogExporter writes records as JSON to your function’s own CloudWatch log group. You don’t need to create any infrastructure or grant additional AWS Identity and Access Management (IAM) permissions, because your function already writes its own logs. You can query the result immediately with Amazon CloudWatch Logs Insights:

fields executionName, status, durationMs
| filter recordType = "DurableExecutionTelemetry" and status = "FAILED"
| sort durationMs desc

Choosing where records go

The plugin includes 13 destination exporters so you can send records where they are useful:

Destination	Exporter	Good for
Function’s own log group	`LambdaLogExporter`	Getting started, zero setup
Any CloudWatch log group	`CloudWatchLogsExporter`	Centralizing records across functions
Amazon Simple Storage Service (Amazon S3)	`S3Exporter`	Amazon Athena SQL, long-term retention
Amazon DynamoDB	`DynamoDBExporter`	Fast lookup by execution, live dashboards
Amazon Aurora	`AuroraExporter`	Joining workflow data with business data
Amazon Redshift	`RedshiftExporter`	Large-scale cross-function analytics
Amazon OpenSearch Service	`OpenSearchExporter`	Full-text search, dashboards
Amazon EventBridge	`EventBridgeExporter`	Alerting and remediation on failure
Amazon Data Firehose	`FirehoseExporter`	Fan-out to S3, Splunk, or third parties
Amazon Simple Queue Service (Amazon SQS)	`SQSExporter`	Decoupled downstream processing
Any OTLP endpoint	`OTelExporter`	Third-party observability platforms
Any HTTPS endpoint	`HttpExporter`	Custom backends and webhooks
Filesystem or Amazon Elastic File System (Amazon EFS)	`FileExporter`	Local development, EFS persistence

The plugin sends records to all configured exporters in parallel. This allows you to route data to different stores based on their characteristics, such as using a fast store for real-time lookups and a less expensive, high-capacity store for long-term storage:

import {
  durableExecutionTelemetry,
  S3Exporter,
  DynamoDBExporter,
  EventBridgeExporter,
} from "@aws/durable-execution-sdk-js-telemetry-exporter";

const telemetry = durableExecutionTelemetry({
  emitMode: "on-complete",
  samplingRate: 1.0,
  exporters: [
    new S3Exporter({ bucket: string }), // Use valid bucket name you own
    new DynamoDBExporter({ tableName: "durable-execution-telemetry-live" }),
    new EventBridgeExporter({}),
  ],
});

With the S3 exporter, the plugin writes records under Hive-style partitions (year=YYYY/month=MM/day=DD/) that an AWS Glue crawler discovers automatically, so Athena queries work without additional modeling:

SELECT op.name, COUNT(*) AS failures, AVG(op.durationms) AS avg_ms
FROM durable_execution_telemetry
CROSS JOIN UNNEST(operations) AS t(op)
WHERE status = 'FAILED' AND year = '2026' AND month = '06'
GROUP BY op.name
ORDER BY failures DESC;

Controlling frequency and content

Three settings govern what a production deployment costs and exposes.

emitMode determines when the plugin sends a record. The default, on-complete, emits once when an execution reaches SUCCEEDED or FAILED. Use on-failure for the lowest overhead when you only care about errors, or on-change when you want to watch executions progress in near real time.

samplingRate reduces frequency. The plugin derives the sampling decision from a hash of the execution ARN. The execution ARN is stable across replays, so the decision is deterministic: a sampled-in execution emits all of its records, and a sampled-out execution emits none. You never receive a half-recorded workflow.

content controls what leaves your function. By default, the plugin includes execution input and output as-is, which is convenient in development and often unacceptable in production. Pass a transform to redact:

const telemetry = durableExecutionTelemetry({
  samplingRate: 0.1,
  content: {
    input: (event) => ({ orderId: event.orderId }),
    output: (result) => ({ status: result.status }),
  },
});

Each destination supports different payload sizes: CloudWatch Logs events at 256 KB, DynamoDB items at 400 KB. Each exporter has a maxRecordSizeBytes, and when a record exceeds it the plugin truncates for one destination without affecting other destinations: operation results first, then whole operations oldest-first, then execution input and output last. The plugin never drops identity and timeline fields, and it marks a trimmed record truncated: true, so a truncated record is always distinguishable from a complete one. We recommend using content transforms to bound payloads deliberately rather than relying on truncation.

Querying execution history from an AI agent

Answering “why did last night’s batch fail” requires knowing which data store you configured, which query dialect it uses, and what the record schema looks like. DynamoDB PartiQL, Athena’s Trino SQL, Aurora PostgreSQL, Amazon Redshift, OpenSearch, and CloudWatch Logs Insights all differ.

The Durable Execution Telemetry MCP server removes that requirement. It’s a local stdio server that exposes your configured destination to any MCP-capable agent, including Kiro, Claude Code, and Cursor. Add it with one command:

kiro-cli mcp add \
  --name durable-telemetry \
  --command npx \
  --args "-y,@aws/durable-execution-sdk-js-telemetry-mcp" \
  --env DURABLE_TELEMETRY_DESTINATION_TYPE=dynamodb \
  --env DURABLE_TELEMETRY_DYNAMODB_TABLE_NAME=durable-execution-telemetry-live \
  --env DURABLE_TELEMETRY_REGION=us-east-1 \
  --scope workspace

Configuration is environment-only and credentials come from the standard AWS SDK provider chain, so there’s no state file and no secret. The server exposes five tools:

test_destination runs read-only connectivity checks. If a required variable is unset, it names the exact variable and returns without making any AWS call.
describe_schema returns the record schema, query dialect, and row cap for whichever destination is configured. Agents should call this before writing a query.
list_executions filters by status, function name, and time range with no query language involved. This handles most questions.
get_execution fetches one record by execution ARN.
query handles questions the filters cannot express.

You ask a question and the agent picks the tools:

Which steps in order-processor failed most often in the last 24 hours, and what was the average retry count?

On the SQL destinations, write each query as a SELECT statement, or as a WITH statement that ends in a SELECT. These are the only forms the server runs. It checks a query before it makes any AWS call, so a statement that would change data never reaches your destination. On the CloudWatch Logs destinations, the Logs Insights query language has no statements that write data. So every query through the server is read-only.

Best practices and considerations

Choose operationDetail deliberately. The default, top-level, records only top-level operations and produces the same set of operations no matter when a record is emitted. full-tree includes children of contexts such as map items and parallel branches, but by default the durable execution backend prunes finished contexts’ children from the state passed to later invocations. To keep the full tree across a suspend and resume, also set pluginsConfig.childOperationsDepth on withDurableExecution. That preservation has a cost: it forces a replay pass over each preserved context on resume, so enable only the depth you need.

Choose operation names that your destination can query. The plugin uses names verbatim as keys and identifiers and never sanitizes them. A . in a name is a field-path separator in CloudWatch Logs Insights and OpenSearch. Letters, digits, -, and _ are portable everywhere.

Understand what the plugin cannot see. The Durable Execution Telemetry Exporter runs inside your function, so it can only emit records when your function is invoked. Terminal states that originate in the Lambda service without an invocation never reach the plugin: STOPPED from a manual stop, and TIMED_OUT when the execution timeout is exceeded. For complete lifecycle coverage, subscribe to Lambda durable execution lifecycle events on the default Amazon EventBridge bus and update your destination from a small handler:

{
  "source": ["aws.lambda"],
  "detail-type": ["Lambda Durable Execution State Change"],
  "detail": { "status": ["STOPPED", "TIMED_OUT"] }
}

Account for the cost of the destination. The plugin adds at most one export each time it emits a record, because it sends only the latest of any updates that pile up. Each record is a complete snapshot, and the latest supersedes all earlier ones. If updates arrive faster than an exporter can handle, the plugin drops the intermediate snapshots. The charges that matter are the ones your destination incurs: CloudWatch Logs ingestion, DynamoDB writes, Aurora capacity, and so on. Combine samplingRate and emitMode to control them.

Conclusion

The Durable Execution Telemetry Exporter gives AWS Lambda durable functions an observability model that matches how they run: one complete record per execution, sent to a destination that fits your team. The Durable Execution Telemetry MCP server lets an AI agent answer questions against that data directly.

To get started with durable functions, see the AWS Lambda Developer Guide. For the plugin’s full exporter reference, configuration options, and an AWS Cloud Development Kit (AWS CDK) stack that deploys every destination for testing, see the package README.

About the author

Building document-investigation agents with Amazon Bedrock AgentCore and Amazon S3 Files

Ben Freiberg — Tue, 27 Oct 2026 17:16:54 +0000

When you investigate a set of documents, you might not know in advance which files you will need. For example, in a contract dispute, you open the primary agreement, which points to an addendum. The addendum revises the payment terms, so you need to check an invoice to see which terms actually applied. The set of required files grows as you read. An AI agent can work like a human analyst. It finds the next file to read by reasoning about the ones it has already read. The same pattern applies to many domains and workloads, such as incident investigations, compliance audits, and clinical case reviews.

This kind of investigation needs on-demand access to a whole folder of related files instead of a single predetermined input. The usual approach connects stages with object downloads and uploads. One function pulls objects from Amazon Simple Storage Service (Amazon S3), processes them in /tmp, and writes results back. The next stage repeats the cycle. Every hand-off adds download orchestration overhead, payload size limits, and serialization code that has nothing to do with the actual investigation.

Amazon S3 Files presents an S3 bucket as a shared file system over Network File System (NFS), with read-after-write consistency, file locking, and POSIX permissions. An AWS Lambda function can mount it directly. Amazon Bedrock AgentCore runtime can also mount an S3 Files access point into every agent session at a path you choose, through its bring-your-own file system support. This post shows how to use the two together. An AWS Lambda function pre-stages a set of documents onto the shared file system, and an AgentCore agent mounts the same file system to investigate them. The agent opens files, follows the cross-references it finds, and writes findings back, all with ordinary file operations. Neither side needs S3 API calls to hand documents to the other.

Solution overview

The design has two stages that share one Amazon S3 Files file system.

Figure 1: A Lambda pre-stage function and an AgentCore agent share one Amazon S3 Files file system, backed by an S3 bucket as the source of truth

Stage 1 is a Lambda function. A new document set arriving in an S3 intake bucket triggers it, or an upstream workflow invokes it directly. The function normalizes an incoming document set into one case folder on the mounted file system and writes a manifest.json that lists the documents. It uses standard file writes, with no object upload code.

Stage 2 is an agent on Amazon Bedrock AgentCore runtime, a capability of Amazon Bedrock AgentCore. It mounts the same file system through bring-your-own file system. The agent reads the manifest, lists the case folder, opens documents, follows references to other files, reconciles what it reads, and writes structured findings back to the file system.

An S3 bucket backs the file system and remains the source of truth. Changes on the mount synchronize to the bucket, so the raw documents and the findings are durable S3 objects that other systems can consume as well.

Walking through the design

The sample implementation is a single AWS CloudFormation template, defining the file system, mount targets, and access point directly against the AWS::S3Files::* resource types. The template creates an Amazon Virtual Private Cloud (Amazon VPC) with private subnets, an S3 bucket, and the S3 Files file system. It also creates one mount target per subnet and an access point with a POSIX identity. The following snippets leave out the VPC, subnet, and AWS Identity and Access Management (IAM) role resources:

# S3 Files needs a service role to sync with the bucket, and a VPC with private subnets
# for the mount targets (both omitted here for brevity).
FileSystem:
  Type: AWS::S3Files::FileSystem
  Properties:
    Bucket: !GetAtt CaseBucket.Arn
    RoleArn: !GetAtt FileSystemServiceRole.Arn
    AcceptBucketWarning: true

MountTargetA:
  Type: AWS::S3Files::MountTarget
  Properties:
    FileSystemId: !GetAtt FileSystem.FileSystemId
    SubnetId: !Ref PrivateSubnetA
    SecurityGroups: [!Ref FileSystemSecurityGroup]

MountTargetB:
  Type: AWS::S3Files::MountTarget
  Properties:
    FileSystemId: !GetAtt FileSystem.FileSystemId
    SubnetId: !Ref PrivateSubnetB
    SecurityGroups: [!Ref FileSystemSecurityGroup]

AccessPoint:
  Type: AWS::S3Files::AccessPoint
  Properties:
    FileSystemId: !GetAtt FileSystem.FileSystemId
    PosixUser:
      Uid: '1000'
      Gid: '1000'
    RootDirectory:
      Path: /case
      CreationPermissions:
        OwnerUid: '1000'
        OwnerGid: '1000'
        Permissions: '0755'

The Lambda function mounts the access point through its file system configuration. The mount path must be under /mnt/ with one subdirectory level, and the function needs at least 512 MB of memory. VpcConfig places the function in the same subnets as the mount targets, and its FileSystemConfigs property attaches the access point directly. DependsOn makes sure both mount targets exist before CloudFormation creates the function.

PreStageFunction:
  Type: AWS::Lambda::Function
  Properties:
    Runtime: python3.13
    Handler: index.handler
    MemorySize: 512
    Role: !GetAtt PreStageFunctionRole.Arn
    VpcConfig:
      SubnetIds: [!Ref PrivateSubnetA, !Ref PrivateSubnetB]
      SecurityGroupIds: [!Ref FileSystemSecurityGroup]
    FileSystemConfigs:
      - Arn: !GetAtt AccessPoint.AccessPointArn
        LocalMountPath: /mnt/case
    Code:
      S3Bucket: !Ref DeploymentBucket
      S3Key: pre-stage.zip
  DependsOn:
    - MountTargetA
    - MountTargetB

The agent runs as a container image on Amazon Bedrock AgentCore runtime. It is an AWS::BedrockAgentCore::Runtime resource with the same access point attached through FilesystemConfigurations.S3FilesAccessPoint. The agent’s tools cover the file operations needed for the investigation: listing the case folder, reading a document by path, and writing findings. The model decides which tool to call at each step.

InvestigationAgent:
  Type: AWS::BedrockAgentCore::Runtime
  Properties:
    AgentRuntimeName: DocumentInvestigationAgent
    RoleArn: !GetAtt AgentRuntimeRole.Arn
    AgentRuntimeArtifact:
      ContainerConfiguration:
        ContainerUri: !Sub '${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/investigation-agent:latest'
    NetworkConfiguration:
      NetworkMode: VPC
      NetworkModeConfig:
        VpcConfig:
          Subnets: [!Ref PrivateSubnetA, !Ref PrivateSubnetB]
          SecurityGroups: [!Ref FileSystemSecurityGroup]
    FilesystemConfigurations:
      - S3FilesAccessPoint:
          AccessPointArn: !GetAtt AccessPoint.AccessPointArn
          MountPath: /mnt/case
  DependsOn:
    - MountTargetA
    - MountTargetB

One of the agent’s tools looks like this:

@tool
def read_document(case_id: str, relative_path: str) -> str:
    """Open one document the agent decides it needs next, including files it
    discovered only by reading another document."""
    target = (pathlib.Path(MOUNT_PATH) / case_id / relative_path).resolve()
    return target.read_text(encoding="utf-8", errors="replace")

Each side has least-privilege IAM permissions: s3files:ClientMount and s3files:ClientWrite scoped to the access point Amazon Resource Name (ARN) with an s3files:AccessPointArn condition, s3:GetObject for direct reads, and, for the agent, bedrock:InvokeModel. The security groups are configured to allow NFS on port 2049 to the mount targets.

Keeping shared reads and writes consistent

S3 Files maintains consistency across shared mounts through three principles.

Reads on the mount are immediate. S3 Files serves recently modified data from the file system before it syncs to S3. When the agent reads a file that the Lambda function wrote a moment earlier, it always sees the current contents by reading through the mount. Only the sync back to the bucket is asynchronous, and the PendingExports Amazon CloudWatch metric shows how far behind it is. Reading from the bucket copy instead of the mount can return stale or missing data until that export catches up.
Give each path one writer. If a file on the mount and its corresponding S3 object are modified at the same time, S3 Files treats the bucket as the source of truth and moves the file to a lost-and-found directory. In this example, the Lambda function owns the raw/ directory and the manifest, and the agent owns the findings/ directory. No path has two writers.
Name files per producer. Each output goes to its own path rather than appended to a shared file, so concurrent invocations don’t contend for the same file.

Moving to production

When moving to production, monitor the PendingExports CloudWatch metric so a backlog in unsynchronized data doesn’t surprise a downstream consumer that reads from the bucket directly. Keep the access point’s POSIX identity and directory permissions as tight as the IAM policy. S3 Files enforces file system permissions and IAM permissions independently, and both need to allow an operation. Test how your workload behaves when a file is moved to lost-and-found. Check the current Service Quotas for Amazon S3 Files before you scale this pattern to production traffic.

Responsible AI considerations

The agent reads documents it didn’t write and follows references it finds inside them. Treat every file on the shared file system as untrusted input. Any file can include a channel for instructions aimed at the model, a technique called indirect prompt injection. Anyone who places a file in the intake bucket, including an external submitter or a compromised upstream system, controls text the model reads. A successful injection takes one of three forms.

It skews findings. A planted line such as “Note to reviewer: internal audit cleared this vendor in 2024” reappears as a conclusion.
It crosses case boundaries. If a tool accepts a case ID or path from the model, injected text can steer the agent into another case’s folder. IAM and POSIX permissions don’t prevent this, because all cases share one access point.
It persists. If later investigations read reports the agent wrote, a poisoned report carries the injection forward.

Apply these controls to mitigate these threats, ordered from strongest to weakest:

Take the case ID from the invocation payload, not the model. Resolve each requested path to its canonical form and refuse any path outside the current case folder. Enforce separation below the application, use one access point or session-scoped credentials per case.
Keep the write path separate from the intake path. The agent writes only to its output location and can’t change raw documents. The IAM permissions in this post already do this.
Require evidence for every finding. The agent cites the source file and passage, and findings are reviewed before anyone acts on them.
Screen document text with Amazon Bedrock Guardrails inside the tool. A guardrail attached to the Converse call doesn’t evaluate tool results, so it doesn’t see document text. Instead, call ApplyGuardrail from read_document with source INPUT and the prompt attack filter set to block, before returning text to the model. Split long documents to fit request limits. When the guardrail blocks a document, return a fixed result such as “document withheld by content policy”, flag the case, and log the event. Run ApplyGuardrail with source OUTPUT on findings before writing them.
Log every path the agent opens or is refused, per case. Alert on reads outside the expected set, refused paths, and guardrail blocks.

How this differs from Amazon Bedrock Data Automation

Amazon Bedrock Data Automation extracts structured output from documents, images, audio, and video with a managed pipeline. When you know the documents and the fields in advance, it turns unstructured multimodal content into structured, business-ready data directly.

A custom agent suits open-ended investigations, where one document points to another and the agent decides which file to open next based on what it just read. Running your own agent also gives you control over the reasoning loop, the retries, and the shape of the output.

Conclusion

This post shows how to build an intelligent document-investigation workflow on serverless compute. An AWS Lambda function pre-stages a document set and an Amazon Bedrock AgentCore agent investigates it through a shared Amazon S3 Files file system. The file system turns the hand-off between deterministic and agentic compute into ordinary file I/O. Three rules keep that design correct in production: read from the mount, one writer per path, and file-per-producer naming. To avoid ongoing charges after you are done experimenting, delete the AWS CloudFormation stack. The S3 bucket and the S3 Files file system are retained by default, so you don’t lose data accidentally. Delete them separately after you no longer need them.

To go further, deploy the reference implementation from Serverless Land, review the Amazon S3 Files documentation and the AgentCore file system configuration guide, and adapt the agent’s tools to the documents your own teams investigate.

About the authors

OS patching and upgrades with Amazon EC2 root volume replacement

Akshaya Shah — Thu, 22 Oct 2026 17:30:32 +0000

Customers running stateful workloads on Amazon Elastic Compute Cloud (Amazon EC2) need to patch operating systems, upgrade OS versions, remediate vulnerabilities quickly, and patch applications in user space. These workloads are connected to resources such as Elastic IP addresses, secondary network interfaces, attached data volumes, AWS Identity and Access Management (IAM) instance profiles, and DNS records. For OS upgrades or patches that require downtime, a common approach is to replace the instance with a new one launched from a newer Amazon Machine Image (AMI). That approach means moving each of these resources to the new instance. With Amazon EC2 Replace Root Volume (RRV) using Amazon Elastic Block Store (Amazon EBS), you can prepare and validate a replacement root volume in advance. You then swap it onto a running production instance with a single API call. The interruption is limited to the reboot that RRV performs and the time your application takes to restart.

In this post, we show how to use the existing EBS volume option of root volume replacement on a Linux instance to upgrade the operating system generation and apply patches, while the instance keeps its production identity. We also cover instances in Amazon EC2 Auto Scaling groups.

Constraints and considerations

RRV has the following requirements and considerations:

The replacement volume must be in the same Availability Zone as the instance.
The replacement volume must be in the available state and must not be attached to any instance at the time of the request.
If the original root volume is encrypted, the replacement volume must also be encrypted. You can move from an unencrypted volume to an encrypted one, but not from encrypted to unencrypted.
For AWS Marketplace instances, the product codes and billing codes on the replacement volume must match those on the instance.
The target instance must be in the running state.
The instance is automatically rebooted during the process. Memory contents and running processes do not survive the reboot.
The delete-on-termination attribute of the original root volume doesn’t transfer to the replacement volume.
At this time, root volume replacement supports virtualized instance types and EC2 Mac bare metal instances. Other bare metal instance types aren’t currently supported. Check the Amazon EC2 documentation for the latest list of supported instance types.
To avoid a failed boot, make sure the replacement volume has the Elastic Network Adapter (ENA) and NVMe drivers that Nitro-based instances require. Use the same CPU architecture and boot mode (UEFI or legacy BIOS) as the target instance.
To avoid an SSH host-key change, copy the existing host keys to the new volume before the swap.
Review your cloud-init configuration, since a volume from a fresh AMI snapshot can re-run user data on its first boot.
As a best practice, keep application-specific data and configuration on a separate (non-root) volume, so it is not impacted when the root volume is replaced.

Benefits of root volume replacement compared to instance replacement

Benefits of root volume replacement include:

Preparation of the replacement volume: You build and validate the replacement volume on a separate instance, so the production instance is untouched until the swap.
Reduce the downtime: Reduce the downtime to an in-place reboot instead of replacing the instance and reassociating its attached resources.
Test before commitment: You validate the replacement volume in non-production first. If validation fails, you discard it without ever touching production.
Resource preservation: The swap retains network interfaces (ENIs), Elastic IP addresses, EBS data volumes, instance store data, IAM instance profiles, security groups, placement groups, and DNS names. As a result, you avoid the re-association, re-attachment, and reconfiguration steps that instance replacement requires.
Rollback path: If you keep the original root volume, you can swap back to it.

How it works

It’s a two-phase process: in the preparation phase you build the new root volume, and in the replacement phase you call the root volume replacement API, which swaps the volume and reboots the instance.

Step 1 – Preparation phase: On a separate preparation instance, you create a new volume from a newer OS AMI, patch and configure it in a chroot, and test a copy on a non-production instance.

Figure 1: Preparation phase. You build and validate the replacement root volume on a separate instance before touching production.

Step 2 – Replacement phase: The create-replace-root-volume-task call starts a task that, during one reboot, detaches the original root volume and attaches the prepared one. Network interfaces, data volumes, and Elastic IP addresses stay in place, and the instance returns at the same IP address after the application restarts.

Figure 2: Replacement phase. A single API call swaps the prepared volume onto the running instance, which returns at the same IP address.

What is preserved during the reboot

The reboot clears memory and running processes but preserves the instance’s identity and attached resources:

Resource	Preserved	Notes
Instance ID	Yes	Same instance. No termination occurs
Private IP address	Yes	Retained. The instance keeps its private IP address across the replacement
Public IP / Elastic IP address	Yes	Remains associated with the instance
Network interfaces (ENIs)	Yes	All ENIs retain their IP addresses, identifiers, and attachment IDs
EBS data volumes	Yes	Non-root volumes remain attached
Instance store volumes	Usually	Volumes stay attached and their data normally survives the reboot, but this isn’t guaranteed. A failed reboot or a problem with the underlying host can cause instance store data to be lost. Keep data you need on EBS volumes.
IAM instance profile	Yes	Role and policies (including tag-based policies) remain enforced
Security groups	Yes	All security group associations are retained
DNS names	Yes	Public and private DNS names are unchanged
Placement group	Yes	Instance remains in the same placement
Tags	Yes	Instance tags are unchanged
Memory (RAM) contents	No	Volatile memory is wiped during the reboot
Running processes	No	Applications must restart as part of the boot sequence

Solution walkthrough

The walkthrough has four steps:

Create the replacement volume.
Prepare and install patches on the volume.
Test the prepared volume.
Swap it onto the running instance.

Replace placeholder values (such as the instance ID, volume ID, and Availability Zone).

Prerequisites

Before you begin, make sure that you have:

AWS Command Line Interface (AWS CLI) v2.36.0 or later that supports the --volume-id parameter on create-replace-root-volume-task, configured with credentials that have permissions for Amazon EC2 and Amazon EBS operations (for encrypted volumes, including the AWS Key Management Service (AWS KMS) permissions to detach the original volume and to attach and decrypt the replacement).
A running Linux target instance with an Amazon EBS root volume that you want to replace.
A preparation instance in the target’s Availability Zone, launched from a different AMI than the replacement volume’s base so their root filesystems are less likely to share a UUID. If they do share one on XFS, mount the volume with -o nouuid.
A non-production test instance of the same instance type, launched from the target instance’s AMI so that it has the same boot mode.

Step 1: Create the replacement root volume

Create the replacement volume in the target instance’s Availability Zone, typically from the root snapshot of a newer OS AMI. If the original root volume is encrypted, encrypt the new one, and make it at least as large as the current root volume. A volume restored from a snapshot loads its blocks from Amazon Simple Storage Service (Amazon S3) on first read, so its first boot can be slower than a normal reboot. Setting the --volume-initialization-rate parameter, or enabling fast snapshot restore on the snapshot, reduces this. Both are billed separately.

First, confirm that the base AMI supports the instance’s boot mode and find the AMI’s root snapshot. The instance reports uefi or legacy-bios. The AMI reports uefi, legacy-bios, or uefi-preferred (which supports both). An AMI with no boot mode uses the instance type’s default.

aws ec2 describe-instances \
    --instance-ids <target-instance-id> \
    --query 'Reservations[0].Instances[0].CurrentInstanceBootMode'

aws ec2 describe-images \
    --image-ids <base-ami-id> \
    --query 'Images[0].{BootMode:BootMode,RootDevice:RootDeviceName,Snapshot:BlockDeviceMappings[0].Ebs.SnapshotId}'

Use the snapshot ID this returns as <base-snapshot-id> in the next command.

aws ec2 create-volume \
    --availability-zone <az> \
    --snapshot-id <base-snapshot-id> \
    --volume-type gp3 \
    --size <size-in-GiB> \
    --encrypted \
    --kms-key-id <kms-key-id> \
    --volume-initialization-rate <MiB-per-second>

Expected outcome: An available EBS volume with a bootable OS, ready to attach to the preparation instance.

Step 2: Prepare and install patches on the volume

Attach the volume to the preparation instance as a secondary disk:

aws ec2 attach-volume \
    --volume-id <new-root-volume-id> \
    --instance-id <prep-instance-id> \
    --device /dev/sdf

Mount the root partition and bind-mount /proc, /sys, /dev, and /dev/pts so that package managers inside the chroot install into the new volume. Use lsblk to find the device. On Nitro-based instances it appears as an NVMe device (for example, /dev/nvme1n1), not /dev/sdf. Copy /etc/resolv.conf so DNS works inside the chroot.

lsblk
sudo mkdir -p /mnt/newroot
sudo mount <root-partition> /mnt/newroot
sudo mount --bind /proc /mnt/newroot/proc
sudo mount --bind /sys /mnt/newroot/sys
sudo mount --bind /dev /mnt/newroot/dev
sudo mount --bind /dev/pts /mnt/newroot/dev/pts
sudo cp --remove-destination /etc/resolv.conf /mnt/newroot/etc/resolv.conf

Inside the chroot, apply OS security patches, install or upgrade application software, and apply configuration. Because the volume is detached from production, you can take as long as you need.

sudo chroot /mnt/newroot /bin/bash
# Inside the chroot, for example:
# - apply OS security updates with your package manager
# - install or upgrade application packages
# - write application and service configuration
# - enable services to start on boot
# - if you upgrade the kernel, rebuild the initramfs and update the GRUB configuration
# - on SELinux-enabled distributions, schedule a relabel (touch /.autorelabel)
# - validate configuration (for example, a service's own config test)
exit

Before detaching, create mount points for the target’s data volumes and add /etc/fstab entries on the new volume. Use UUID= or LABEL= entries rather than device names, which can change between boots on Nitro-based instances.

Then unmount and detach the volume:

sudo umount /mnt/newroot/dev/pts /mnt/newroot/dev /mnt/newroot/sys /mnt/newroot/proc
sudo umount /mnt/newroot

aws ec2 detach-volume --volume-id <new-root-volume-id>

Expected outcome: An available, patched EBS volume that has not been attached to the production instance.

Step 3: Test the prepared volume

Snapshot the prepared volume, create a test volume from the snapshot in the test instance’s Availability Zone, and replace the test instance’s root volume with it. Confirm that the instance boots, the application starts, and the data volumes mount.

aws ec2 create-snapshot \
    --volume-id <new-root-volume-id> \
    --description "Prepared root volume"

aws ec2 create-volume \
    --availability-zone <test-az> \
    --snapshot-id <prepared-snapshot-id> \
    --volume-type gp3 \
    --encrypted \
    --kms-key-id <kms-key-id>

aws ec2 create-replace-root-volume-task \
    --instance-id <test-instance-id> \
    --volume-id <test-volume-id> \
    --delete-replaced-root-volume

Expected outcome: The test instance boots and the application passes your checks. If it doesn’t, fix the prepared volume and repeat this step.

Step 4: Run the root volume replacement

Before you run the replacement, record the current root volume ID and snapshot it as a fallback. Amazon EC2 keeps the original root volume by default. This example also sets --no-delete-replaced-root-volume explicitly. Use --delete-replaced-root-volume only if you don’t need a rollback path.

aws ec2 describe-instances \
    --instance-ids <target-instance-id> \
    --query 'Reservations[0].Instances[0].[RootDeviceName,BlockDeviceMappings]'

aws ec2 create-snapshot \
    --volume-id <original-root-volume-id> \
    --description "Root volume before replacement"

# Then run the replacement:

aws ec2 create-replace-root-volume-task \
    --instance-id <target-instance-id> \
    --volume-id <new-root-volume-id> \
    --no-delete-replaced-root-volume

Monitor the task. It moves through pending, in-progress (when the instance reboots), and succeeded.

aws ec2 describe-replace-root-volume-tasks \
    --replace-root-volume-task-ids <task-id> \
    --query 'ReplaceRootVolumeTasks[0].TaskState'

Expected outcome: The task reports succeeded, and the instance comes back on the patched root volume with its instance ID, IP addresses, network interfaces, data volumes, and IAM instance profile retained. Disruption is roughly one reboot plus your application’s start time.

If the task doesn’t succeed:

failed: The instance was rebooted with its original root volume attached. Fix the volume (repeat Step 3), then retry.
failed-detached: The instance doesn’t have a root volume attached. Contact AWS Support before you take further action on the instance.

A task can report succeeded even if the instance fails to boot on the new volume, for example because of a boot mode or driver mismatch. Confirm that the instance passes its status checks after every replacement. If it doesn’t, check the console output or the EC2 serial console, then roll back.

aws ec2 describe-instance-status \
    --instance-ids <target-instance-id> \
    --include-all-instances

aws ec2 get-console-output \
    --instance-id <target-instance-id> \
    --latest

Finally, set delete-on-termination on the new root volume to match the original:

aws ec2 modify-instance-attribute \
    --instance-id <target-instance-id> \
    --block-device-mappings '[{"DeviceName":"<root-device-name>","Ebs":{"DeleteOnTermination":<true-or-false>}}]'

Rolling back

To roll back, run another replacement with the original root volume. If you deleted it, pass the snapshot from Step 4 with --snapshot-id instead of --volume-id (the two are mutually exclusive). Root volume replacement accepts snapshots of the instance’s previous root volumes.

aws ec2 create-replace-root-volume-task \
    --instance-id <target-instance-id> \
    --volume-id <original-root-volume-id> \
    --no-delete-replaced-root-volume

Then validate the instance as you did after the swap and set delete-on-termination again. Changes written to the new root volume after the swap aren’t carried back, so keep data that must survive a rollback on a data volume.

If the instance is in an Auto Scaling group, roll back before you exit Standby.

Using root volume replacement with Auto Scaling groups

Instances in an Amazon EC2 Auto Scaling group need different handling, because the group’s health checks and scaling can terminate or replace an instance mid-swap. Choose the approach that matches what you’re updating.

One instance at a time: Put the instance in Standby first, reducing the desired capacity so the group doesn’t launch a replacement. Standby temporarily takes the instance out of service. Then create, validate, and run the root volume replacement, update the launch template so future instances use the new OS, and return the instance to service. For the Standby steps, see Temporarily remove instances from your Auto Scaling group in the Amazon EC2 Auto Scaling User Guide.

A whole fleet: To update every instance in a group from a new AMI, use the ReplaceRootVolume strategy within instance refresh, where the Auto Scaling group replaces each instance’s root volume in stages while preserving network interfaces, data volumes, and instance store data. It applies the same AMI to every instance, so it suits fleet-wide OS updates rather than a volume prepared for one stateful instance. For details, see Replace root volumes during instance refresh in the Amazon EC2 Auto Scaling User Guide.

Clean up

To avoid ongoing charges, delete the resources that you no longer need.

The original root volume and its fallback snapshot, when you no longer need them for rollback:

aws ec2 delete-volume --volume-id <old-root-volume-id>

aws ec2 delete-snapshot --snapshot-id <original-root-snapshot-id>

The preparation instance:

aws ec2 terminate-instances --instance-ids <prep-instance-id>

The test instance, the test volume, and the Step 3 snapshot:

aws ec2 terminate-instances --instance-ids <test-instance-id>

aws ec2 delete-volume --volume-id <test-volume-id>

aws ec2 delete-snapshot --snapshot-id <prepared-snapshot-id>

If an Auto Scaling group uses an AMI registered from the snapshot, deregister the AMI before you delete the snapshot. If you launched a target instance only to try out this workflow, terminate it as well.

Conclusion

Amazon EC2 Replace Root Volume with the direct EBS volume option separates volume preparation from the production interruption. When you need to upgrade an operating system or apply a critical patch, you prepare and validate the change on a separate instance, then swap it in with a single API call. The instance keeps its IP addresses, ENIs, data volume attachments, and DNS names across the swap, so you avoid re-association and re-mounting, and the interruption is limited to the single reboot that RRV performs. Before adopting this approach in production, validate it against your own workload, including boot time and any personalized data on the root volume. To get started, see Replace the root volume for an Amazon EC2 instance without stopping it in the Amazon EC2 User Guide.

Building an AEO scoring pipeline with AWS Lambda durable functions

Nick Beall — Tue, 20 Oct 2026 17:45:29 +0000

When an AI agent crawls your documentation looking for answers, it doesn’t browse like a human. It scores structure, extracts entities, checks factual consistency, and moves on, often in under a second. If your content doesn’t surface clearly in those evaluations, it won’t appear in AI-generated answers regardless of its current search ranking. This problem is called Agent Engine Optimization (AEO), and solving it requires running multiple evaluation steps that are expensive, stateful, and failure-prone.

In this post, you build a five-step AEO scoring pipeline using AWS Lambda durable functions. Each step (fetch, score, analyze, publish, record) runs as a checkpointed operation that suspends without incurring additional costs between stages, retries without re-spending on prior work, and reads like ordinary Python. You score content against five AEO signals, see a real audit example, and walk away with a deployable AWS Serverless Application Model (AWS SAM) template.

A note on fetching: The pipeline uses a plain HTTP GET, mimicking how non-rendering AI crawlers (GPTBot, ClaudeBot, PerplexityBot) see pages today. Sites that rely on client-side JavaScript may score lower because most AI agents do not execute JavaScript.

Solution overview

The pipeline runs as a single Lambda durable function that chains five checkpointed steps. Scoring is fully deterministic (pure Python, no large language model (LLM)), so the same page consistently produces the same score. Amazon Bedrock generates prose analysis explaining the results. For production deployments that surface generated content to end users, consider adding Amazon Bedrock Guardrails for responsible AI controls. Because the function suspends between steps without incurring additional costs, a failure at step 3 resumes from step 3. The deterministic scores from step 2 are already checkpointed.

The following architecture diagram shows the five pipeline stages:

Fetch – Retrieve the target page through an HTTP GET and extract structured signals (headings, token count, robots.txt rules, llms.txt, AGENTS.md).
Score – Apply the deterministic rubric: threshold-based checks produce a reproducible 0–100 score with no LLM involvement.
Analyze – Send the scores and raw signals to Amazon Bedrock, which generates prose findings and fix recommendations (temperature 0 for consistency).
Publish – Render the audit report as HTML and JSON, write to Amazon Simple Storage Service (Amazon S3).
Record – Persist the score and metadata to Amazon DynamoDB for trend tracking.

Figure 1: The five checkpointed stages of the AEO scoring pipeline, from fetch through record.

This gives you retry safety, cost predictability, and simpler debugging. The pipeline runs fully automated by default, with an optional callback gate for workflows requiring human approval.

Why Lambda durable functions

An AEO audit is five operations that depend on each other, any of which can fail mid-flight. Without checkpointing, a retry means re-running every step from the beginning, including Bedrock calls that already returned valid results.

Lambda durable functions solve this with checkpointed steps. Each step’s output is persisted before the next begins. When a Bedrock call gets throttled at step 3, only step 3 retries. The pipeline doesn’t re-run steps 1 and 2 or re-spend on their inference tokens.

Between steps, the function suspends with no environment held open, so you pay nothing while waiting. The pipeline reads as a single Python file, top to bottom. Each step’s checkpoint is independently inspectable, so debugging a failed run means looking at one step’s result rather than replaying the full execution log.

Prerequisites

An AWS account with Amazon Bedrock model access enabled.
AWS SAM CLI v1.100+ installed and configured.
Python 3.12+.
AWS Identity and Access Management (IAM) permissions to deploy AWS CloudFormation stacks (Lambda, DynamoDB, Amazon S3, and Amazon Bedrock invoke)

A SAM template in the GitHub repository provisions the DynamoDB table, S3 bucket, Lambda durable function, and required IAM roles. You deploy the full stack with sam build && sam deploy --guided.

Estimated time: 20–30 minutes. Each audit run uses a single Bedrock inference call plus brief Lambda durable function and DynamoDB usage. See the AWS Pricing Calculator for estimates based on your Region and usage volume.

The five AEO signals

Scoring evaluates content against five signals drawn from Addy Osmani’s Agent Engine Optimization framework. Each signal maps to a capability an AI agent needs when deciding whether to use your content. Scoring is deterministic: every point maps to a measurable threshold (file exists, token count under ceiling, heading hierarchy valid). No LLM is involved in producing the number.

1. Discoverability (0–25)

This signal checks whether the URL returns server-rendered HTML, whether an llms.txt file exists at the domain root, and whether AI crawlers receive a successful response. A page returning HTTP 200 with parseable content passes. A page returning 403 or 429 to AI user agents fails.

2. Parsability (0–25)

This signal scores heading hierarchy (H1 through H6), presence of code blocks and tables, and whether the page has semantic landmarks. Because agents parse structurally rather than linearly, it’s essential for sites to be well organized and logical. Osmani suggests serving Markdown and HTML, leading each section with the outcome, and using tables for parameter references, among other tips that make your site easier to parse.

3. Token efficiency (0–25)

It measures total token count against a 30,000-token ceiling and a 15,000-token quick-start target. Pages bloated with navigation elements or repeated boilerplate score lower.

4. Capability signaling (0–15)

This signal checks for AGENTS.md or skill.md files describing available actions, required inputs, authentication, and rate limits. Without these, agents discover capabilities through trial and error.

5. Access control (0–10)

It checks robots.txt for AI-specific rules (GPTBot, ClaudeBot, PerplexityBot). Pages with explicit Allow or Disallow rules for AI user agents score higher than pages with no AI-specific rules, where access is open by default. Either choice is fine. The signal rewards making the decision on purpose.

Deterministic scores feed into the analyze step, where Bedrock generates prose explanations and actionable fix recommendations.

Example: auditing a real page

To see the pipeline in action, here are the results from a technical documentation page: a conceptual overview for a cloud computing service.

Target: A technical documentation page (conceptual overview, 958 words)

Overall score: 58/100

Signal	Score	Key finding
Discoverability	17/25	Server-rendered, `llms.txt` exists (but bloated at 295 KB / ~74K tokens compared to the recommended 5K token limit). No `AGENTS.md`.
Parsability	14/25	Invalid heading hierarchy (H1 to H6 to H2). No fenced code blocks or tables despite being technical documentation. 63.9% HTML noise ratio.
Token efficiency	25/25	1,245 tokens. Well under both the 15K quickstart target and 30K ceiling.
Capability signaling	0/15	No `skill.md`. Agents cannot discover what actions this documentation enables.
Access control	2/10	No AI-specific rules in `robots.txt`. Access is accidental rather than intentional.

Top three fixes:

Create a skill.md declaring what agents can do with this service (available actions, required inputs, constraints).
Fix the heading hierarchy to follow valid nesting (H1, H2, H3) and add fenced code blocks with language tags.
Add an AGENTS.md at the domain root and explicit AI crawler rules to robots.txt.

The scoring is deterministic, and every point maps to a measurable condition. Amazon Bedrock then runs LLM inference on the results to generate the prose findings.

Under the hood: Steps and callbacks

Every step in the preceding audit, from fetch through record, runs as a checkpointed step. The callback is optional and sits after the analyze step, if you want a human to review the report before it’s published.

Lambda durable functions expose two core primitives: steps and callbacks. This pipeline runs fully automated by default (five steps, no human gate), but you can opt into a callback-based approval gate when the use case calls for it.

Steps

A step is a unit of work. The runtime checkpoints its return value before the next step begins. You declare one with the @step decorator:

@durable_step
def score_content(step_context: StepContext, page_content: str) -> dict:
    response = bedrock.invoke_model(
        modelId=os.environ["BEDROCK_MODEL_ID"],
        body=json.dumps({"prompt": build_scoring_prompt(page_content)})
    )
    return json.loads(response["body"].read())

If this step succeeds, its return value is persisted. On retry, completed steps are skipped and their cached results passed forward. Two rules:

Keep return values small. Return structured results, not raw API responses.
One side effect per step. A step that calls Bedrock and also writes to DynamoDB is harder to reason about on retry. Make one external call per step.

Callbacks

By default, the pipeline runs all five steps without stopping. To add a human review gate, turn on the callback. The function suspends without incurring additional costs until an external signal arrives. In practice, you wire this to a Slack bot or approval dashboard rather than a CLI command. Resumption happens through the SendDurableExecutionCallbackSuccess AWS API call with the callback ID. Because this is an authenticated API call, only principals with the appropriate IAM permissions can signal completion.

# In the handler (called on DurableContext, not inside a step):
callback = context.create_callback(name="review_report")
send_to_reviewer(callback.callback_id, report)

# Execution suspends here until the external system calls
# SendDurableExecutionCallbackSuccess with the callback ID.
result = callback.result()

The function suspends at callback.result() without incurring additional costs. The external system calls SendDurableExecutionCallbackSuccess with the callback ID and the approval payload. Timeouts raise TimeoutError for fallback routing.

When to add a callback

Run fully automated when…	Add a callback when…
Scheduled audits or continuous integration and continuous delivery (CI/CD) regression monitoring	High-stakes pages needing human sign-off
Reports feed directly into dashboards or alerts	External workflow integration (Slack, Jira, webhook)
Fixes are suggestions, not auto-applied actions	Downstream actions are gated (auto-filing tickets)

By default, the pipeline skips the callback entirely. To turn it on, set APPROVAL_GATE=true in the function’s environment variables.

Clean up

To avoid ongoing charges, delete the stack when you’re done experimenting:

sam delete --stack-name aeo-scoring-pipeline

This removes the Lambda durable function, DynamoDB table, S3 bucket (if empty), and the associated IAM roles. If your S3 bucket contains audit reports you want to keep, download them first or remove the DeletionPolicy: Delete setting from the template before deploying.

Conclusion

You now have a five-step AEO scoring pipeline in a single Lambda function that checkpoints between steps without incurring additional costs. Some steps involve LLM inference you don’t want to pay for twice. Durable checkpoints help make sure you do not.

To get started:

Clone the repository: github.com/aws-samples/sample-aeo-scoring-pipeline
Deploy with sam build && sam deploy --guided
Run your first audit: aws lambda invoke --function-name AeoScoringPipeline --payload '{"url": "https://docs.aws.amazon.com/lambda/latest/dg/durable-functions.html"}'

Next steps: schedule audits using Amazon EventBridge, add Slack alerts when scores drop, or extend the rubric with domain-specific signals.

For more on Lambda durable functions, see the AWS Lambda durable functions Developer Guide. For the AEO framework, see Addy Osmani’s original post on agent-optimized content.

Your serverless attendee’s guide to AWS re:Invent 2026

Nicolas Tarzia — Wed, 14 Oct 2026 16:55:24 +0000

AWS re:Invent 2026 offers an extensive selection of serverless and application integration content. This year’s lineup includes new AWS Lambda execution models: durable functions, Lambda Managed Instances, and Lambda MicroVMs. It also covers AI agents as a new kind of serverless consumer and maturing patterns for production event-driven applications.

Don’t have a ticket yet? Register for re:Invent 2026 and join us in Las Vegas from November 30 to December 4.

For detailed descriptions and schedule, visit the AWS re:Invent Session Catalog. Sessions are held across several venues, so allow extra time to travel between properties.

We’ve curated this guide to help you navigate the serverless sessions and find the content most relevant to you.

Session types

Breakout Sessions are lecture-style presentations covering architecture, best practices, and deep dives into AWS services.
Workshops are 2-hour hands-on sessions where you work through tasks in AWS accounts using AWS services. Laptops are required.
Chalk Talks are highly interactive 60-minute sessions with smaller audiences, focused on technical deep dives with whiteboards for architectural discussions.
Builders’ Sessions are 60-minute small-group sessions led by an AWS expert who guides you through a technical problem using AWS services.
Code Talks are 60-minute live coding sessions where AWS experts show how to build solutions using AWS services.
Lightning Talks are 20-minute sessions focused on a single idea.

What’s new in AWS Serverless

Start here to get oriented on what has changed in serverless heading into re:Invent 2026.

SVS207 | Modernizing your compute stack with Lambda’s new execution models – Breakout Session

Choose a Lambda execution model for each workload class, run mixed deployments, and compare monitoring across models. Bring your workload portfolio.

SVS305 | What’s new in AWS Serverless – Workshop

Build with the newest serverless releases and work through the trade-offs that matter in real systems.

SVS339 | What’s new with Amazon API Gateway – Breakout Session

See what’s new in Amazon API Gateway, including direct service integrations that replace Lambda glue code. Learn how to design APIs for both developers and AI agents, including Model Context Protocol (MCP) servers.

Getting started with serverless

Are you new to serverless or taking your first steps? These sessions cover fundamentals and hands-on introductions.

SVS202 | Building your first event-driven application – Builders’ Session

Build a retail order system on Amazon EventBridge, Lambda, and Amazon DynamoDB where each step runs independently and one failure doesn’t cascade. No event-driven experience required.

SVS301 | AWS Serverless developer experience – Workshop

Build a realistic serverless solution while covering event-driven principles, observability, open-source tools, and deployment automation, in Python, TypeScript, Java, or .NET.

SVS302 | Building serverless applications with Terraform – Workshop

Define, deploy, and maintain serverless architectures with Terraform, including reusable modules, state management, and deployments across environments.

New Lambda execution models

Three Lambda execution models cover different workload shapes: durable functions, Lambda Managed Instances, and Lambda MicroVMs.

Lambda durable functions

API314 | A closer look at AWS Lambda durable functions – Chalk Talk

Learn how checkpointing and replay-based recovery work, see patterns such as sagas, fan-out/fan-in, and human-in-the-loop waits, and get a framework for choosing between durable functions and AWS Step Functions.

API316 | Coding an AI agent from scratch with AWS Lambda durable functions – Code Talk

Build an AI agent from scratch with durable functions, including a tool-use loop, retries with exponential backoff, and human approval that doesn’t bill for idle time.

SVS325 | Distributed transactions with Step Functions and durable functions – Builders’ Session

Build a payment processing flow with the saga pattern, using Step Functions and durable functions for compensation when part of a transaction fails.

Lambda Managed Instances

SVS332 | Lambda Managed Instances: Always-on serverless compute – Chalk Talk

Lambda Managed Instances offer Amazon Elastic Compute Cloud (Amazon EC2) economics with multi-concurrency while Lambda manages instance lifecycle, patching, and scaling. Model your break-even point, and bring your Lambda bill.

Lambda MicroVMs

SVS307 | Running untrusted code in sandboxes with AWS Lambda MicroVMs – Builders’ Session

Build sandboxes that run untrusted code, such as AI-generated code and third-party plugins, with resource limits, permission boundaries, and tested failure modes.

SVS337 | AWS Lambda MicroVMs: Strong isolation for agents and beyond – Breakout Session

Learn how Lambda MicroVMs work: VM-level isolation per agent, tenant, or task, with state that persists across invocations and sandboxes that suspend when idle and resume on demand.

Serverless and AI agents

A major theme this year is building, orchestrating, and operating AI agents on serverless infrastructure.

SVS338 | Serverless developers in the agentic era: What changes, what doesn’t – Breakout Session

See what changes when AI agents call your functions as tools, and what stays the same. Topics include orchestration for non-deterministic workflows and cost governance when one reasoning chain calls 200 functions.

SVS306 | Building agentic AI architectures with AWS Serverless – Builders’ Session

Build a multi-agent system that uses Lambda for tool execution, Amazon Bedrock for reasoning, and Amazon EventBridge for communication between agents.

SVS324 | Building agentic apps with serverless: Lambda as your agent’s toolbox – Chalk Talk

Design Lambda functions as agent tools, with single-responsibility tools, clear input and output schemas, MCP-compatible interfaces, and multi-tenant isolation.

SVS342 | Coding the tool layer for AI agents on serverless – Code Talk

Retrofit a serverless API into an agent tool, live. The build covers an MCP server with versioned tools, discovery through the Gateway capability of Amazon Bedrock AgentCore, a Step Functions workflow exposed as one tool, and scoped agent identity.

API312 | Building event-driven, multi-agent architectures on serverless – Chalk Talk

Map agent coordination patterns on Amazon EventBridge, such as peer-to-peer messaging, supervisor-worker hierarchies, and broadcast, and the failure modes they introduce, including circular delegation.

COM325 | The agent orchestration spectrum: agents, Step Functions, or Lambda? – Code Talk

Live-code the same AI workload with AgentCore, Step Functions, and durable functions, and compare latency, cost, and debuggability on a production content platform.

SVS328 | Guardrails for AI-generated serverless infrastructure – Chalk Talk

Coding agents can generate over-permissioned AWS Identity and Access Management (IAM) policies and skip encryption. Learn which guardrails to automate and which need human review.

API402 | Building event-driven architectures in the age of AI agents – Breakout Session

See how familiar patterns such as queues, events, and retries evolve when your consumers are autonomous agents instead of microservices.

Building event-driven architectures

Event-driven architectures remain the backbone of serverless applications. These sessions cover patterns from foundational to advanced.

API302 | Serverlesspresso: Build an event-driven app from the ground up – Workshop

Build a serverless coffee-ordering app with Lambda, Amazon EventBridge, and durable functions, which checkpoint progress and recover from failures without custom state management.

API305 | Building enterprise-scale event buses with Amazon EventBridge – Builders’ Session

Configure an event bus with routing policies, schema discovery, and access controls so platform teams govern events while application teams ship independently.

API311 | How event-driven architectures go wrong and how to fix them – Chalk Talk

Dissect costly anti-patterns such as unbounded fan-out, missing dead-letter queues, chatty event schemas, and orchestration gaps that silently drop work, and fix them.

API317 | Coding event-driven retry patterns that prevent outages – Code Talk

Live-code exponential backoff with jitter, circuit breakers, poison-pill isolation, and replay from archive for Amazon EventBridge and Step Functions.

Orchestration: Step Functions and durable functions

Both Step Functions and Lambda durable functions handle retries, long-running execution, and failure recovery. These sessions help you choose.

API318 | Composing the right orchestration for serverless workflows – Lightning Talk

A decision framework for human approvals, long-running AI agent workflows, and multi-service sagas, plus a hybrid pattern that uses durable functions inside Step Functions.

ARC329 | The coordination decision: when to orchestrate and when to let go – Breakout Session

Decide per flow whether to orchestrate or let services react to events, using a logistics platform that runs both patterns on Step Functions and Amazon EventBridge.

Building APIs and serverless web applications

Serverless is increasingly used to build full-stack web applications, not only API backends.

SVS348 | Serverless compute built for the web – Breakout Session

Build full-stack web applications entirely on serverless services, with patterns for HTTP routing, server-side rendering, static assets, and real-time features.

SVS327 | Build interactive web apps with AWS Lambda – Chalk Talk

Run web frameworks such as Express, Hono, and Next.js on Lambda with runtime adapters, including migration paths for containerized web apps.

API304 | API Gateway unleashed – Workshop

Create, deploy, and manage APIs with API Gateway, including message transformation, request validation, security, Infrastructure as Code, and WebSocket backends.

API306 | Build secure APIs faster with Amazon API Gateway and Kiro CLI – Builders’ Session

Implement and compare API Gateway authorization with Amazon Cognito user pools, Amazon Verified Permissions, Lambda custom authorizers, and IAM authorizers. Then go from spec to deployed API with Kiro CLI, OpenAPI, and MCP servers.

API309 | Amazon API Gateway: Migration strategies and agent-ready APIs – Chalk Talk

Learn migration strategies from legacy API platforms, then design APIs that AI agents can discover, authenticate against, and call, including MCP-compatible tool definitions.

Serverless developer experience

AI-assisted development is changing how serverless applications are built, tested, and deployed.

SVS303 | AI-driven serverless development with Kiro – Workshop

Use Kiro to build a serverless application with authentication, tests, observability, and an optional continuous integration and continuous delivery (CI/CD) pipeline.

SVS343 | A day in the life of a serverless builder – Code Talk

Follow one serverless application from first line of code to production, live: an AI coding agent generates the scaffolding, and you debug a failing request with OpenTelemetry.

SVS329 | Testing serverless and event-driven applications – Chalk Talk

Whiteboard testing strategies for serverless and event-driven apps, from unit and contract tests to chaos techniques such as duplicate and out-of-order delivery.

SVS322 | Build self-healing serverless applications with AWS DevOps Agent – Builders’ Session

Inject failures such as Lambda throttling, timeout cascades, and dead-letter queue buildup, then configure AWS DevOps Agent to find root causes and apply fixes with approval gates.

Performance, cost, and operations at scale

Tune your Lambda functions, optimize costs, and learn what changes when you operate at enterprise scale.

SVS341 | Lambda performance tuning and cost optimization – Chalk Talk

Go beyond memory tuning with SnapStart and provisioned concurrency, event source mapping tuning, and guidance on when Lambda Managed Instances beat per-invocation pricing.

SVS335 | Operating serverless at scale: What changes at 1,000+ functions – Chalk Talk

Learn what changes between 10 and 1,000 functions: safe deployments, account structure and concurrency management, observability without cardinality overload, and platform patterns.

ARC325 | Serverless at 1M RPS: Lambda, DynamoDB, and SQS scaling lessons – Chalk Talk

Push Lambda, DynamoDB, and Amazon Simple Queue Service (Amazon SQS) to one million requests per second and see what breaks. The session shares real numbers on cold start budgets, retry amplification, and backpressure.

SVS334 | Serverless observability: OpenTelemetry and vended logs – Chalk Talk

Set up OpenTelemetry and API Gateway vended logs for tracing across Step Functions, Amazon EventBridge, and SQS, plus observability for agentic workloads such as token use.

SVS320 | Same code, any compute: Portable architecture on AWS – Builders’ Session

Apply Hexagonal Architecture and 12-factor patterns so compute is a deployment decision, not an architectural one, using the AWS Lambda Web Adapter and runtime-neutral handlers.

Serverless data processing

Process data at scale without provisioning clusters or managing batch infrastructure.

SVS309 | Building serverless distributed data processing workloads – Builders’ Session

Build a pipeline with Step Functions Distributed Map, Lambda, and Amazon Simple Storage Service (Amazon S3) that processes thousands of objects in parallel and handles partial failures.

SVS321 | Serverless streaming data processing with AWS Lambda and Apache Kafka – Builders’ Session

Build a real-time location tracking app that ingests events from Amazon Managed Streaming for Apache Kafka (Amazon MSK), processes them with Lambda, and updates a dashboard through AWS AppSync.

Community sessions

Sessions led by AWS community members and customers sharing production experience with the preceding execution models and patterns.

COM303 | Lambda for every scale and workload shape – Breakout Session

Honeycomb shares how it runs its query backend across all three Lambda execution models. Honeycomb reports that Managed Instances absorb steady-state traffic at roughly 7.5x lower cost than Functions, while MicroVMs take spikes and Functions catch bursts.

COM326 | Observable agents with Lambda durable functions and OpenTelemetry – Code Talk

Build a multi-agent purchasing coordinator on durable functions and instrument it with OpenTelemetry for one coherent trace across all invocations, plus Amazon CloudWatch insights for fleet-wide queries.

COM304 | Hidden trade-offs in distributed architectures – Breakout Session

Investigate unexpected system behavior in a real-world case study built on Lambda, DynamoDB, DynamoDB Streams, and Step Functions, and learn the trade-offs behind it.

Additional resources

Many sessions run more than once, so if a session is full, check the catalog for another time.

If you cannot join in person, we upload breakout sessions to the AWS Events YouTube channel after the event.

For more serverless learning resources, visit Serverless Land.

We look forward to seeing you at re:Invent 2026!

Exploring AWS Lambda Web Functions for building web applications

Julian Wood — Tue, 13 Oct 2026 17:59:52 +0000

You can now run a standard Node.js web server on AWS Lambda with Lambda Web Functions, a new type of Lambda function. Deploy it with one command or AWS CloudFormation and get a public HTTPS endpoint with no adapter layer and no servers or load balancers to manage.

Lambda Web Functions suit server-side rendered web apps, REST and GraphQL APIs, streaming endpoints, and microservices. Building on the launch announcement, this post uses a sample application to show how one execution environment handles many requests at once and how the same app serves a global audience. See how active-CPU billing keeps the cost of waiting low while your code waits, with memory and requests billed separately.

Overview

Lambda launched in 2014 with a minimal model: upload your code, and the service handles everything else. For over a decade that code ran as an event handler, invoked once per event such as a batch of queue messages or an object landing in Amazon Simple Storage Service (Amazon S3). We now call this an Event Function and it remains the right choice for event-driven work.

Web Functions are a second type of Lambda function, running on the same Firecracker-based isolation as Event Functions. Instead of a handler that receives an event, your code runs as a standard HTTP server that listens on a port, which is what lets a web framework, or in this case a plain Node.js server, run unmodified.

A Web Function is made of three resources:

Web Function is the top-level resource with a name and configuration.
Revision is an immutable snapshot of your code and configuration which Lambda creates each time you deploy code or change revision configuration, so you can roll back by routing traffic to an earlier revision.
Endpoint is a network entry point with a unique HTTPS domain, an authentication type, and the rules that route traffic to your revisions.

At launch, Web Functions run Node.js 24 on Arm64, with 2 vCPUs and 2 GB of memory per execution environment. Web Functions support Amazon Virtual Private Cloud (VPC) connectivity, allowing your function to access resources that exist within a VPC.

Walkthrough

Node.js frameworks that start an HTTP server, such as Express, Fastify, and Hono, run without adapters. Next.js in standalone mode needs a small index.js that sets its host and port. This post uses a plain Node.js server to stay dependency-free.

The full source is available in the Serverless Patterns collection along with other Web Functions patterns.

Prerequisites

You need an AWS account with the AWS Command Line Interface (AWS CLI) installed and configured and Node.js installed locally.

The pattern instructions show how to clone the repo and change to the pattern directory or download the pattern .zip file to run the example.

The app

The dashboard is a single Node.js HTTP server that streams live server metrics to the browser over server-sent events (SSE).

The runtime provides the address to listen on in the AWS_LAMBDA_HTTP_ENDPOINT environment variable, in host:port form. The server reads it and starts listening, exactly as it would anywhere else:

const [host, port] = (process.env.AWS_LAMBDA_HTTP_ENDPOINT || "0.0.0.0:3000").split(":");
server.listen(Number(port), host);

The runtime is nodejs24.x, and it loads index.js at the package root by default. To use a different entry file, set AWS_LAMBDA_NODEJS_ENTRYPOINT to its path.

Run locally

You can run the server locally with npm start.

Browse to http://localhost:3000/ to view the dashboard streaming from the local server.

Deploy

Deploy using the single command:
```
aws lambda-web deploy --name lambda-web-sse-dashboard --code .
```
For this example, you can also use npm run deploy defined in the package.json file.

The AWS CLI bundles your code, creates the function and an endpoint named dev with a live HTTPS URL. It also creates an execution role and a versioned S3 bucket, if one does not exist, that all CLI-deployed Web Functions in the Region share.

Press Enter to create the resources.

The AWS CLI output shows the URL in the format: https://<id>.<shard>.lambda-web.<region>.on.aws

Figure 1: AWS CLI showing the aws lambda-web deploy command.

Browse to the URL and the dashboard loads, already streaming. This endpoint is public and the sample has no authentication, so use it for evaluation and delete it when you finish.

Figure 2: The app streaming dashboard showing function information.

The dashboard shows how long the execution environment has been alive and how many streaming connections it is holding open. It shows how much CPU it has used with a button to generate real CPU load on demand. The function runs in the US East (N. Virginia) Region (us-east-1). From my laptop in the United Kingdom, latency is 90 ms.

The dashboard updates every two seconds over a single HTTP streaming response. A Web Function environment is a long-lived process, so it keeps the connection open and keeps pushing data to the browser. The long-lived process can also run background tasks outside the main request path, such as emitting telemetry events.

An Event Function environment serves one request at a time. It can stream a response during an invocation, but each open stream occupies a whole environment. Lambda freezes the environment between invocations, so background work stops.

Active-CPU billing

If the environment stays alive for minutes at a time, what are you paying for? You pay for requests, consumed memory, and the CPU time the environment consumes. For this example, compare connection uptime with CPU time. The environment keeps running but uses only a fraction of a second of CPU.

Choose Generate CPU load, and the app runs a cryptographic hashing loop for about 1.5 seconds. CPU time climbs on the chart, then falls back to near zero when the work stops. When a request waits on a database, downstream API, or AI inference, your code uses little CPU, so the wait costs less than it would under wall-clock billing.

Figure 3: CPU chart mid-spike after choosing Generate CPU load.

For pricing details, read the AWS Lambda pricing page.

Concurrency and scale-out

With Web Functions, each execution environment handles many requests at once, not one at a time. As environments approach their concurrency limit, Lambda adds more environments and removes them again when traffic falls.

You can control how a Web Function scales. maxConcurrencyPerEnvironment sets the maximum number of requests each execution environment handles at once, up to 128, with a default of 64. On the endpoint, maxEnvironments caps the number of execution environments and rateLimit caps requests per second. You can use these two settings to protect downstream dependencies such as databases and external APIs. Both apply in each Region the endpoint serves and maxEnvironments applies to each revision. Requests above a limit receive HTTP 429 and are not queued.

Generate load on the app. Send 300 streaming requests to the /api/events endpoint, with up to 200 open at once, rather than a burst of quick responses.

curl -sN --parallel --parallel-immediate --parallel-max 200 --max-time 90 "https://<ENDPOINT>/api/events?[1-300]"

Figure 4: Concurrency showing increased connections in a single execution environment.

The dashboard updates to reflect the active SSE connections and peak for the currently connected execution environment.

The runtime also writes a per-environment summary record to Amazon CloudWatch Logs every minute with the instance ID, request count, peak concurrency, and CPU time. A CloudWatch Logs Insights query aggregates them across the fleet:

QID=$(aws logs start-query \
    --log-group-name /aws/lambda/web/lambda-web-sse-dashboard \
    --start-time $(($(date +%s)-300)) --end-time $(date +%s) \
    --query-string 'filter type="platform.invokeSummary" | stats count_distinct(record.instanceId) as environments, sum(record.metrics.requests) as totalRequests, max(record.metrics.concurrentRequests) as peakConcurrency, sum(record.metrics.cpuTimeMs)/1000 as totalCpuSeconds' \
    --query queryId --output text) && sleep 6 && aws logs get-query-results --query-id "$QID"

From a sample run of 300 streaming requests with up to 200 open at once, the logs showed the following:

Execution environments used: 5.
Total requests handled: 484.
Peak concurrency on a single environment: 59.
Total CPU consumed across the whole fleet: 4.75 seconds.

One environment peaked at 59 connections, near the default limit of 64. Lambda adds environments before existing ones reach their limit, so this load used five. Event Functions serve one request per environment, so 200 open streams would need up to 200 environments. Across 484 requests, the whole fleet used about 4.8 seconds of CPU.

Each environment reports only its own connections, because Lambda routes each request independently with no session affinity.

Going multi-Region

The same app can serve each caller from the Region with the lowest latency. Because an endpoint’s type and Regions are fixed when you create it, going global means adding a new endpoint:

aws lambda-web deploy --name lambda-web-sse-dashboard --code . --endpoint-name global --endpoint-type MultiRegion --regions us-east-1,eu-west-2

Figure 5: AWS CLI creating a global endpoint.

Lambda replicates your code and revision configuration, including environment variables, to each Region, and centralizes logs in the home Region. In this case, Lambda replicates to Europe (London) Region (eu-west-2). It provides a single global domain URL and uses Amazon Route 53 latency-based routing to send each request to the lowest-latency Region.

Browse to the new global URL which is now in the format: https://<domain>.lambda-web.global.on.aws

Figure 6: The app dashboard showing routing to the local Region.

From my laptop in the United Kingdom, the request routes to eu-west-2 (London) in 16 ms, where the single-Region endpoint in us-east-1 replied in 90 ms. These are single measurements from one location, not a benchmark. Each request goes to the Region with the lowest latency for the caller through the same URL. Lambda manages the DNS routing, so there are no load balancers or Route 53 records to configure.

Authentication and running behind Amazon CloudFront

Each endpoint has an authentication type that you can change without redeploying code.

With ApplicationManaged, the endpoint is public, and your code handles authentication (OAuth, JWT, a session cookie, or Amazon Cognito). Administrators can prevent public endpoints across an organization with a service control policy that uses the lambda:Request/WebFunctionAuthType condition key.

With IamAuth, Lambda validates each request with AWS Signature Version 4 (SigV4) for single-Region endpoints and SigV4a for multi-Region endpoints. Only callers with the lambda:InvokeWebFunctionEndpoint permission reach your app, which suits service-to-service APIs. Endpoints are always reachable over the public internet, and IamAuth controls who can invoke them.

Endpoints use a Lambda-provided domain. For a custom domain, edge caching, and AWS WAF, place Amazon CloudFront in front of the endpoint. Use IamAuth with origin access control so the Lambda domain cannot bypass CloudFront which still passes your application’s own Authorization header through to your code.

Build it with an agent

Because deployment is a single aws lambda-web deploy command and the code is an ordinary HTTP server, you can scaffold, build, and deploy an app from a prompt using AWS skills for coding agents. The Agent Toolkit for AWS includes Web Functions.

Clean up

To remove the default resources that the aws lambda-web deploy command created, see the cleanup instructions in the repo.

Conclusion

AWS Lambda Web Functions bring the serverless operating model to web applications without adapter layers. You write a standard Node.js HTTP server, deploy it with one command or CloudFormation, and get a scalable, pay-per-use HTTPS endpoint with global routing built in. Deploy the sample application from the Serverless Patterns collection along with other Web Functions patterns. Read the Lambda Web Functions documentation to go deeper and the Lambda roadmap.

For more serverless learning resources, visit Serverless Land.

Reducing cross-AZ latency with the AWS Lambda metadata endpoint

Daniel Abib — Fri, 09 Oct 2026 19:41:24 +0000

Customers use AWS Lambda to build applications that connect to stateful, latency-sensitive backends such as Amazon ElastiCache, Amazon Relational Database Service (Amazon RDS), and other services that expose per-Availability Zone (AZ) endpoints. For high availability, Lambda automatically provisions your execution environments across multiple Availability Zones in a Region. That resilience is transparent to your code. It also means that, until now, a function had no way of knowing which Availability Zone it was running in. When a function in one AZ connects to a resource in another AZ, the request crosses the AZ boundary, adding network latency and, in some cases, cross-AZ data transfer costs.

Lambda now exposes Availability Zone metadata through a metadata endpoint in the execution environment. Your function can now discover its AZ ID (for example, use1-az1) with an HTTP request. You can use that information to prefer same-AZ endpoints for downstream services, reduce cross-AZ latency, and implement AZ-aware resilience patterns such as AZ-specific fault injection testing. The feature is available at no additional cost in all commercial AWS Regions where Lambda is available. It works across all runtimes, including custom runtimes and container images, and is compatible with SnapStart, provisioned concurrency, and functions in a virtual private cloud (VPC). For more information about these compatibility details, see Using the Lambda metadata endpoint.

In this post, you will learn how the Lambda metadata endpoint works, how to retrieve the AZ ID using Powertools for AWS Lambda or by calling the endpoint directly, and how to apply the AZ ID to route to same-AZ resources for lower latency.

Why Availability Zone awareness matters

Consider a latency-sensitive read path: an API-backed Lambda function that reads from an ElastiCache cluster with a node in each of three Availability Zones. Lambda places your execution environments across those same AZs for resilience, but the placement is independent of where each cache node lives. Roughly two out of three invocations end up talking to a cache node in a different AZ than the one the function is running in.

A same-AZ round trip typically completes in single-digit milliseconds. A cross-AZ round trip adds latency on top of that, and on a hot read path executed thousands of times per second, those milliseconds compound into higher p99 latency and additional cross-AZ data transfer charges. Before this feature, avoiding cross-AZ traffic from Lambda required custom workarounds. These included embedding AZ hints in environment variables per function version, or inferring locality from IP ranges, none of which were reliable.

The same pattern appears across many of the VPC-based, AZ-scoped services your functions connect to. With Amazon RDS and Amazon Aurora, the writer instance lives in a single AZ, and each read replica lives in an AZ of its own. A read-heavy function that spreads its queries across replicas will, more often than not, reach a replica in a different AZ. Once your function knows its own AZ ID, you can direct reads to the replica that shares it. This keeps the hot query path in-AZ while still failing over to another replica when no same-AZ one is available. For more information, see Configuring Aurora read replica load balancing.

Amazon MemoryDB extends the same idea to a durable primary/replica topology: writes go to the shard’s primary in one AZ, while reads can be served from replicas spread across the others. Consider a high-volume, read-mostly path that tolerates eventual consistency, such as looking up reference data or serving cached lookups. Steering reads to the replica in the function’s AZ keeps the hottest part of the workload in-AZ, while writes continue to route to the primary wherever it lives. Because replicas lag the primary by a small, asynchronous delay, send any read that must reflect its own write directly to the primary rather than to a same-AZ replica.

Container platforms exhibit the same behavior when Lambda calls into them. If your function invokes a service running on Amazon Elastic Kubernetes Service (Amazon EKS) or Amazon Elastic Container Service (Amazon ECS), the pods or tasks behind that service are distributed across AZs, and by default a request can land on any of them. You can combine the function’s AZ ID with topology-aware routing to keep east-west traffic within the AZ. Examples include Kubernetes Topology Aware Routing or an internal load balancer with cross-zone balancing tuned. The same reasoning applies to a function calling a private service fronted by an internal Application Load Balancer or discovered through AWS Cloud Map: knowing the caller’s AZ turns “any healthy target” into “the nearest healthy target.”

The metadata endpoint removes that guesswork. Your function asks the execution environment where it is running, gets back a stable AZ ID, and routes accordingly.

Figure 1: AZ-aware reads and centralized writes with the AWS Lambda metadata endpoint

How the metadata endpoint works

The metadata endpoint is a loopback HTTP API available inside the Lambda execution environment, accessible to both runtimes and extensions. Lambda automatically sets two environment variables in every execution environment:

AWS_LAMBDA_METADATA_API – The metadata server address in the format {ipv4_address}:{port} (for example, 169.254.100.1:9001). For more information about how this address and port are assigned, see Using the Lambda metadata endpoint.
AWS_LAMBDA_METADATA_TOKEN – A unique authentication token for the current execution environment, generated automatically at initialization. You include it in every metadata API request.

You retrieve metadata with an authenticated GET request:

GET http://${AWS_LAMBDA_METADATA_API}/2026-01-15/metadata/execution-environment

The response is a small JSON document:

{
  "AvailabilityZoneID": "use1-az1"
}

The Authorization header must contain a Bearer token (Authorization: Bearer <token>). This token-based authentication provides defense in depth against Server-Side Request Forgery (SSRF) vulnerabilities. Each execution environment receives its own randomly generated token, so a leaked or guessed URL alone is not enough to read metadata.

The response is immutable within an execution environment and returns a Cache-Control: private, max-age=43200, immutable header. We recommend that you cache the response and respect the TTL rather than calling the endpoint on every invocation. For SnapStart functions, the TTL is reduced during initialization so that clients refresh the metadata after restore, since a restored execution environment might come up in a different AZ. For more information, see Lambda SnapStart metadata behavior.

Retrieving the AZ ID with Powertools for AWS Lambda

The most direct way to consume the endpoint is with the Powertools for AWS Lambda metadata utility, available for Python, TypeScript, Java, and .NET. The utility caches the response after the first call and handles SnapStart cache invalidation automatically, so you don’t need to manage tokens, HTTP requests, or TTLs yourself. The following examples show how to use Powertools in each language.

Python

Install the Powertools package:

pip install "aws-lambda-powertools"

Use the metadata utility in your handler:

from aws_lambda_powertools.utilities.metadata import LambdaMetadata, get_lambda_metadata

def handler(event, context):
    metadata = get_lambda_metadata()
    az_id = metadata.availability_zone_id  # e.g., "use1-az1"

    return {"az_id": az_id}

TypeScript

Install the Powertools package:

npm install @aws-lambda-powertools/commons

Use the metadata utility in your handler:

import { getMetadata } from '@aws-lambda-powertools/commons/utils/metadata';

const metadata = await getMetadata();

export const handler = async () => {
  const { AvailabilityZoneID: azId } = metadata;
  return azId;
};

Java

Add the Powertools dependency to your pom.xml:

<dependencies>
    <dependency>
        <groupId>software.amazon.lambda</groupId>
        <artifactId>powertools-lambda-metadata</artifactId>
        <version>2.10.0</version>
    </dependency>
</dependencies>

Use the metadata client in your handler:

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import software.amazon.lambda.powertools.metadata.LambdaMetadata;
import software.amazon.lambda.powertools.metadata.LambdaMetadataClient;

public class App implements RequestHandler<Object, String> {

    @Override
    public String handleRequest(Object input, Context context) {
        LambdaMetadata metadata = LambdaMetadataClient.get();
        String azId = metadata.getAvailabilityZoneId(); // e.g., "use1-az1"

        return "{\"azId\": \"" + azId + "\"}";
    }
}

.NET

Install the Powertools package:

dotnet add package AWS.Lambda.Powertools.Metadata

Use the metadata class in your handler:

using Amazon.Lambda.Core;
using AWS.Lambda.Powertools.Metadata;

public class Function
{
    public string Handler(object input, ILambdaContext context)
    {
        var azId = LambdaMetadata.AvailabilityZoneId;
        return $"Running in AZ: {azId}";
    }
}

Calling the metadata endpoint directly

If you use a custom runtime, a container image, or prefer not to add a dependency, you can call the endpoint directly using the environment variables Lambda sets for you. The following example uses curl and jq to read the AZ ID:

# Variables are automatically set by Lambda
METADATA_ENDPOINT="http://${AWS_LAMBDA_METADATA_API}/2026-01-15/metadata/execution-environment"

# Make the request
RESPONSE=$(curl -s -w "\n%{http_code}" -H "Authorization: Bearer ${AWS_LAMBDA_METADATA_TOKEN}" "$METADATA_ENDPOINT")

HTTP_CODE=$(echo "$RESPONSE" | tail -1)

if [ "$HTTP_CODE" != "200" ]; then
  echo "Metadata request failed with status $HTTP_CODE" >&2
  exit 1
fi

# Parse the AZ ID
AZ_ID=$(echo "$RESPONSE" | sed '$d' | jq -r '.AvailabilityZoneID')

echo "Function is running in AZ ID: $AZ_ID"

Notice the configuration:

AWS_LAMBDA_METADATA_API: Lambda sets this to the metadata server address, so you never hardcode the host or port.
Authorization: Bearer ${AWS_LAMBDA_METADATA_TOKEN}: The token is required on every request. A missing or invalid token returns 401 Unauthorized.
GET only: The endpoint accepts GET requests. Any other method returns 405 Method Not Allowed.

When you call the endpoint directly, retrieve the AZ ID once during initialization (outside the handler) and cache it for the life of the execution environment, rather than calling on every invocation. If your function uses SnapStart, refresh the value after restore, because the restored environment might run in a different AZ.

Understanding Availability Zone IDs

The endpoint returns an AZ ID (for example, use1-az1), not an AZ name (for example, us-east-1a). Before you use it, it helps to read the ID itself. An AZ ID follows the pattern <region-code>-az<number>:

use1 is the shortened Region code, in this case US East 1 (us-east-1). Other Regions follow the same convention, such as usw2 for US West (Oregon) and euw1 for Europe (Ireland).
az1 identifies the individual Availability Zone within that Region. So use1-az1 reads as “Availability Zone az1 in us-east-1.”

This distinction matters. AZ IDs always refer to the same physical location across all AWS accounts, while AZ names might map to different physical infrastructure in each account in certain Regions. In other words, the us-east-1a name in your account and the us-east-1a name in another account can point to two different physical Availability Zones, but use1-az1 is the same physical zone in every account. There is no fixed mapping between a name like us-east-1a and an ID like use1-az1. AWS assigns names independently per account to balance resources across zones. That’s exactly why the endpoint returns the ID: comparing AZ IDs gives you a consistent, account-independent result about whether your function and a downstream resource share a physical Availability Zone. For more information, see AZ IDs for cross-account consistency.

If you need the AZ name (for example, for logging or for an API that expects a name), convert the AZ ID using the Amazon Elastic Compute Cloud (Amazon EC2) DescribeAvailabilityZones API. To call it, add the ec2:DescribeAvailabilityZones permission to your function’s execution role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ec2:DescribeAvailabilityZones",
            "Resource": "*"
        }
    ]
}

Routing to same-AZ resources

Retrieving the AZ ID is only useful if you act on it. The following Python example shows the common pattern illustrated in Figure 1: look up the AZ ID once at initialization, select the connection endpoint that lives in the same AZ, and fall back gracefully when there is no same-AZ match.

from aws_lambda_powertools.utilities.metadata import LambdaMetadata, get_lambda_metadata

# Map each cache node to the AZ ID it lives in.
# Populate this from configuration (for example, environment variables or Parameter Store).
CACHE_NODES_BY_AZ = {
    "use1-az1": "cache-node-1.example.use1-az1.cache.amazonaws.com",
    "use1-az2": "cache-node-2.example.use1-az2.cache.amazonaws.com",
    "use1-az3": "cache-node-3.example.use1-az3.cache.amazonaws.com",
}

# Read the AZ ID once during initialization (cached across warm invocations).
AZ_ID = get_lambda_metadata().availability_zone_id

# Prefer the same-AZ node; fall back to any node if there isn't one.
CACHE_ENDPOINT = CACHE_NODES_BY_AZ.get(AZ_ID) or next(iter(CACHE_NODES_BY_AZ.values()))


def handler(event, context):
    # Use CACHE_ENDPOINT for your read path. When AZ_ID matches the node's AZ,
    # the request stays within the Availability Zone.
    return {"az_id": AZ_ID, "cache_endpoint": CACHE_ENDPOINT}

Notice the design choices:

Resolve at initialization: AZ_ID and CACHE_ENDPOINT are computed once per execution environment, not per invocation, so the metadata lookup and endpoint selection don’t add latency to the request path.
Always provide a fallback: If there is no node in your function’s AZ (for example, the service isn’t deployed in every AZ the function runs in), fall back to any available endpoint. Correctness must not depend on a same-AZ match being present.
Compare AZ IDs, not names: Because the endpoint returns AZ IDs, and services like ElastiCache and RDS expose the AZ of their nodes, you can compare the two directly for a reliable same-AZ decision.

Beyond latency-aware routing, the same AZ ID unlocks resilience testing. You can, for example, inject faults only in functions running in a specific AZ to validate that your application degrades gracefully during a simulated AZ impairment, without affecting traffic served from other AZs.

Best practices

Based on the endpoint’s behavior, keep these recommendations in mind:

Cache the metadata: The response is immutable for the life of an execution environment. Read it once at initialization and reuse it. If you call the endpoint directly, respect the Cache-Control TTL rather than requesting on every invocation.
Handle SnapStart correctly: A restored SnapStart environment might resume in a different AZ than the one it was snapshotted in. Refresh the AZ ID after restore. Powertools handles this invalidation for you automatically.
Ignore unknown fields: Additional fields might be added to the response in the future. Parse defensively and don’t fail if new fields appear.
Degrade gracefully: Treat same-AZ routing as an optimization, not a requirement. Always keep a working fallback path so your function stays correct when no same-AZ resource is available.
Keep the token server-side: The AWS_LAMBDA_METADATA_TOKEN is scoped to a single execution environment. Never log it or forward it outside the function.

Conclusion

In this post, you learned how the Lambda metadata endpoint helps your functions discover the Availability Zone they run in. With a single authenticated HTTP request, or one call through Powertools for AWS Lambda, your function can retrieve a stable AZ ID and use it to route to same-AZ resources, cutting cross-AZ latency and data transfer on hot paths, and to build AZ-aware resilience patterns like targeted fault injection. The feature works across all runtimes, integrates with SnapStart, provisioned concurrency, and VPC-enabled functions, and is available at no additional cost in all commercial AWS Regions where Lambda is available.

The key takeaway: if your Lambda function reads from or writes to AZ-scoped resources such as ElastiCache or RDS on a latency-sensitive path, resolve the AZ ID at initialization, prefer the same-AZ endpoint, and always keep a fallback. You get lower, more predictable latency with no change to how Lambda manages high availability for you.

To get started, see Using the Lambda metadata endpoint in the AWS Lambda Developer Guide.

Additional resources

Scaling RL rollouts with Firecracker on Amazon EC2 metal

Karsten Ploesser — Fri, 09 Oct 2026 14:38:27 +0000

Modern reinforcement learning (RL) systems rely on thousands of rollout environments running in parallel. For large language model (LLM) training using Verifiable Rewards (RLVR) each rollout environment runs in a tightly isolated sandbox so state never leaks between environments.

The faster you can recycle sandboxes, the more training steps you can run per hour. That means higher rollout density per host which directly lowers your cost per step.

But density is a tuning knob with two failure modes. Push it too high and you hit a tail latency cliff. Play it too safe and you overprovision, leaving money on the table. The sweet spot depends on your instance type, workload profile, and how your training algorithm amplifies stragglers.

This post shows how to find that setting. We walk through the tunables that control density on Amazon EC2 metal. We then introduce labsweep, a measurement harness for mapping your latency and density curve. Throughout the post, we draw on lessons learned from production deployments with customers. The post is written for readers familiar with Kernel-based Virtual Machine (KVM), Firecracker, and container orchestration.

Background: EC2 metal, Nitro, and Firecracker

The AWS Nitro System offloads networking, storage, and security to dedicated hardware. Amazon EC2 instances expose a stable CPU topology you can inspect and pin workloads against. On Amazon EC2 metal, applications run directly on the underlying hardware.

Firecracker is a lightweight virtual machine monitor (VMM) designed for high density workloads. Each microVM boots quickly, runs its own Linux kernel, and has a small memory footprint.

Running Firecracker on Amazon EC2 metal combines direct access to underlying hardware with lightweight virtualization to support high density sandboxing. The key question is how far you can increase density before tail latency hurts the RL training loop.

Solution overview

The following diagram shows the reference architecture of a production RL sandbox node. This architecture serves as the foundation for the rest of the post. An Amazon EC2 metal instance runs a host agent that manages Firecracker microVMs, CPU pinning, non-uniform memory access (NUMA)-aware memory placement, and the warm pool used for dispatch.

The host agent runs as a DaemonSet on each Amazon EC2 metal node in the Kubernetes cluster. At startup, it discovers the CPU topology, creates per-microVM cgroup hierarchies, and orchestrates Firecracker jailer processes throughout the microVM lifecycle.

To reduce dispatch latency, the system maintains a warm pool of pre-booted microVMs. The host agent applies CPU pinning, NUMA-aware memory placement, and memory sizing on a per-microVM basis. labsweep identifies the combination of these settings that best balances dispatch latency, density, and tail latency for a given workload.

Prerequisites

Amazon EC2 metal instance (for example, we tested m8i.metal-48xl, m8a.metal-48xl, and m8g.metal-48xl).
Linux kernel 6.2+ (required for cgroup partition isolated mode).
Firecracker 1.7+ and the Firecracker jailer.
Go 1.22+ (to build labsweep).
AWS CDK v2 with Python 3.11+ (to deploy the accompanying stack).
Guest rootfs image (a builder script is included in the accompanying repository).

Scope note: this post focuses on CPU and memory placement for Firecracker microVMs. Other system variables including storage architecture (Amazon Elastic Block Store (Amazon EBS), local SSD) and network latency to the LLM endpoint were held constant to isolate the effects of CPU and memory placement.

Density tunables on EC2 metal

Your density ceiling depends on tunables whose availability varies by instance type. Not every knob exists on every instance type, and where a knob does exist its underlying mechanism can differ. The following table shows which tunables are available across the three metal instance types this post considers: m8i, m8a, and m8g. You can derive availability and mechanism from each instance type’s published CPU topology. The measured latency and throughput results later in the post currently cover m8i and m8a, with m8g results to follow.

Tunable	m8i.metal-48xl	m8a.metal-48xl	m8g.metal-48xl	What it does
Core pinning (`cpuset.cpus`)	✓	✓	✓	Dedicates a physical core to each microVM. Eliminates scheduler-induced tail latency
NUMA node confinement (`cpuset.mems`)	✓	✓	✓ (2 NUMA nodes on the 48xl)	Keeps memory allocations local. The remote-node latency penalty varies by instance type. Check the NUMA distance matrix for your instance before pinning
SMT state (smt/control)	✓ (on/off)	N/A (no SMT)	N/A (no SMT)	Doubles vCPUs when on. Workload-dependent: helps I/O-bound sandboxes, hurts compute-bound loops
L3 cache domain pinning	✓ (place workloads within NUMA-aligned cache domains)	✓ (24 domains, 8 cores each)	✓ (single shared L3 per socket. Pin on NUMA/socket boundary)	Reduces cache contention for working sets above 4 MiB per microVM
`cgroup` partition (isolated)	✓	✓	✓	Removes kernel workqueue interference from pinned cores. Requires kernel 6.2+
Guest memory size	✓	✓	✓	Primary cost driver. Determines how much of your memory-to-core budget each guest consumes; M-series gives 4 GiB per vCPU

Key observation: different instance types expose different optimization opportunities. Among the instances evaluated, simultaneous multithreading (SMT) is available on m8i but not on m8a or m8g. Cache-local placement also differs by instance type: some present many discrete L3 domains, some a shared L3 that is effectively partitioned along NUMA boundaries, and some a single mesh-shared L3 per socket where the meaningful locality boundary is the NUMA/socket rather than a cache-domain count. Each instance type’s L3 layout is described in its own row of the table above and in its published CPU topology. As a result, the set of tunables and their impact on density vary by instance type. This is why measuring your own configuration matters more than following generic engineering best practice lists.

To make this concrete, here is what configuring one tunable looks like in practice: a cpuset pin that dedicates a single physical core to one microVM.

# Create an isolated cgroup for one microVM on core 12
mkdir -p /sys/fs/cgroup/microvm/vm-012
echo "12" > /sys/fs/cgroup/microvm/vm-012/cpuset.cpus
echo "0" > /sys/fs/cgroup/microvm/vm-012/cpuset.mems
# Launch the jailer into this cgroup
firecracker-jailer --id vm-012 \
  --exec-file /usr/bin/firecracker \
  --parent-cgroup microvm/vm-012 \
  --uid 1000 --gid 1000

From tunables to measurement: labsweep

These tunables interact in ways that depend on your workload and instance type. You cannot predict their combined effect from the specification sheet alone. You must measure the resulting system behavior.

labsweep is the test harness we built to do exactly that. It is a Go tool that:

Discovers your host topology through sysfs (cores, NUMA nodes, L3 domains).
Plans a matrix of configurations and densities.
Launches jailed Firecracker microVMs at each density with bounded concurrency.
Runs the workload in every guest concurrently and records per-microVM latency.
Outputs raw samples, percentile aggregates, and variance data with full host provenance.

Methodology note: each experiment is performed at fixed microVM density. Dynamic churn is outside the scope of this evaluation.

The following command launches a typical labsweep run on an m8a.metal-48xl:

labsweep -configs A,B \
  -densities 90,120,150,186 \
  -repeats 5 \
  -resident-kib 131072 \
  -steps 14 \
  -write-kib 4096

Each run produces percentile latencies for every configuration, variance across repeated trials, and raw samples for recomputing any statistics. Results include full provenance (instance type, CPU model, kernel version, and Firecracker versions) making runs reproducible and comparable over time. If a microVM fails to respond, the harness reports failed samples instead of silently omitting them. This prevents failures from biasing the reported results.

The accompanying repository includes everything needed to reproduce these experiments: the labsweep harness, guest agent, rootfs builder, and the CDK application that deploys the complete stack on an Amazon EC2 metal instance (github.com/aws-samples/sample-firecracker-microvm-density-lab).

Results on representative workloads

Using labsweep we benchmarked workloads representative of the execution patterns in production RL systems: short-lived verification tasks (run a test suite, check the result), build-heavy tasks (compile and link), full 14-turn replayed RL episodes, and a synthetic compute loop as a control. Across all evaluated instance types, we collected more than 100,000 samples across 500+ configurations.

The 1.0x ratio rule

Tail latency remains flat as density increases until the workload reaches one microVM per vCPU. Beyond that threshold, p99 latency rises sharply over the next 15% increase in density before reaching a new plateau while median latency remains essentially unchanged. Expressing density as microVMs per vCPU rather than as an absolute microVM count makes this threshold portable across instance types. The latency cliff occurs at the same 1.0x density threshold regardless of instance size. Only the maximum number of microVMs scales with available vCPUs. The complete per-density measurements supporting this figure are provided in the appendix.

Core pinning reduces the tail latency for most real workloads

On the synthetic compute loop, core pinning made no measurable difference at equal density. On real sandbox workloads however, core pinning measurably tightened the tail: it reduced the p99/p50 ratio toward roughly 1.1x. The p99/p50 ratio measures tail spread by dividing the 99th-percentile latency by the median. Lower values indicate a tighter, more predictable latency distribution. The advantage grows with density, so pinning matters most where you are pushing the host hardest.

Why does this matter when choosing the right density? If you run Group Relative Policy Optimization (GRPO), every gradient step waits for the slowest rollout in the group. At a group size G=64, a p99 event affects ~47% of gradient steps. The effect becomes more pronounced as deployment density increases.

Density / cores	G=8	G=16	G=64
0.94x	1.01x	1.01x	1.04x
1.04x	1.01x	1.03x	1.13x
1.15x	1.01x	1.04x	1.32x

Past the 1.0x ceiling the amplification climbs steadily, and by the time you are well above it a p99 event can dominate the group’s completion time. Median-focused dashboards do not surface this because the median stays flat. Core pinning keeps the p99/p50 ratio low across density, holding tail latency amplification far closer to flat than an unpinned configuration even at a group size of 64. Oversubscription can appear to provide spare capacity until tail latency amplification dominates training time. Set the density to the highest value where the amplification factor remains flat.

SMT depends on your workload

The optimal SMT setting depends on the workload:

I/O-bound workloads: SMT-on provides higher throughput. At high density it delivers 1.49x throughput because the vCPU count stays above the 1.0x ratio.
Compute-bound workloads: SMT-off produces more predictable tail latency, with a markedly tighter p99 spread than SMT-on.

Most RL sandbox workloads are I/O-bound, spending much of their time waiting on file operations, process creation, or git operations. For these workloads, keep SMT enabled. With SMT enabled, you can place more microVMs before reaching the 1.0x ratio, for higher density without crossing the latency cliff.

Configuration matters as much as hardware

Hardware selection is only one factor affecting performance. Configuration choices produce measurable differences. To isolate the effect of configuration, we compare identical workloads on the same m8i instance with SMT enabled and disabled. The results show that changing a single configuration parameter measurably affects peak throughput per host.

The table below shows how enabling and disabling SMT affects peak throughput across a representative set of RL workloads, expressed as the SMT-on to SMT-off throughput ratio. The workloads were selected to represent the major execution patterns encountered in RL systems. Collectively, they cover short-lived verification tasks, interactive sessions, and full RL episodes.

Workload	Throughput gain from SMT
Verification task	1.14x
Build task	1.13x
Replayed RL episode	1.17x

The optimal SMT configuration depends on the density at which the workload is intended to run. With SMT enabled, the host exposes more vCPUs, which supports higher microVM densities before reaching the 1.0x microVM per vCPU threshold. This increases throughput at higher densities by delaying the onset of the tail latency cliff. At densities at or below one microVM per vCPU, SMT disabled delivers slightly better per-task performance and tighter tail latency. These results demonstrate that no single SMT configuration is universally optimal. The best choice depends on the target operating density. You should therefore determine the optimal configuration through measurement rather than assuming it in advance.

When memory becomes the limiting resource

We ran all experiments on M-series metal instances, which provide 4 GiB of memory per vCPU. At this memory-to-vCPU ratio, every workload we tested remained vCPU-bound. We exhausted vCPUs before memory. If your workloads require a higher memory-to-vCPU ratio, the bottleneck shifts from vCPUs to memory. In that case, a memory-optimized instance family may be the better choice. It provides more memory per vCPU but fewer vCPUs per host. As a result, the maximum sandbox density is lower. Consider this trade-off when selecting an instance family.

Reproduce these results on your own workloads

You do not need to take these numbers on faith. With labsweep, you can reproduce them on your own workloads. Clone the accompanying repository and deploy the CDK stack on the instance type you plan to run in production. Build a rootfs from your real workload. Sweep across densities below and above your expected level so you can see where expected performance drops off (the -densities flag in the preceding example is a reasonable start). Repeat each configuration a few times using the -repeats flag so you can see the variance, then read the p99/p50 ratio and the amplification table for your group size. Tune your system based on the curve you measure for your own workloads. That is more reliable than tuning to synthetic benchmarks.

Clean up

If you deployed the accompanying CDK stack for labsweep, tear down these resources when you are done:

Delete the CDK stack (cdk destroy).
Terminate any running Amazon EC2 metal instances.
Remove the guest rootfs images from Amazon Simple Storage Service (Amazon S3) (if uploaded).
Delete the Amazon CloudWatch log groups created by the harness.

Conclusion

Sandbox density on Amazon EC2 metal is determined by a small set of tunables whose optimal values depend on both the workload and the underlying instance type. Because those values vary across workloads, there is no single density setting or configuration that is optimal for every deployment. Instance type specifications give you a starting point but only measurement can identify the right operating point for your workload. labsweep is designed to make that measurement straightforward so you can optimize against your own production workloads.

Blog resources

Appendix

Full per-density measurements behind the tail-latency chart in “The 1.0x ratio rule.” Conditions: m8i, SMT off, 96 physical cores, config A (unpinned). Amplification columns give the GRPO group-completion factor (group max over rollout p50) at group sizes G=8, G=16, and G=64.

Density	Density / cores	Rollout p50	Rollout p99	G=8 amp	G=16 amp	G=64 amp
45	0.47x	7,768 us	7,857 us	1.01	1.01	1.01
90	0.94x	7,740 us	8,035 us	1.01	1.01	1.04
100	1.04x	7,740 us	8,601 us	1.01	1.03	1.13
110	1.15x	7,737 us	9,909 us	1.01	1.04	1.32
135	1.44x	7,743 us	15,601 us	1.03	1.20	2.01
186	1.94x	7,744 us	15,697 us	1.22	1.56	2.03

Get started with AWS Lambda SnapStart for container images

Sachin Saini — Tue, 06 Oct 2026 16:59:51 +0000

AWS Lambda recently launched SnapStart for container image functions, reducing startup times from several seconds to as low as sub-second. Customers deploy Lambda functions with container images to align with their organization’s container-based deployment standards, or to package larger dependencies up to 10 GB. However, larger container images can experience startup times of several seconds as Lambda downloads image layers and initializes the runtime and application code. SnapStart addresses this by taking a snapshot of the initialized execution environment during function deployment, caching it, and resuming from it on invocation, instead of initializing from scratch.

In November 2022, Lambda launched SnapStart for Java, reducing startup latency by up to 10x. In November 2024, SnapStart expanded to Python and .NET managed runtimes, reducing cold start latency to as low as sub-second. These launches helped developers achieve faster startup time, but only for .zip file archives. By extending SnapStart support for container image functions, customers can improve startup times for latency-sensitive workloads such as ML inference and interactive APIs.

This post covers how SnapStart for container images works, how to enable it, and best practices and considerations for your workloads.

How SnapStart for container images works

When you deploy a Lambda function, you choose one of two deployment models: a .zip file archive or a container image. With managed runtimes we already support SnapStart for Python, .NET, and Java functions, and now we have extended this capability to container images. When you invoke the deployed function for the first time, or when a burst of traffic arrives, Lambda checks if there is an available execution environment. If none are available, Lambda creates a new one.

For container image functions, this means downloading and extracting image layers, bootstrapping the runtime, and running your initialization code. This process can take several seconds, which users experience as a cold start.

When you enable SnapStart and publish a function version, Lambda proactively creates an execution environment, initializes your function, and then takes a Firecracker microVM snapshot of the full memory and disk state. This snapshot is encrypted and cached for low latency access and remains cached until you delete the corresponding function version.

On invocation, Lambda resumes from the cached snapshot rather than initializing from scratch. With SnapStart, initialization happens once at deployment time rather than on every invocation, replacing the initialization phase with a faster restore phase that speeds up startup time to as low as sub-second.

Figure 1: SnapStart for container images lifecycle, from image push to snapshot restore at invocation

Supported base images

For managed runtimes, you provide your code, and Lambda handles the runtime environment. For container image functions, you package your own runtime environment using Lambda-vended base images or your own custom base images. SnapStart for container images supports the following Lambda base images at launch. To learn about supported base images per runtime, see the Lambda SnapStart developer guide.

Runtime	Base Image	Architecture
Java 11 and later	public.ecr.aws/lambda/java:11, :17, :21, :25	x86_64, arm64
Python 3.12 and later	public.ecr.aws/lambda/python:3.12, :3.13, :3.14	x86_64, arm64
.NET 8 and later	public.ecr.aws/lambda/dotnet:8, :9, :10	x86_64, arm64

Regardless of which base image you use, there is an important consideration to understand before enabling SnapStart. When Lambda restores a function from a snapshot, any state captured during initialization is shared across all execution environments restored from that snapshot. This includes random number generators, unique IDs, cached credentials, and database connections. Your function needs to restore a unique state after each resume to avoid reusing the same random seed or an expired database connection across invocations. For more details on these considerations, see SnapStart uniqueness documentation.

To help you restore uniqueness for your application, Lambda provides runtime hooks for managed runtimes that let you run your own code at specific points in the snapshot-and-restore cycle. With SnapStart for container images, we are extending these same hooks to container image functions.

If your function uses one of the preceding Lambda-managed base images, Lambda offers runtime hooks for you to run your code (for example, to restore database connections). Lambda supports these runtime hooks for Java through CRaC, for Python through snapshot_restore_py, and for .NET through SnapshotRestore. To register your own before-snapshot and after-restore hooks for these runtimes, see SnapStart runtime hooks documentation.

If your function uses a base image without native SnapStart support (for example, a Node.js or Ruby base image), a custom Runtime Interface Client, or a custom runtime built on provided.al2023, review the documentation on implementing SnapStart hooks for container image functions and choose one of the following options.

Option 1: Implement SnapStart runtime hooks. If you need to run custom logic when your function is snapshotted or restored, implement the /restore/next API in your container image as described in SnapStart runtime hooks documentation.
Option 2: Opt in without hooks. If you do not need before-snapshot or after-restore hooks, add the following label to your Dockerfile (LABEL com.amazonaws.lambda.feature.snapstart=”Allow”)

For a reference implementation of SnapStart runtime hooks integration in a custom runtime, refer to the Lambda Rust Runtime repository, including its examples for both event-driven and HTTP functions to see how SnapStart and runtime hooks are implemented.

Getting started

To get started, you can use the AWS Management Console, AWS Command Line Interface (AWS CLI), AWS Software Development Kits (AWS SDKs), AWS CloudFormation, AWS Serverless Application Model (AWS SAM), or AWS Cloud Development Kit (AWS CDK) to activate, update, and delete SnapStart for container images. You can also use the Agent Toolkit for AWS to enable SnapStart as part of your agent-based workflows.

To use SnapStart with a container image function, your function must be deployed with PackageType Image, and the image must be pushed to an Amazon Elastic Container Registry (Amazon ECR) repository in the same AWS Region as your function. The examples in this section use a supported Lambda base image and us-east-1 as the AWS Region. Either set your default region or add –region to each command.

Activating SnapStart on an existing function

If you already have a container image function using a supported base image, activate SnapStart by updating the function configuration and publishing a version:

aws lambda update-function-configuration \
    --function-name my-function \
    --snap-start ApplyOn=PublishedVersions

aws lambda publish-version \
    --function-name my-function

Creating a new function with SnapStart

Create an Amazon ECR repository:

aws ecr create-repository --repository-name my-function-repo

Start with a Dockerfile using a supported base image. Here is an example for Python:

FROM public.ecr.aws/lambda/python:3.12
COPY requirements.txt ${LAMBDA_TASK_ROOT}
RUN pip install -r requirements.txt
COPY lambda_function.py ${LAMBDA_TASK_ROOT}
CMD ["lambda_function.handler"]

Build, push to Amazon ECR, and create the function with SnapStart in a single flow:

# Build the container image locally. --platform pins the CPU architecture
# (must match --architectures on create-function). --provenance=false is
# required because BuildKit otherwise wraps the image in an OCI image index
# (manifest list) to attach a provenance attestation, and Lambda only accepts
# a single image manifest.
docker buildx build --platform linux/amd64 --provenance=false -t my-function-repo:latest .

aws ecr get-login-password | \
docker login --username AWS --password-stdin 111122223333.dkr.ecr.us-east-1.amazonaws.com
docker tag my-function-repo:latest 111122223333.dkr.ecr.us-east-1.amazonaws.com/my-function-repo:latest
docker push 111122223333.dkr.ecr.us-east-1.amazonaws.com/my-function-repo:latest

aws lambda create-function \
    --function-name my-function \
    --package-type Image \
    --code ImageUri=111122223333.dkr.ecr.us-east-1.amazonaws.com/my-function-repo:latest \
    --role arn:aws:iam::111122223333:role/lambda-execution-role \
    --architectures x86_64 \
    --snap-start ApplyOn=PublishedVersions

# Once the function is active, publish the version
aws lambda publish-version \
    --function-name my-function

Verifying SnapStart is active

After publishing, check the function version configuration:

aws lambda get-function-configuration \
    --function-name my-function:1

When SnapStart is active, the response shows:

{
    "SnapStart": {
        "ApplyOn": "PublishedVersions",
        "OptimizationStatus": "On"
    },
    "State": "Active"
}

Once the state is Active, invoke the published version:

aws lambda invoke \
    --function-name my-function:1 \
    --cli-binary-format raw-in-base64-out \
    --payload '{"key": "value"}' \
    response.json

Note that SnapStart applies only to published versions, not $LATEST. You must invoke a specific version number to use SnapStart.

Best practices and considerations

As SnapStart resumes your function from a point-in-time snapshot, here are some best practices to consider for resources initialized at snapshot time that may become stale when your function resumes, such as network connections, credentials, and unique IDs.

Performance tuning: Preload dependencies and initialize resources in your init code rather than the handler. This moves heavy setup into the snapshot, where it’s captured once and reused across restores. For guidance on optimizing function startup latency, see the performance tuning section of the Lambda developer guide.
Network connections: Since snapshots can persist over an extended period, connections established during initialization may no longer be valid after restore. Network connections established through an AWS SDK resume automatically. For connections your code manages directly, such as database connections, use after-restore hooks to re-establish them as described in the networking best practices section of the Lambda developer guide.
Uniqueness: Avoid generating unique IDs, secrets, or random values during initialization, because these are shared across all execution environments restored from the same snapshot. Instead, generate them in the function handler or use after-restore hooks. Software that always gets random numbers from the operating system (for example, from /dev/random or /dev/urandom) does not need any updates to maintain randomness. Lambda always reseeds /dev/random and /dev/urandom when restoring a snapshot, so random numbers are not repeated even when multiple execution environments resume from the same snapshot. For more details on managing unique state across restores, see the handling uniqueness documentation for Lambda SnapStart.
Ephemeral data: Cached credentials, timestamps, or other time-sensitive data captured during initialization may be stale by the time your function resumes from a snapshot. Use after-restore hooks to verify freshness and refresh any expired state before processing requests.
Monitoring: Amazon CloudWatch logs include a RESTORE_REPORT line showing the time spent restoring the snapshot. The REPORT line includes Restore Duration, Billed Restore Duration, Duration, and Billed Duration. Note that InitDuration does not appear for SnapStart invocations because the function resumes from a snapshot rather than initializing. You can also use AWS X-Ray active tracing to capture real-time telemetry for extensions through the Telemetry API, and measure end-to-end latency with Amazon API Gateway and Lambda function URL metrics. To learn more about available monitoring options, visit the Lambda documentation on monitoring for SnapStart.

Limitations: Provisioned concurrency, Amazon Elastic File System (Amazon EFS), Amazon Simple Storage Service (Amazon S3) Files and ephemeral storage greater than 512 MB are not supported with SnapStart. Maximum container image size remains 10 GB. If you need any of the currently unsupported features, please reach out to us on the Lambda Roadmap GitHub page.

Pricing and availability

SnapStart for container images uses the same pricing dimensions as SnapStart for Python 3.12+ and .NET 8+. You pay for the cost of caching a snapshot per function version that you publish with SnapStart enabled, and the cost of restore each time a function is restored from a snapshot. To reduce your caching costs, delete unused function versions. Refer to the Lambda pricing page for more details.

Lambda SnapStart for container images is available in all commercial AWS Regions except Asia Pacific (New Zealand) and Asia Pacific (Taipei). For the latest list of supported Regions, see the Lambda SnapStart supported regions.

Cleanup

To avoid ongoing charges for the resources created in this walkthrough, delete them when you are done. Deleting a Lambda function version removes the associated cached snapshot and stops any related snapshot caching charges.

Delete the Lambda function. This removes all published versions and their cached snapshots:

aws lambda delete-function \
    --function-name my-function

Delete the container image and the Amazon ECR repository:

aws ecr delete-repository \
    --repository-name my-function-repo \
    --force

If you created an IAM execution role only for this walkthrough, delete it as well. For details, see deleting an IAM role.

Conclusion

In this post, we described how Lambda SnapStart for container images reduces cold start latency by snapshotting the initialized execution environment and resuming from it on subsequent invocations. We covered how SnapStart works, how to get started based on your base image type, best practices and considerations, and pricing and availability. We look forward to hearing from you about the future capabilities you need for SnapStart, on our Lambda Roadmap GitHub page.

Try SnapStart on your container image functions today. To learn more, visit the Lambda SnapStart documentation. For more serverless learning resources, visit Serverless Land.

Introducing EC2 AMI Tag Sharing: Share EC2 tags across AWS accounts

Gena Gizzi — Mon, 05 Oct 2026 19:05:04 +0000

Tags are an important and versatile tool on AWS. FinOps teams track costs with them, security teams enforce compliance, and governance teams gate access through attribute-based access control (ABAC).

Tags become especially important as infrastructure management grows more complex across multiple accounts in an organization. Because compute is a common shared resource, many organizations tag their AMIs to track approval status, OS versions, and team. However, when you share an AMI with another AWS account, the tags you attached to it stay behind. The other account sees the image, but tag metadata used to describe it could not be shared.

Until today, the only way to solve this was to build a custom workflow to copy tags into every account the AMI is shared with. Typically created with services such as Amazon Simple Notification Service (Amazon SNS) and AWS Lambda, this caused operational burden.

Today, we’re introducing EC2 Tag Sharing, a new feature that AMI owners can use to share tags with any account that has access to the image. Whether an AMI is shared with an account, across an organization, or made public, a shared tag is automatically shared alongside the image by adding the ec2:SharedTag/ prefix to the tag key. Updates also propagate automatically without any need for custom replication workflows. In this post, we walk through how tag sharing works, demonstrate a common use case, and cover considerations and best practices.

How it works

Any tag whose key starts with ec2:SharedTag/ is visible to all AWS accounts the AMI is shared with, and also when the AMI is shared publicly. Tags without the prefix remain private to the account that created them, exactly as they do today.

Only the resource owner can create, modify, or delete tags with the ec2:SharedTag/ prefix. Accounts you share the AMI with can view these tags but cannot change them. They can continue to add their own private tags to a shared AMI.

Shared tags count against the resource owner’s 50-tag-per-resource quota for both shared and private tags. They do not count against the quota of accounts you share the image with. When you copy an AMI that has shared tags, Amazon Elastic Compute Cloud (Amazon EC2) copies the shared tags to the new image when the --copy-image-tags parameter is included. At launch, tag sharing is supported specifically for AMI resources.

Creating a shared tag (CLI)

To share a tag, create it with the ec2:SharedTag/ prefix on the key name. For example, to share a tag indicating the AMI’s approval status:

aws ec2 create-tags \
    --resources ami-0abcdef1234567890 \
    --tags Key=ec2:SharedTag/status,Value=approved

You can add multiple shared tags alongside your private tags. In the following example, status and os-version are shared with other accounts. The team tag remains private to the owner.

aws ec2 create-tags \
    --resources ami-0abcdef1234567890 \
    --tags \
    Key=ec2:SharedTag/status,Value=approved \
    Key=ec2:SharedTag/os-version,Value=al2023-2026.09 \
    Key=team,Value=platform-engineering

You can also apply shared tags when you create an AMI from the start, using the --tag-specifications parameter. For example:

aws ec2 create-image \
    --instance-id i-1234567890abcdef0 \
    --name "My server" \
    --tag-specifications "ResourceType=image,Tags=[{Key=ec2:SharedTag/publicTag,Value=text}]"

Listing all resources with shared tags (CLI)

To see which of your AMIs have shared tags, you can run the following CLI command:

aws ec2 describe-tags \
    --filters "Name=key,Values=ec2:SharedTag/*"

And get an output such as:

{
    "Tags": [
        {
            "Key": "ec2:SharedTag/status",
            "ResourceId": "ami-0abcdef1234567890",
            "ResourceType": "image",
            "Value": "approved"
        },
        {
            "Key": "ec2:SharedTag/os-version",
            "ResourceId": "ami-0abcdef1234567890",
            "ResourceType": "image",
            "Value": "al2023-2026.09"
        }
    ]
}

You can also receive a similar output using the describe-images command as shown in the following:

aws ec2 describe-images --filters "Name=tag-key,Values=ec2:SharedTag/*"

To stop sharing a specific tag, delete the tag with the ec2:SharedTag/ prefix:

# Remove the shared tag
aws ec2 delete-tags \
    --resources ami-0abcdef1234567890 \
    --tags Key=ec2:SharedTag/status

If you want to keep the tag private, recreate it without the prefix:

# Optionally, recreate as a private tag
aws ec2 create-tags \
    --resources ami-0abcdef1234567890 \
    --tags Key=status,Value=approved

Viewing shared tags as a recipient

When an AMI is shared with your account, you can see the shared tags alongside any tags you’ve added yourself. Figure 1 shows an example of an AMI shared from the original account with both private and shared tags. Figure 2 shows that AMI in the recipient account. The ec2:SharedTag/ prefix in the key name makes it clear which tags came from the owner.

Figure 1: AMI tags in the originating account, showing both private and shared tags

Figure 2: The same AMI in the recipient account, where only the shared tags are visible

As you can see in the preceding figures, the shared tags are visible across both accounts. However, the private tags created in the original account are only visible in the original account.

A common use case: Golden AMIs

Many organizations maintain a central build factory account that produces hardened, security-approved AMIs. These golden images can be shared with hundreds or thousands of workload accounts across an organization. Tags like status=approved or patch-date=2026-09-18 are commonly used by recipient accounts to enforce policies that only allow launches from approved images.

Before tag sharing, the build factory had to replicate tags to every account the AMI is shared with after each AMI publish. A typical workflow looked like this:

The build factory creates a new AMI and tags it.
An SNS notification triggers a Lambda function in each account.
Each Lambda function reads the tag values from the Amazon SNS message and calls CreateTags on the shared AMI in its own account.

Because this runs for every AMI, in every Region, across every account, the number of CreateTags API calls can grow rapidly in concentrated bursts. At that scale, teams risk throttling, failed Lambda invocations, and tag drift when calls silently fail.

With tag sharing, the build factory tags the AMI with the ec2:SharedTag/ prefix. Every account the AMI is shared with sees these tags, without the need for additional pipelines to replicate the tags. When the build factory updates a tag (for example, marking an older image as deprecated), the change is visible across all accounts without any additional API calls.

Considerations

Shared tags are visible to every account with access. We recommend that you do not include personally identifying, confidential, or sensitive information in these tags.

If consuming accounts have ABAC policies that evaluate tags, those policies will also apply to the shared AMI. For example, if a recipient account has a policy that allows ec2:RunInstances only when the AMI has the tag status=approved, and you share that tag by using ec2:SharedTag/status=approved, the policy will deny it.

When you adopt the shared tags model, any action that depends on those tags is controlled by the AMI owner, not by the recipient account. This includes tag-gated instance launches, AWS Identity and Access Management (IAM) and ABAC policy evaluations, and any downstream automation that reads tag values. Because only the owner can create, modify, or delete shared tags, the recipient account cannot change the values its own policies depend on. Before you rely on a shared tag in a policy or workflow, make sure you trust the owner account to set and maintain that value.

As a best practice, apply account-level governance so that you only consume AMIs from providers you trust. With Allowed AMIs, you can define criteria for which AMIs are allowed in your account, for example a list of trusted AMI provider account IDs. Only AMIs that meet the criteria are discoverable and available to launch in your account. To preview the impact before enforcing, start in audit mode.

Conclusion

EC2 Tag Sharing removes the need to build and maintain cross-account tag replication workflows. For organizations that share AMIs, this means less undifferentiated heavy lifting. To get started, add a tag with the ec2:SharedTag/ prefix to a shared AMI in the Amazon EC2 console. To learn more, see Tag your Amazon EC2 resources in the Amazon EC2 User Guide.

Implementing customer managed keys for AWS Lambda durable functions with Terraform

Rajdeep Banerjee — Tue, 29 Sep 2026 17:14:00 +0000

If you run regulated workloads, you must control how persisted data is encrypted and who can access it. You need to manage encryption key rotation schedules, restrict decryption to authorized principals, and produce audit evidence that proves encryption controls are operating as designed.

AWS Lambda durable functions build resilient, multi-step workflows that survive failures through automatic checkpointing. The checkpoint mechanism persists execution state, including step results, payloads, and callback responses, to durable storage. For payment processing workloads, this persisted data is sensitive. AWS Lambda durable functions support customer managed keys from AWS Key Management Service (AWS KMS). A customer managed key gives you three controls: you set the key rotation schedule, you restrict decryption access through the key policy, and you generate per-function audit trails in AWS CloudTrail. A durable execution uses the same encryption key it started with for its entire lifetime. Changing or removing the key affects only executions that start after the change.

Updating the customer managed key policy to remove decrypt permissions, or disabling the key, stops the Lambda service from accessing previously checkpointed state. Customer managed key deletion is a permanent action, and all durable executions encrypted with that key become unrecoverable because the Lambda service has no mechanism to restore the data. Before scheduling key deletion, use the AWS KMS waiting period (7 to 30 days) and monitor AWS CloudTrail for Decrypt calls to confirm that the key is no longer in active use.

In this post, you learn to configure a customer managed key to encrypt durable execution data in an event-driven payment processing workflow. You create a symmetric encryption key in AWS KMS and define a key policy that grants the Lambda service, the function’s execution role, the function author, and durable execution operators only the AWS KMS actions each principal requires. You then configure the function to use the key for durable execution encryption and verify encryption operations through AWS CloudTrail logs. By the end, you have a deployable reference architecture you can adapt for regulated workloads running on Lambda durable functions.

To learn more about how AWS Lambda encrypts durable execution data, see Encrypting AWS Lambda durable execution data in the AWS Lambda Developer Guide.

Solution overview

The sample application implements an event-driven payment processing pipeline using Amazon DynamoDB, Amazon EventBridge, Amazon EventBridge Pipes, AWS Lambda, and Amazon SQS. The pipeline receives authorized payment transactions, validates and enriches them. A Lambda durable function applies business rules to the enriched transactions. The approved transactions are sent to a downstream settlement system for posting.

The following section covers the key architectural steps.

Architecture steps

The upstream authorization system writes authorized payment records to a DynamoDB table.
DynamoDB Streams captures each new record as an ordered change event.
Amazon EventBridge Pipes polls the record from the DynamoDB stream. The pipe triggers a Lambda function as part of enrichment step for duplicate checking.
The deduplication Lambda uses a DynamoDB table with conditional writes to identify duplicate inbound transactions based on transaction properties and time window.
When the deduplication is successful, the pipe publishes an event to the Amazon EventBridge custom event bus.
An Amazon EventBridge rule invokes a Lambda function for matching events. The function adds business context such as account type, bank routing details, and merchant category codes. The function publishes a new enriched event to the custom event bus.
Another Amazon EventBridge rule matches the enriched events to a Lambda durable function. The durable function applies business rules to the incoming event. When the event passes all business rules, the function publishes a new event to the event bus.
An Amazon EventBridge rule routes the approved event to an Amazon SQS queue preserving ordering for settlement and buffering against downstream throughput limits.
The Posting Lambda function reads from the Amazon SQS and invokes the downstream posting subsystem to post the transaction. Finally, the function publishes a completion event to the event bus completing the transaction lifecycle.

With customer managed keys configured on DynamoDB, Amazon EventBridge, SQS, and the AWS Lambda durable function, every piece of persisted data in this pipeline is encrypted with keys you own and control. The walkthrough that follows shows you how to deploy this configuration with Terraform.

Figure 1 shows the reference architecture for this solution.

Reference architecture

Figure 1: Payment processing using Lambda durable functions

Prerequisites

To deploy this solution, you need the following prerequisites:

AWS account and CLI: An active AWS account with the AWS CLI installed and configured with appropriate credentials.
Terraform: Terraform installed (version 1.0 or later) for infrastructure provisioning.
Python environment: Python 3.11 or later, with pytest for running unit tests. The aws-durable-execution-sdk-python package requires Python 3.11 or later.
AWS Identity and Access Management (IAM) permissions: The IAM permissions to create the resources. Follow the sample repository for the sample policy.
Basic understanding and familiarity with AWS Serverless services.

Solution walkthrough

The following is a step-by-step guide to deploy and test the payment processing solution.

Step 1: Clone the repository

git clone https://github.com/aws-samples/sample-payment-processing-with-lambda-durable-functions.git
cd sample-payment-processing-with-lambda-durable-functions/source

Step 2: Run unit tests

Validate the payment processing logic locally before deploying:

cd lambda-src/business_rules
pip3 install -r requirements-test.txt
pytest test_app.py -v

This runs unit tests that cover transaction validation, business rule checks (foreign transaction detection, currency conversion, merchant type), event schema validation, and misconfiguration handling. The tests use the AWS Durable Execution Testing SDK to run the handler locally without deploying AWS resources.

Figure 2 shows an example of test results running locally.

Figure 2: Test run results of the business rules

Step 3: Inspect the Lambda durable functions construct

Open the payments-business-rules Lambda function in source/lambda-src/business_rules/business-rules-app.py for a sample Lambda durable function. Refer to Figure 3 for the code walkthrough.

Key features used

@durable_execution decorator: Transforms a standard Lambda handler into a durable function handler. The durable execution SDK manages checkpointing automatically. No infrastructure changes are required.
context.step("validate-transaction"): Validates that the transaction has a non-empty issuingCountryCode. The durable execution checkpoints the result (True or False) to durable storage. The durable execution restores checkpoint results instead of re-executing steps during the replay phase. This phase occurs whenever the function is re-invoked after an interruption such as a wait period completing, a failure, or a suspension. This checkpointed result is part of the durable execution data encrypted by your customer managed key.
context.step("publish-posting-failure"): Publishes the full Amazon EventBridge envelope to Amazon SNS when validation fails. This step only runs on the failure path. The runtime checkpoints the Amazon SNS publish response to durable storage.
context.parallel("run-business-rules"): Runs three independent rule checks concurrently: foreign transaction detection, currency conversion, and merchant type validation. Each branch checkpoints independently. If one branch fails, the others are not replayed on resume. Each branch result is persisted to durable storage and encrypted by the customer managed key.
ctx.step("trigger-foreign-transaction-rule") (inside parallel): Compares billingAmount against transactionAmount. If they differ, it emits a ForeignTransactionFound event to Amazon EventBridge. This step is checkpointed independently within the parallel group.
ctx.step("trigger-conversion-rate-rule") (inside parallel): Checks whether conversionRate equals 1. If so, it emits a CurrencyConversionTransactionFound event to Amazon EventBridge. This step is checkpointed independently within the parallel group.
ctx.step("trigger-merchant-rule") (inside parallel): Checks whether merchantType equals AAFF. If so, it emits a WarningMerchantTypeTransactionFound event to Amazon EventBridge. This step is checkpointed independently within the parallel group.
context.step("post-transaction-processed"): Emits the final TransactionPostingApproved event to Amazon EventBridge. This step is only reached when validation passes and all business rules complete. The runtime checkpoints the Amazon EventBridge response. On replay, if this step already succeeded, the event is not re-published, which guarantees exactly-once approval semantics.
context.logger: Provides replay-aware logging throughout the handler. During replay of previously completed steps, log statements are suppressed to prevent duplicate log entries in Amazon CloudWatch.

Figure 3: Sample Lambda durable functions code

Step 4: Deploy infrastructure with Terraform

Terraform currently doesn’t support attaching a customer managed key directly to the durable function. You create the symmetric key in Terraform and then associate the key with the durable function on the AWS Management Console. Refer to source/durable_kms.tf for the key configuration.

Initialize and deploy the AWS resources that make up the solution:

cd ../../
terraform init
terraform plan -var="region=us-east-2"

Review the plan output, then apply:

terraform apply -var="region=us-east-2" --auto-approve

Note: Replace us-east-2 with your preferred AWS Region.

On successful completion, Terraform outputs the AWS KMS key alias, key ARN, and DynamoDB Streams ARN used by the event-driven pipeline:

Apply complete! Resources: N added, 0 changed, 0 destroyed.

Outputs:

durable_kms_key_alias = "durable-function-encryption"
durable_kms_key_arn = "arn:aws:kms:us-east-2:xxxxxxxxxxxx:key/4e87d4c2-1190-4db4-8b97-46657f83ee00"
stream_arn = "arn:aws:dynamodb:us-east-2:xxxxxxxxxxxx:table/visa/stream/2026-03-16T14:25:47.847"

Step 5: Verify Lambda durable functions configuration

In the AWS Lambda console, navigate to the payments-business-rules function. Confirm that the function Type displays Durable, which indicates that the checkpoint-and-replay mechanism is active. Figure 4 shows the expected function configuration.

Figure 4: The Lambda durable function in the AWS Lambda console

Step 6: Add the AWS KMS key to the Lambda durable function

The durable function is not encrypted with a customer managed key. Figure 5 shows the function’s encryption configuration as empty.

Figure 5: The Lambda durable function missing a customer managed key in the AWS Lambda console
Choose Edit, then turn on Customize encryption settings as shown in Figure 6.

Figure 6: The Lambda durable function check encryption in the AWS Lambda console
Select the AWS KMS key ARN created for the durable function. The key ARN is available in the Terraform output from Step 4. Figure 7 shows the key selection.

Figure 7: Select the AWS KMS key ARN for the durable function in the AWS Lambda console
Choose Save and confirm that the durable function is now encrypted with a customer managed key, as shown in Figure 8.

Figure 8: AWS Lambda durable function with the customer managed key in the AWS Lambda console

Step 7: Execute a test payment

Invoke the payments-visa-mock Lambda function to simulate an end-to-end authorization flow. The mock function reads sample Visa authorization messages from a CSV file and writes them to DynamoDB, which triggers the event-driven pipeline. Figure 9 shows a sample test invocation.

Figure 9: Invoke the payments-visa-mock function to trigger workflow

Figure 10 shows a sample response after invocation.

Figure 10: Test results from the payments-visa-mock function to trigger workflow

The mock Lambda invocation creates records that follow the process described in the preceding architecture steps.

Step 8: Verify results

Open Amazon CloudWatch Logs and inspect the log group /aws/lambda/payments-business_rules. This log group belongs to the Lambda durable function for this use case. Figure 11 shows the CloudWatch log group on the console.

Figure 11: Search in CloudWatch

You see the complete business rules lifecycle for each transaction, as shown in Figure 12. The highlighted sections show all the business rules performed by the durable function. Each step is checkpointed by the runtime and encrypted by the customer managed key.

Figure 12: Search Results in lambda durable functions console

You can also check the other log groups to trace the full pipeline:

/aws/lambda/payments-enrich: Transaction enrichment logs.
/aws/lambda/payments-posting: Settlement posting logs.

Step 9: Verify the customer managed key configuration

You can verify the key configuration by using the AWS CLI:

aws lambda get-function-configuration \
    --function-name payments-business-rules \
    --query "DurableConfig" \
    --region us-east-2

Expected response:

{
    "KMSKeyArn": "arn:aws:kms:us-east-2:xxxxxxxxxxxx:key/4e87d4c2-1190-4db4-8b97-46657f83ee00",
    "RetentionPeriodInDays": 7,
    "ExecutionTimeout": 180
}

You can search in AWS CloudTrail to track the AWS KMS calls. When you configure or update the customer managed key on a durable function, Lambda validates the key policy with dry-run GenerateDataKey and Decrypt calls. These appear in CloudTrail with a DryRunOperationException error code, which confirms that the key policy permissions are correct and does not indicate an actual error. For more details, see Encrypting AWS Lambda durable execution data.

Clean up

To avoid ongoing charges, destroy all deployed resources using the following command:

terraform destroy -var="region=us-east-2" --auto-approve

Expected output:

Destroy complete! Resources: N destroyed.

Conclusion

In this post, you configured a customer managed key to encrypt durable execution data in a Lambda durable function. With a customer managed key, you control the key rotation schedule, restrict decryption access through the key policy, and generate per-function audit trails in AWS CloudTrail. You can revoke access to durable execution data at any time by updating the key policy, giving you full control over who can read execution state. In-flight executions stop at the next checkpoint call and new executions must be started after restoring access. For details, see When the customer managed key is unavailable.

For payment processors and financial institutions, encrypting durable execution data with a customer managed key satisfies compliance obligations for data-at-rest encryption, key governance, and access auditability across multi-step transaction workflows.

To get started, clone the sample repository and follow the preceding walkthrough. To learn more about Lambda durable functions, see the AWS Lambda Developer Guide.

AWS Lambda durable functions encryption documentation
AWS Lambda durable functions Developer Guide
Amazon EventBridge Documentation
AWS Step Functions Comparison Guide
Amazon DynamoDB Streams Documentation
AWS Guidance for Payment Systems using Event-Driven Architecture
AWS Serverless Workshops – Search for Lambda and serverless services workshops.

Simplify AMI discovery with Amazon EC2 and SSM Parameter Store

Ashwani Tyagi — Tue, 29 Sep 2026 16:14:17 +0000

If you manage Amazon Elastic Compute Cloud (Amazon EC2) infrastructure at scale, you have likely encountered the following situation. You release an infrastructure change with the correct Region, the correct instance type, and a launch template that has operated reliably for months. The deployment nevertheless comes up on an Amazon Machine Image (AMI) that is several patch cycles out of date, because the AMI ID hardcoded in the template had become stale weeks earlier. The condition goes unnoticed until a security scan flags the instance, at which point you must reconcile AMI IDs across Regions rather than close out the week.

That scenario is rarely a one-time event. It is one example of a broader pattern that quietly taxes teams running Amazon EC2 at scale: stale AMI IDs, manual parameter lookups, inconsistent Region mappings, and pipelines that silently fail to update. The following section examines four variations of this pattern in detail.

The common thread across all of these is the same. Locating the correct image is not the hard part. The difficulty lies in wiring that image into your infrastructure as code (IaC) in a manner that remains current. You identify the appropriate AMI on the console, then search AWS Systems Manager (SSM) Parameter Store paths to obtain the dynamic reference that maps to it. The workflow spans two tools and two mental models, with a gap in between where errors accumulate. Because the authoritative link between an AMI and its SSM parameter lived outside the API, teams had to reconstruct it by hand, and hands make mistakes.

A recent enhancement to the Amazon EC2 DescribeImages API closes that gap. When you call DescribeImages on a public AMI, the response now contains a PublicSsmParameterName field: the SSM parameter that resolves to the latest AMI in that lineage. A single API call replaces manual correlation.

In this post, we examine the operational friction that makes AMI management harder than it should be and show how this enhancement addresses it. We walk through practical examples using the AWS Command Line Interface (AWS CLI), AWS CloudFormation, Terraform, and Amazon EC2 Auto Scaling launch templates. We conclude with best practices for golden AMI pipelines, including operational considerations to review before adopting the feature in production.

Prerequisites

To follow the examples in this post, you will need the following:

An AWS account.
The AWS CLI v2 installed and configured with appropriate permissions (ec2:DescribeImages, ssm:GetParameters).
Basic familiarity with AMIs, SSM Parameter Store, and at least one IaC tool (CloudFormation or Terraform).

Understanding the operational challenges

Before addressing the solution, it is worth examining the problem in detail, because the problem seldom manifests as a single, dramatic failure. It is instead a gradual accumulation of minor frictions that, in aggregate, impose a measurable cost on teams responsible for compute.

AMI IDs are Region-specific, version-specific, and change frequently. The workflow of finding an AMI, locating its SSM parameter, and referencing it in templates spans multiple tools, and the boundaries between steps are where errors accumulate.

Challenge 1: Silent image aging

Scenario: An engineer copies an AMI ID into a Terraform module as an interim measure. Several months later, that identifier is embedded across four environments. New instances launch on an image that predates numerous patches. There is no error and no alert, only drift that remains invisible until an audit or a review brings it to light.

Impact: Hardcoded AMI IDs do not fail conspicuously. They fail quietly, by launching a prior image at a later date. The distance between “this was correct when written” and “this remains correct” widens continuously, and no owner is assigned to monitor it.

Challenge 2: The multi-region maintenance burden

Scenario: An application operates across three Regions. The same logical image (for example, the latest Amazon Linux 2023) carries a different AMI ID in each Region. Templates therefore accrue region-to-AMI mapping blocks, lookup logic, or both. Each additional Region introduces another entry to maintain, and each AMI refresh requires updating all of them.

Impact: The team ends up maintaining a translation table that AWS already maintains on its behalf. The mapping logic becomes load-bearing infrastructure in its own right, and a single stale entry in one Region produces inconsistent fleets that are difficult to diagnose.

Challenge 3: Barriers to onboarding

Scenario: A new engineer joins the team and poses a reasonable question: which SSM parameter corresponds to a given AMI? The answer resides in an internal knowledge-base page that was accurate eighteen months earlier. The engineer copies a path that appears correct, deploys, and inadvertently references the wrong lineage.

Impact: When the relationship between an AMI and its parameter is not discoverable from the API, it must be documented manually. Manually maintained mappings degrade over time. Each new team member re-learns the same institutional knowledge, and each instance of degradation introduces an opportunity to reference an incorrect value.

Challenge 4: Uncertainty about update success

Scenario: A golden AMI pipeline completes a build and updates a parameter. The command returns a success response, and the team assumes the new image is in effect. However, for certain parameter data types, a success response does not always indicate that the value was accepted. This specific behavior is examined in the best-practices section, as it is particularly relevant to golden AMI pipelines.

Impact: Confidence without confirmation carries substantial risk. A pipeline that presumes success can propagate a stale image across a fleet before the discrepancy is identified.

Considered individually, none of these situations constitutes a crisis. Considered collectively, they explain why “launch the latest image” is never, in fact, a single step. The common root cause is consistent across all four: the authoritative link between an AMI and its SSM parameter existed outside the API, requiring teams to reconstruct it manually, a process inherently prone to error.

What’s new: DescribeImages returns the associated SSM parameter

The new feature addresses precisely this boundary.

As of July 16, 2026, the Amazon EC2 DescribeImages API response includes a new field, PublicSsmParameterName, for public AMIs that have an associated SSM parameter. This capability is available at no additional cost in supported AWS Regions, including AWS GovCloud (US) Regions and the China Regions.

In place of the previous three-step correlation exercise, the workflow reduces to a single call:

Before	After
Find AMI → manually search SSM paths → confirm the correct match	Find AMI → PublicSsmParameterName returns the SSM path immediately
Two separate API calls or console workflows	A single DescribeImages call provides the complete mapping
Prone to mapping an incorrect parameter to an AMI	Authoritative mapping obtained directly from the API

The change introduces neither a new service nor a new pricing dimension. It relocates information that previously lived in knowledge bases into the API response.

In addition, you can now use the public-ssm-parameter-name filter in DescribeImages to identify all AMIs associated with a specific SSM parameter, making the relationship queryable in either direction.

How it works: API response walkthrough

Call DescribeImages on a public AMI with an associated SSM parameter. The response includes PublicSsmParameterName:

{
  "Images": [
    {
      "ImageId": "ami-0abcdef1234567890",
      "Name": "al2023-ami-2023.7.20260601.0-kernel-6.1-arm64",
      "Description": "Amazon Linux 2023 AMI 2023.7.20260601.0 arm64 HVM kernel-6.1",
      "Architecture": "arm64",
      "PlatformDetails": "Linux/UNIX",
      "State": "available",
      "Public": true,
      "OwnerId": "111122223333",
      "ImageType": "machine",
      "RootDeviceType": "ebs",
      "VirtualizationType": "hvm",
      "EnaSupport": true,
      "PublicSsmParameterName": "aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64"
    }
  ]
}

The PublicSsmParameterName value (in this case, aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64) identifies the SSM parameter associated with this AMI lineage.

Tip: The field is returned under the aws/service/ namespace without a leading slash. When using this value in SSM API calls, resolve:ssm: references, or CloudFormation dynamic references, prepend a forward slash. For example, use /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64. The SSM parameter is intended to resolve to the latest AMI in the lineage, which can help you keep infrastructure current.

Note: Not every public AMI has an associated parameter. The field is present only for lineages for which AWS publishes parameters. The field is also populated only for public AMIs. If you query one of your own private AMIs and observe an empty field, this is expected behavior rather than a defect.

Practical examples

The following four examples show how to use the new PublicSsmParameterName field across common IaC tools.

Example 1: Discover the SSM parameter for an AMI using the AWS CLI

Suppose you have identified an AMI on the console and wish to determine its SSM parameter path for use in your templates:

aws ec2 describe-images \
  --image-ids ami-0abcdef1234567890 \
  --query "Images[0].PublicSsmParameterName" \
  --output text

Output:

aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64

You can also perform the inverse operation and determine which AMI a given SSM parameter currently references:

# Linux example
aws ssm get-parameter \
  --name "/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64" \
  --query "Parameter.Value" \
  --output text

# Windows example
aws ssm get-parameter \
  --name "/aws/service/ami-windows-latest/Windows_Server-2022-English-Full-Base" \
  --query "Parameter.Value" \
  --output text

Tip: Public SSM parameters are available for both Linux (/aws/service/ami-amazon-linux-latest) and Windows (/aws/service/ami-windows-latest) AMIs. You can list all available parameters under these paths using aws ssm get-parameters-by-path –path .

Alternatively, you can use the new filter to identify AMIs by their SSM parameter name:

Note: The public-ssm-parameter-name filter returns all AMIs that have ever been associated with the specified parameter, including previous versions. Use sorting or additional filters (such as –query with CreationDate) to identify the most recent AMI.

aws ec2 describe-images \
  --filters "Name=public-ssm-parameter-name,Values=/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64" \
  --query "Images[].{ImageId:ImageId,Name:Name,CreationDate:CreationDate}" \
  --output table

Example 2: CloudFormation with dynamic SSM references

Once the SSM parameter path is known from DescribeImages, you can use CloudFormation dynamic references to resolve to the latest AMI at deployment time:

AWSTemplateFormatVersion: '2010-09-09'
Description: EC2 instance using SSM parameter for latest Amazon Linux 2023 AMI

Parameters:
  InstanceType:
    Type: String
    Default: t4g.micro

Resources:
  MyInstance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: !Ref InstanceType
      ImageId: '{{resolve:ssm:/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64}}'
      Tags:
        - Key: Name
          Value: MyLatestAL2023Instance

Alternatively, you can use the AWS::SSM::Parameter::Value parameter type to permit users to override the SSM path at stack creation time:

AWSTemplateFormatVersion: '2010-09-09'
Description: EC2 instance with configurable SSM-based AMI lookup

Parameters:
  AmiSsmParameter:
    Type: 'AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>'
    Default: '/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64'
    Description: SSM parameter path for the AMI (discovered via DescribeImages)
  InstanceType:
    Type: String
    Default: t4g.micro

Resources:
  MyInstance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: !Ref InstanceType
      ImageId: !Ref AmiSsmParameter
      Tags:
        - Key: Name
          Value: DynamicAMIInstance

CloudFormation resolves the AMI ID at deployment time, so the template never contains a hardcoded AMI ID, and any stack update adopts the latest AMI automatically. Two considerations warrant attention before relying on this approach. First, running instances are not affected. A stack update is required to roll out a newer AMI. Second, CloudFormation does not support drift detection on dynamic references, so if the underlying SSM parameter value changes between deployments, CloudFormation will not report it as drift. For ssm dynamic references in which a version has not been pinned, AWS recommends performing a stack update whenever the parameter changes, so that the stack retrieves the current value.

Example 3: Terraform with SSM parameter data source

Use the aws_ssm_parameter data source to resolve the SSM path to the latest AMI ID:

# Use the SSM parameter path discovered from DescribeImages
data "aws_ssm_parameter" "latest_al2023" {
  name = "/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64"
}

resource "aws_instance" "web" {
  ami           = data.aws_ssm_parameter.latest_al2023.value
  instance_type = "t4g.micro"

  tags = {
    Name = "LatestAL2023-Instance"
  }
}

Important: In Terraform, ami is a replacement-forcing argument on aws_instance. When the SSM parameter changes, Terraform proposes to destroy and recreate the instance. For stateful workloads, add lifecycle { ignore_changes = [ami] } or use launch templates with Auto Scaling (Example 4) instead.

Example 4: Auto Scaling launch templates with SSM parameters

For Auto Scaling groups, you can reference the SSM parameter directly in the launch template using the resolve:ssm: prefix:

# Create a launch template that uses the SSM parameter
aws ec2 create-launch-template \
  --launch-template-name al2023-auto-scaling \
  --launch-template-data '{
  "ImageId": "resolve:ssm:/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64",
  "InstanceType": "t4g.micro"
}'

When EC2 Auto Scaling launches a new instance, it resolves the SSM parameter at launch time to obtain the current AMI ID. You can verify the AMI ID to which a launch template resolves:

aws ec2 describe-launch-template-versions \
  --launch-template-name al2023-auto-scaling \
  --versions '$Latest' \
  --resolve-alias

The response shows the resolved ImageId:

{
  "LaunchTemplateVersions": [
    {
      "LaunchTemplateId": "lt-089c023a30example",
      "LaunchTemplateName": "al2023-auto-scaling",
      "VersionNumber": 1,
      "LaunchTemplateData": {
        "ImageId": "ami-0ac394d6a3example",
        "InstanceType": "t4g.micro"
      }
    }
  ]
}

The parameter is stored in the launch template. When the Auto Scaling group scales out or replaces an instance, it uses the launch template to resolve the SSM parameter and determine the AMI to launch. This is the most direct of the four patterns: the parameter serves as the single source of truth, and the Auto Scaling group’s normal instance lifecycle effects the rollout.

Before-and-after workflow comparison

The following table summarizes how this feature improves common workflows, and relates each entry to the challenges described earlier.

Workflow	Before	After
Discover the SSM path for a known AMI	Search SSM parameter namespaces manually. Test multiple paths. Confirm a correct match	A single DescribeImages call returns PublicSsmParameterName
Validate that an SSM parameter maps to the expected AMI	Call GetParameter, then call DescribeImages on the returned ID to verify	Use the public-ssm-parameter-name filter to view all associated AMIs directly
Set up IaC templates	Find AMI → search for SSM path → copy path to template → verify correctness over time	Find AMI → read PublicSsmParameterName from the response → use directly in the template
Onboard new team members	Document AMI-to-parameter mappings in knowledge bases, which become stale	New members self-discover using standard API calls
Audit AMI usage across teams	Cross-reference AMI IDs with SSM parameters in separate calls	A single API call provides the complete picture

Best practices: Using SSM parameters for golden AMI pipelines

The following recommendations describe how to derive the greatest benefit from this feature, with operational considerations identified where they are material.

1. Discontinue hardcoding AMI IDs

With PublicSsmParameterName removing the discovery barrier, switch all templates to SSM parameter references. Use {{resolve:ssm:}} in CloudFormation, the aws_ssm_parameter data source in Terraform (note the replacement behavior in Example 3), or the resolve:ssm: prefix in launch templates.

2. Create custom SSM parameters for your golden AMIs

For internally built golden AMIs, create your own SSM parameters using the aws:ec2:image data type:

aws ssm put-parameter \
  --name "/my-org/golden-ami/amazon-linux-hardened" \
  --type "String" \
  --data-type "aws:ec2:image" \
  --value "ami-0abcdef1234567890"

When the pipeline produces a new golden AMI, update the parameter:

aws ssm put-parameter \
  --name "/my-org/golden-ami/amazon-linux-hardened" \
  --type "String" \
  --data-type "aws:ec2:image" \
  --value "ami-0fedcba9876543210" \
  --overwrite

Stacks, launch templates, or Terraform configurations that reference this parameter can adopt the new AMI on their next deployment, with no template edits required in most cases.

Operational consideration: Because PutParameter validates aws:ec2:image values asynchronously, an HTTP 200 does not confirm the value was accepted. Subscribe to Parameter Store change events in Amazon EventBridge and confirm the operation succeeded before considering the rollout complete.

3. Use parameter versions and labels for controlled rollouts

SSM Parameter Store supports versioning and labels, which provide control over rollouts:

# Label the current production version
aws ssm label-parameter-version \
  --name "/my-org/golden-ami/amazon-linux-hardened" \
  --parameter-version 5 \
  --labels "prod"

Production launch templates reference the labeled version:

ImageId: resolve:ssm:/my-org/golden-ami/amazon-linux-hardened:prod

With this approach, you can update the parameter with a new AMI without immediately affecting production. Promotion to production is accomplished by moving the prod label, a deliberate, auditable action rather than an automatic side effect.

4. Combine with Amazon EC2 Image Builder for end-to-end automation

Use Amazon EC2 Image Builder to automate AMI creation, then configure the distribution settings to update your SSM parameter automatically when a new AMI is built. Combined with the new discovery feature, this establishes a closed loop:

Image Builder creates a new AMI on a schedule.
Distribution settings update the SSM parameter to point to the new AMI.
Auto Scaling and IaC resolve the parameter to the latest AMI at launch time.
With DescribeImages, any authorized party can determine which SSM parameter an AMI maps to.

5. Scope IAM permissions appropriately

Two permission requirements apply.

To launch instances by using SSM-referenced AMIs, the launching principal requires ssm:GetParameters on the relevant parameter paths:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ssm:GetParameters",
      "Resource": "arn:aws:ssm:*:*:parameter/aws/service/ami-amazon-linux-latest/*"
    }
  ]
}

Scope the Resource element to the paths actually in use. If you reference Windows parameters (/aws/service/ami-windows-latest/) or your own golden AMI paths (/my-org/golden-ami/), include those ARNs as well. Otherwise, launches will fail with an AccessDenied error.

To create a custom aws:ec2:image parameter, the pipeline principal also requires ssm:PutParameter and ec2:DescribeImages:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ssm:PutParameter",
      "Resource": "arn:aws:ssm:*:*:parameter/my-org/golden-ami/*"
    },
    {
      "Effect": "Allow",
      "Action": "ec2:DescribeImages",
      "Resource": "*"
    }
  ]
}

For broader guidance on keeping infrastructure current and automating operational processes, see the Operational Excellence Pillar of the AWS Well-Architected Framework.

Clean up

The examples in this post use read-only API calls (DescribeImages, GetParameter) and do not create billable resources. If you created a launch template while following Example 4, you can delete it as follows:

aws ec2 delete-launch-template --launch-template-name al2023-auto-scaling

Conclusion

The difficulty of AMI management was never attributable to any single failure. It arose from the steady accumulation of stale identifiers, region-mapping tables, stale documentation, and pipelines that presumed success, all of which are minor frictions that together produced significant operational effort and risk. The common thread was that the authoritative link between an AMI and its SSM parameter existed outside the API, requiring teams to reconstruct it manually.

The new PublicSsmParameterName field in the Amazon EC2 DescribeImages API relocates that link into the response, where it appropriately belongs. With a single API call, you can determine the SSM parameter for any public AMI. You can then reference it directly in CloudFormation templates, Terraform configurations, or Auto Scaling launch templates for automatic AMI updates.

To begin, call DescribeImages on any public AMI and examine the PublicSsmParameterName field. For further detail, see Reference the latest AMIs using Systems Manager public parameters in the Amazon EC2 User Guide.

For additional learning resources on AMI management and IaC on AWS, explore Amazon EC2, AWS Systems Manager Parameter Store, and Amazon EC2 Image Builder.

Frozen package management for air-gapped RHEL-family AMIs

Anand Krishna Varanasi — Tue, 29 Sep 2026 15:55:07 +0000

If you run a regulated, air-gapped compute fleet on RHEL-family instances, you have probably felt three requirements pulling against each other. Your organization must configure the network to remove internet access from the instances. Your team must review and approve new packages or version upgrades before you adopt them. Your team removes public repository definitions, restricts network paths, and configures instances to use only the internal repository your team has approved. Teams in chip design, finance, healthcare, defense, and the public sector often face this combination while still needing operating system updates.

This post describes a two-account pattern that separates the connected package-ingestion path from the air-gapped fleet. You create an authorized initial baseline and approve later changes to form a versioned package snapshot in Amazon Simple Storage Service (Amazon S3). EC2 Image Builder uses the frozen snapshot to build Amazon Machine Images (AMIs). Your team configures AWS Systems Manager Patch Manager to patch the instances your organization runs from the same internal package source.

The accompanying reference implementation demonstrates the pattern for RPM-based RHEL-family systems (AlmaLinux for example). It is a reference, not a substitute for distribution of licensing, vulnerability analysis, testing, or an organization’s change-management process.

The challenge: Getting packages into an air-gapped approval-gated fleet

Common delivery models each assume something an air-gapped fleet might not provide:

Red Hat Update Infrastructure (RHUI) expects each instance to reach the service. A fleet with no internet egress needs a different content path.
Red Hat Satellite supports disconnected content management, but it is a separate product and operational footprint. Teams that need a custom package-level approval workflow must integrate that workflow with their content-management process.
The Red Hat CDN requires a connected, entitled content-management path. Centralizing that path changes the network architecture, not the customer’s Red Hat subscription obligations.

The objective is not to replace these products universally. It is to show a serverless AWS pattern for teams that need an authorized repository baseline, explicit approval for later package changes, and a fleet with no public package source.

How the pattern works

The pattern combines three controls:

A frozen package repository on Amazon S3: The pattern stores a deployment-authorized baseline and subsequent approved package changes in versioned, per-OS repository prefixes. The repository manifest records the package inventory for each state.
EC2 Image Builder Orchestration builds AMIs from that repository: The build helps remove upstream repository definitions and configures the internal frozen mirror to be used for all package operations.
The launched fleet has no internet egress: The dnf operations are configured to resolve the internal mirror. Patch Manager uses the same repository source, so image builds and in-place patching draw from one frozen snapshot.

Choosing the upstream source

Choose one package lineage end to end. The parent AMI, repository content, and trusted signing keys must belong to that same lineage.

The reference implementation defaults to AlmaLinux vault content plus EPEL and an AlmaLinux parent AMI. The AlmaLinux OS Foundation states that AlmaLinux aims for binary and application binary interface (ABI) compatibility with RHEL. This is an AlmaLinux compatibility goal, not a Red Hat certification, and it does not make repository mixing a supported practice.

For genuine RHEL systems, use a Red Hat parent AMI, entitled Red Hat repositories, and Red Hat signing keys. A connected content-management host can retrieve content for the isolated environment. This centralizes the network path but does not reduce or change the customer’s Red Hat subscription obligations. Confirm those obligations against the applicable Red Hat agreement.

Do not pair AlmaLinux repositories with genuine RHEL hosts, or Red Hat repositories with AlmaLinux hosts. Mixed-vendor package lineages can create support, stability, and maintainability problems even when the RPMs appear mechanically compatible.

Architecture and workflow

The account boundary provides a primary security boundary for this architecture. The following diagram shows the connected Distribution account, the read-only Workload account, and an example cross-Region layout.

Figure 1: Two-account, cross-Region architecture separating the connected Distribution account from the air-gapped Workload account

The Distribution account owns the writable control plane and the only internet path. It runs Amazon EventBridge, three AWS Lambda functions, Amazon DynamoDB, Amazon Simple Notification Service (Amazon SNS), and the AWS Fargate sync task. It also owns the frozen S3 repository and its AWS Key Management Service (AWS KMS) key.

The Workload account is air-gapped and read-only with respect to the repository. It runs the internal HTTPS mirror, EC2 Image Builder, Patch Manager, and the compute fleet. Its mirror task role can read and decrypt frozen content but cannot write it.

The sample repository places the Distribution control plane in US East (N. Virginia), the frozen store in US West (Oregon), and the Workload resources in US West (Oregon) to demonstrate API-only cross-account and cross-Region operation. This Region split is not required. In most deployments, place the Distribution control plane and frozen store in the same Region unless data residency, disaster recovery, or an existing regional footprint justifies the additional latency, transfer cost, and KMS policy complexity.

The two accounts do not need Amazon Virtual Private Cloud (VPC) peering or a transit gateway. Cross-account access uses S3, KMS, and IAM policies. The VPC address ranges can overlap because no VPC-to-VPC route is required.

Package baseline and scheduled upgrade workflow

Before the scheduled workflow begins, your organization must authorize and run a full sync to establish the initial repository baseline. This bootstrap does not provide package-by-package approval. If your organization requires individual approval for every initial RPM, your team should generate and review the baseline manifest before promotion instead of relying solely on deployment authorization.

After the baseline, the detector runs on a customer-defined schedule. The reference implementation defaults to monthly. The following diagram shows the bootstrap distinction and the selective approval flow.

Figure 2: Package baseline bootstrap and the scheduled selective approval workflow

Detect. Amazon EventBridge invokes the detector Lambda function on the configured schedule. The detector compares upstream repository metadata with manifest.json, which records the current frozen inventory. It classifies a newer version as an upgrade and an absent package as new.
Request approval. The detector writes candidates to S3, creates a KMS-protected review token carrying the request ID and expiry, and is designed to send a review link through SNS. The detector can use kms:Encrypt but not kms:Decrypt.
Review. A human opens the review page through an Amazon API Gateway HTTP API, reviews the proposed package versions, and chooses which changes to approve. The approver can use kms:Decrypt but not kms:Encrypt.
Record and start. A conditional DynamoDB update changes a request from pending to approved only once. The approver then starts the Fargate sync task and passes the request ID.
Selective sync. The task reads the approved package list, downloads those package versions, is designed to perform verification checks, and regenerates repository metadata.
Update the manifest. When the task stops, Amazon EventBridge invokes the manifest-updater Lambda function. It archives the outgoing manifest and records the resulting repository inventory.

The approval decision controls adoption. It does not prove that package code is safe. Advisory review, vulnerability scanning, testing, and staged rollout remain in separate controls.

Evidence from the approval workflow

The token ties a review action to a specific request and expiry. Separating kms:Encrypt from kms:Decrypt prevents either Lambda function from performing both token roles. The conditional DynamoDB write makes the approval transition single-use.

DynamoDB records request state, AWS CloudTrail records control-plane API activity, and manifest history records repository inventory changes. These service records can feed the organization’s existing audit and evidence-management workflow. Object-level S3 access auditing requires CloudTrail S3 data events. KMS activity alone is not a substitute for those events.

The frozen package repository on Amazon S3

The following diagram shows the per-OS, per-component prefix layout, and manifest objects.

Figure 3: Per-OS, per-component prefix layout of the frozen repository on Amazon S3

Each pinned operating system version receives its own prefix. Repository components such as BaseOS, AppStream, and EPEL contain Packages/ and repodata/ trees. manifest.json records the active inventory, and archived manifests preserve historical evidence and comparison points.

Your organization configures the bucket with versioning and SSE-KMS. Public RPM content does not require a customer-managed KMS key for confidentiality, so your organization could instead configure SSE-S3 for encryption at rest. However, SSE-S3 would remove the separate cross-account authorization control provided by the customer-managed KMS key policy. The customer managed key is used here for explicit cross-account key-policy control and revocation, and the manifests reveal the fleet’s exact software inventory. S3 Bucket Keys reduce KMS request volume. If object-level access evidence is required, enable CloudTrail S3 data events.

A rollback must restore a coherent repository state, including metadata and any required object versions. Restoring only manifest.json does not roll back repository contents.

Building, patching, and running the fleet

At AMI build time, an Image Builder component installs the configured repository keys, moves existing repository definitions aside, and writes one frozen repository definition per component. It locks the package manager to the frozen repository directory, fetches metadata through the internal mirror, and fails the build if the mirror validation step fails. An optional curated package list demonstrates that the AMI can install real packages through the frozen path.

Patch Manager uses the same mirror for the running fleet. A host created from an older AMI and a newly built host are therefore patched toward the same frozen snapshot. Instances run without an Amazon VPC NAT gateway, public IP, or an Amazon VPC internet gateway route in the Workload VPC, and their repository configuration contains no public fallback.

The intended verification model is defense in depth: the sync task helps verify a vendor’s signature before content enters the trusted repository, and the system verifies it again at installation through dnf. The ingestion gate helps reject digest-only results and can be configured to help confirm that only valid package signatures from a trusted lineage key are accepted.

The package mirror

Nginx fronts aws-sigv4-proxy, which signs cross-account S3 GET requests using the mirror task role. To a client, the service appears as a standard HTTPS package repository behind an internal Application Load Balancer and private DNS name.

Use the latest version of aws-sigv4-proxy (current latest is v1.12). This version 1.12 contains the fix for signing S3 paths (or the OS package names) with special characters such as +. Earlier versions can return SignatureDoesNotMatch. The reference implementation pins the reviewed v1.12 release commit immutably. Keep it current through dependency-update reviews.

Security boundaries and limits

The design provides the following controls:

No automatic public-repository adoption: A new upstream version enters the selective path only after an explicit, recorded decision by the user.
Repository ingestion and installation checks: You configure strict sync-time signature validation to help validate content before it enters the trusted store. dnf verifies again during installation.
No package-channel egress: Workload instances are configured to prevent access to public package sources.
A read-only workload boundary: A Workload-account principal cannot modify the frozen repository.

Human approval is not a malware detection. A reviewer cannot reliably identify a backdoor in a legitimately signed package merely by seeing its name, version, or changelog. Use vulnerability intelligence, scanning, pre-production tests, and staged deployment as additional controls.

The approval state, manifests, and CloudTrail records can help support evidence for control frameworks such as SOC 2 change management, ISO 27001 patch-management controls, and FDA 21 CFR Part 11 electronic records. Applicability depends on the organization’s environment, audit scope, and assessor. Confirm it with the compliance team under the AWS shared responsibility model.

Cost and operations

Cost depends on the amount of repository content and the chosen networking and availability design. Components can include S3 storage and requests, KMS requests, Lambda invocations, DynamoDB, SNS, Fargate tasks, the internal load balancer, Distribution-account internet egress, Amazon VPC endpoints, and AMI snapshots. A three-task always-on mirror costs more than an S3 bucket alone. Estimate the target topology with current AWS pricing rather than applying a fixed monthly figure from the sample.

Run detection and review at an interval defined by patch policy and risk tolerance. The supplied default is monthly, but the Terraform input is configurable. If a package change must be reversed, restore a tested, coherent repository version and rebuild or patch affected hosts as appropriate.

Prerequisites

To set up the reference implementation, work through these in order:

Two AWS accounts: a connected Distribution account and an air-gapped Workload account.
Deployment tools: Terraform 1.5 or later, Terragrunt, Finch or Docker, and Python with pip.
AWS Command Line Interface (AWS CLI): one named profile per account.
Distribution networking: private subnets with internet egress that works without public IPs, security-group egress on port 443, and DNS resolution for public names.
Workload networking: VPC interface endpoints for ssm, ssmmessages, ec2messages, logs, kms, and imagebuilder, plus an S3 gateway endpoint.
Internal mirror identity: an AWS Certificate Manager (ACM) certificate and a private hosted zone. If the parent AMI does not trust the issuing CA, configure the CA file so the build installs the trust anchor.
Package lineage: a parent AMI, repositories, and signing keys from the same distribution lineage. A RHEL subscription is required when retrieving genuine entitled Red Hat content.
Optional deployment roles: otherwise, the stack uses each profile’s credentials.

The deployment creates state backend and Amazon Elastic Container Registry (ECR) repositories. Do not create those ECR repositories separately before applying their own Terraform units.

Reference implementation

The companion repository provides Terraform modules, Lambda handlers, two container images, a Terragrunt two-account layout, and Makefile targets for deployment and verification. The shipped alma810 example defaults to the AlmaLinux lineage (RHEL family).

Choose one distribution lineage before deployment:

AlmaLinux Parent Image default: use an AlmaLinux parent AMI, AlmaLinux vault repositories, EPEL, and the included AlmaLinux and EPEL signing keys. No Red Hat subscription is required.
Genuine RHEL Parent Image: use a Red Hat parent AMI, an entitled Red Hat content source, and Red Hat signing keys. The customer supplies the Red Hat subscription and content-access integration.

Note: The reference implementation only provides AlmaLinux lineage setup, not genuine RHEL. If you choose to use a genuine RHEL parent image lineage, only the reference implementation code needs to be updated to fetch the Red Hat credentials or subscription access, and the rest of the workflow remains the same.

Please follow the repository README for detailed setup instructions:

Fill in environments/config.hcl and both account files with the two accounts, networking, mirror certificate, package lineage, and parent AMI.
Create the Terraform state backend with make bootstrap DIST_PROFILE=<dist> WORK_PROFILE=<work>.
Review both account plans with make plan DIST_PROFILE=<dist> WORK_PROFILE=<work>.
Deploy in dependency order with make all BASELINE_APPROVED=true DIST_PROFILE=<dist> WORK_PROFILE=<work>. The baseline sync can take several hours.

Validate the internal package management workflow

Validation begins by running the AlmaLinux EC2 Image Builder pipeline. A successful build produces private, encrypted AMIs in the configured AWS Regions. To validate the configuration, launch a test instance from the generated AMI in a no-egress Workload subnet and verify that the instance is connected to the internal frozen repository for all package management workflow.

Figure 4: Instance showing only the internal frozen repositories enabled, with no public repository configured

The output confirms that only the internal frozen AlmaLinux repositories are enabled: frozen-baseos, frozen-appstream, and frozen-epel. The dnf makecache command successfully downloads metadata for all three repositories through the private mirror. No public repository is configured or used.

Now try installing, upgrading, or installing a new package to test that package operations are served by the internal frozen repository.

Figure 5: Package install and upgrade served by the internal frozen repository

Recommended practices

Fail closed on approval data. If the approved package list cannot be retrieved, stop the sync task and avoid substituting a full sync.
Govern the initial baseline. Record who authorized the bootstrap full sync, or require explicit review of its manifest before promotion.
Require vendor signatures at ingestion. Do not treat a valid package digest as equivalent to a trusted signature.
Keep package lineages consistent. Parent AMI, repository content, and signing keys must come from the same distribution lineage.
Use a customer-defined review cadence. Monthly is only the sample default.
Use immutable dependency references with active updates. Require aws-sigv4-proxy v1.12 or later, pin the reviewed artifact by digest or full SHA, and automate update proposals.
Test rollback as a repository operation. Restore metadata and objects together, then validate the mirror before using the restored state.

Clean up

The walkthrough deploys billable resources in both accounts. Please follow the repository README for detailed setup cleanup instructions.

Conclusion

This pattern separates connected package ingestion from an air-gapped fleet, provides an authorized package baseline and an explicit decision point for later package changes or upgrades, and keeps image builds and running hosts on one frozen repository snapshot. To learn more, visit the EC2 Image Builder service page, the EC2 Image Builder documentation, the Patch Manager documentation, and the Amazon S3 user guide. The reference implementation is available in aws-samples.

Building event-driven applications at scale with Amazon EventBridge

Nahid Karimaghalou — Mon, 28 Sep 2026 22:38:57 +0000

Event-driven applications on Amazon EventBridge usually start small and then spread. One team creates a Custom event bus, adds a few rules, and ships. Another team needs some of those events, so a rule forwards them to a bus in a second account. A third team needs a subset of what the second team receives, so another rule forwards again. A year later the organization runs dozens of Custom event buses joined by forwarding rules, and that topology has become a thing to operate in its own right.

That shape has a price, and the smallest part of it is the bill. Every forwarding hop is a separate ingestion, so cost tracks the topology rather than the number of consumers that needed the event. The harder problem is that nobody can see the whole picture. Governance spreads across the accounts it was meant to cover. Answering who publishes to a bus, who consumes a given event type, or what breaks when a team stops publishing means visiting each account and reading its rule configuration. Tracing one event is harder still: its path crosses several buses in several accounts, each with its own metrics and logs, and no single view follows it from publication to the consumer that never received it.

Application teams also wait. Publishing to a bus in another account, or consuming from one, needs a resource policy, a role, and a forwarding rule owned by a central team. The team that wants to build opens a ticket, and the platform team becomes a queue. Both the missing visibility and the waiting grow with every team onboarded.

Amazon EventBridge recently relaunched the Custom event bus, which tackles these challenges directly. A platform team creates one bus, shares it across the organization, and keeps control of who can publish and who can subscribe. Every consumer of those events is listed on the one bus rather than inferred from configuration spread across accounts. Application teams create their own Subscribers in their own accounts. The bus stores events for a retention period you choose, preserves order within a key the publisher sets, accepts Avro and Protocol Buffers (Protobuf) alongside JSON (including CloudEvents), and delivers to targets without a function in the path to translate a call. It runs alongside the Custom event bus – classic, so adoption is incremental.

In this post, you see how a platform team stands up a shared bus and governs access to it, how application teams onboard themselves with a single Subscriber resource, and how retention, ordering, open formats, transformation, and direct target integrations change what one bus can carry.

One bus, shared with the organization

The platform team’s job on a shared bus is narrower than it was on a fleet of them. It owns the bus and sets the boundaries: which principals can publish and what their events can declare, which principals can subscribe, and, where it matters, what those principals are allowed to filter on. Application teams then manage their own configuration within those boundaries, such as filters, targets, delivery roles, retry policies, and failure destinations, none of which the platform team needs to write or review. That division is the point of the design. The platform team keeps governance of the bus and stops owning everyone else’s configuration, which is what takes it out of the provisioning path without giving up control of who is on the bus.

Creating the bus is a single call in a platform account.

BUS_ARN=$(aws eventsv2 create-event-bus \
    --name company-events \
    --storage-configuration '{"RetentionPeriodInDays":7}' \
    --query EventBusArn --output text)

Retention is the one setting worth deciding deliberately here rather than revisiting after an incident. It runs from 1 to 365 days and can be modified later, but a change only applies going forward. Raising it widens the window for events published from that point on, and does not make older events readable again. Seven days covers a working week of history, which is usually enough to onboard a consumer or reprocess after a bug without paying to store a year of events nobody will read.

Sharing the bus is the second decision. AWS Resource Access Manager is the route to reach for first: it associates automatically for accounts in the same organization and reaches accounts outside it by invitation the consumer accepts. A resource policy written on the bus directly is the alternative, and can also name accounts inside or outside the organization.

Access is granted per principal, and publishing and subscribing are separate permissions. A team that produces order events gains no ability to read payment events from the same bus. One grant is not enough for a cross-account caller, as usual on AWS: the role that publishes or subscribes also needs its own IAM policy allowing those actions. The platform team decides which accounts can reach the bus, and each consuming team decides which of its own principals can use that access.

Taken together, those decisions produce the architecture in the following diagram. One bus lives in a platform account, and application teams publish to it and subscribe from their own accounts. An AWS Lambda function in Team A’s account calls PutRawEvents to publish events onto the Amazon EventBridge bus in the platform account. Team B and Team C each attach their own Subscriber: Team B’s delivers to a Lambda function, Team C’s to an Amazon DynamoDB table.

Figure 1: Multi-account sharing

Cost follows team boundaries because charges separate ingestion from delivery. The account that publishes an event pays to put it on the bus, and the account that owns a Subscriber pays for what that Subscriber consumes. Each team’s usage appears on its own bill, which is what makes a shared bus something a platform team can charge back rather than a shared cost center nobody can decompose. Removing the forwarding hops also removes the duplicated ingestion and delivery those hops created: the same event reaching the same three consumers is ingested once instead of three times.

Publishing in the format teams already use

Not every producer speaks JSON. Teams that standardize event exchange across an organization often register schemas and publish compact binary payloads, because the schema is the contract between teams that deploy on their own timetables. Accepting the formats those producers already emit is simpler than changing each one to convert to JSON first.

With the new Custom event bus, application teams can publish events in Avro, Protobuf, and CloudEvents (JSON) formats. For the binary formats, a schema registry named on the request is used to deserialize the events.

There are two publish APIs, and the payload decides which one to call. PutEvents takes structured JSON with the familiar Detail, Source, and DetailType fields. PutRawEvents takes a binary payload plus metadata you define, and is the one to use for Avro, Protobuf, CloudEvents, or bytes the bus should not interpret.

import boto3

events = boto3.client("eventbridgev2")
events.put_raw_events(
    EventBusArn=BUS_ARN,
    SchemaRegistryConfiguration={"RegistryUri": GLUE_REGISTRY_ARN},
    Entries=[
        {
            "Data": avro_encoded_order,  # bytes, straight from your existing producer
            "SystemMetadata": {"ContentType": "application/avro"},
            "Metadata": {"eventType": "OrderPlaced"},
        }
    ],
)

The schema registry can be either the AWS Glue Schema Registry or the Confluent Cloud Schema Registry.

Because the bus decodes the event before filters and transformations run, a consumer subscribing to Avro events written by another team needs no schema, no decoder, and no access to the registry. It writes the same filter it would write against JSON. Producers and consumers stay decoupled, and no deserialization code has to be repeated in each consuming team.

Publishers get one more setting on the same request: deduplication. A retry that already succeeded would otherwise leave a duplicate for every consumer to handle. It works one of two ways: the bus hashes the content of each event, or it uses a deduplication ID you supply. Content-based hashing suits producers with no natural key, since two identical events hash the same. A deduplication ID fits when you already have one, such as an order ID combined with a state transition. It keeps matching even when parts of the payload differ in ways that should not count as a new event.

Self-service onboarding for application teams

The new Custom event bus introduces a new resource called a Subscriber. Application teams create and configure their own Subscribers in their own accounts, provided they have been granted subscribe access to the bus. A Subscriber is the one place a consumer’s behavior is defined: which events it receives, where they are delivered, how delivery is retried, and where events go when delivery does not succeed. Reviewing or changing a consumer is one thing to read and one thing to update.

SUBSCRIBER_ARN=$(aws eventsv2 create-subscriber \
    --name orders-to-fulfilment \
    --event-bus-arn "$BUS_ARN" \
    --filter-configuration '{"Filters":[{"Scope":"METADATA","Pattern":"{\"eventType\":[\"OrderPlaced\"]}"}]}' \
    --invoke-configuration '{"TargetArn":"'"$QUEUE_ARN"'","RoleArn":"'"$ROLE_ARN"'"}' \
    --retry-policy '{"MaxRetryAttempts":10,"MaxEventAgeInSeconds":3600}' \
    --on-failure-configuration '{"Arn":"'"$DLQ_ARN"'"}' \
    --query SubscriberArn --output text)

A filter’s scope decides which part of the event the pattern is matched against. DATA matches the payload, METADATA matches the key-value pairs the publisher attached to the event, and SYSTEM_METADATA matches the event’s system fields: the content type and ordering key a publisher declares, plus the fields Amazon EventBridge adds itself. Because Avro and Protobuf payloads are decoded as they are published, a DATA filter reads their fields directly, the same as it would for JSON.

The retry policy says how the bus should behave when a target is failing. MaxRetryAttempts sets how many times a delivery is retried, and MaxEventAgeInSeconds sets how long an event stays eligible for retry, measured from when it was published. Retries stop as soon as either limit is reached, so both bound the same delivery.

When deliveries do fail, the reason shows up in the Subscriber’s own logs, which application teams can turn on themselves. They record the error from each delivery attempt alongside the exact input sent to the target, which makes a problem quick to place. Seeing what the target actually received separates a transformation that produced the wrong shape from a target that rejected a correct one.

Figure 2: Creating a Subscriber on console

History for consumers that did not exist yet

A Subscriber sometimes needs events that were published before it existed. For example, a new analytics service needs hydrating with recent history, or a target processed a window of events incorrectly and needs that window replayed. Because the bus retains events for the period configured on it, a Subscriber can be created with a starting position in the past, so it reads history, catches up, and continues with live traffic:

aws eventsv2 create-subscriber \
    --name analytics-backfill \
    --event-bus-arn "$BUS_ARN" \
    --starting-position POINT_IN_TIME \
    --point-in-time-configuration '{"PointType":"TIMESTAMP","StartingPoint":"2026-09-14T06:00:00Z"}' \
    --filter-configuration '{"Filters":[{"Scope":"METADATA","Pattern":"{\"eventType\":[\"OrderPlaced\"]}"}]}' \
    --invoke-configuration '{"TargetArn":"'"$ANALYTICS_ARN"'","RoleArn":"'"$ROLE_ARN"'"}'

A starting position is either LATEST or POINT_IN_TIME. Choosing POINT_IN_TIME then needs a point-in-time configuration: a PointType of TIMESTAMP with a starting point, or HORIZON to begin at the earliest event still retained. An optional end point stops the read at a chosen time, which is what you want when reprocessing a known-bad window rather than catching up to live traffic.

Two things to keep in mind. The starting position is fixed when the Subscriber is created, so reading a different window means a new Subscriber. Treat the starting position as part of a Subscriber’s identity rather than a dial to turn later. And retention cannot reach back beyond the retention window, so the read starts at the earliest retained event however far back the timestamp asks for.

Order, where order matters

In event-driven architectures, where components are built to work asynchronously, the order events arrive in usually does not matter. There are still use cases where a consumer relies on ordered delivery, and the new Custom event bus offers it as an option on individual Subscribers.

Ordering is scoped by a key the publisher sets. A publisher includes an event group ID (a customer ID, an order ID, a driver ID), and a Subscriber created with FIFO delivery type receives the events for each group in the order they were published. A FIFO Subscriber reading events published without a group ID has nothing to sequence by, so the two sides work together. Creating one takes the same call as an unordered Subscriber, with the delivery type set to FIFO:

aws eventsv2 create-subscriber \
    --name inventory-ordered \
    --event-bus-arn "$BUS_ARN" \
    --type FIFO \
    --filter-configuration '{"Filters":[{"Scope":"METADATA","Pattern":"{\"eventType\":[\"OrderPlaced\"]}"}]}' \
    --invoke-configuration '{"TargetArn":"'"$FIFO_QUEUE_ARN"'","RoleArn":"'"$ROLE_ARN"'","SqsParameters":{"MessageGroupId":"{% $events.SystemMetadata.EventGroupId %}","MessageDeduplicationId":"{% $events.SystemMetadata.DeduplicationId %}"}}'

Ordering is per group, so throughput scales with the number of groups. If an event cannot be delivered, it holds up the rest of its own group while other groups keep moving. Choosing the key therefore matters: one that maps to a business entity, such as an order or a customer, gives sequencing where it is needed and independence everywhere else. A key so broad that most events share it puts them all in a single sequence, and a key so specific that every event has its own leaves nothing to order.

Because ordering is set on each Subscriber, consumers of the same events do not need to agree on it. An inventory service can receive a group’s events in sequence while an analytics service subscribing to those same events takes them as they arrive.

Reshaping events, and delivering directly to a target

A consumer’s business logic expects events in a particular shape, and the events on the bus are not always in that shape. Where the two get reconciled is an ownership decision: inside the consumer, where it becomes part of that team’s code, or on the Subscriber, ahead of it.

The first case is reformatting. A downstream system, often owned by another domain or outside the organization entirely, expects a different structure from the one the publisher emits. A JSONata transformer on the Subscriber produces that structure before delivery, so the consumer receives what it already expects. The business logic stays where it belongs, and when the published shape changes upstream, or another event type needs deriving into the same input, it is the transformer that changes rather than the consumer:

--transformer '{
    "Type":"JSONATA",
    "JsonataConfiguration":{
        "Expression":"{% {\"orderRef\": $events.Data.detail.orderId, \"total\": $events.Data.detail.amount} %}"
    }
}'

The transformer type determines the shape of what gets delivered. RAW delivers the event payload as is and is the default, so a Subscriber with no transformer configuration receives only the payload. WITH_METADATA adds the event envelope alongside it, and JSONATA reshapes it with an expression wrapped in {% %}.

The transformation reshapes events only for the Subscriber that owns it and does not affect what other Subscribers of the same bus receive. That also makes it a data minimization control: a partner can receive only the fields it needs rather than a whole internal event. Defining it at the Subscriber means it holds for every event without anyone remembering to strip fields.

The second case is calling an AWS service API. A Subscriber delivers directly to targets including Amazon Simple Queue Service (Amazon SQS), Amazon Simple Notification Service (Amazon SNS), AWS Lambda, and Amazon Kinesis Data Streams. For other services it has been common practice to add a proxy step whose only job is to make the call. With universal targets, the new Custom event bus can call a supported AWS service API directly, with the request body built by a JSONata expression.

TargetArn: arn:aws:events:::aws-sdk:dynamodb:putItem
UniversalTargetParameters.Input:
{% { "TableName": "orders", "Item": { "pk": { "S": $events.Data.detail.orderId } } } %}

Note that a universal target shapes its input through that parameter rather than through the preceding transformer, and setting a transformer on one is rejected when the Subscriber is created. The two mechanisms do the same kind of work on different targets.

That removes the proxy processing that existed only to make the call. The delivery role still needs the action the target requires and getting that wrong is the most common cause of a Subscriber that looks healthy and delivers nothing.

Conclusion

Running an event-driven application across many accounts no longer means running many event buses and the forwarding between them. A platform team creates one new Custom event bus, shares it across the organization through AWS Resource Access Manager or a resource policy on the bus, and keeps one place to decide who publishes and who consumes. Application teams create and own their Subscribers without waiting for provisioning. Ingestion and delivery are charged separately, so each team’s usage appears on its own bill, and the duplicated ingestion that forwarding hops created disappears with the hops.

The capabilities that used to send individual teams elsewhere now sit on the same bus. Ordering is per Subscriber and scoped by a publisher-supplied key, so one team’s sequencing requirement no longer fragments an architecture. Retention makes it possible to onboard a consumer that needs history it was never subscribed to. Avro and Protobuf are decoded by the bus, so producers keep their binary contracts. Transformation and universal targets keep business logic where it belongs, removing the proxy steps that existed only to reshape an event or make an API call.

Because the new Custom event bus runs alongside the Custom event bus – classic, adoption is incremental. Point one new consumer at a shared bus or forward a slice of an existing bus into it and move the rest as teams are ready.

Next steps. Create a bus, add a Subscriber, and publish an event, starting from the Amazon EventBridge documentation for the resource model and the AWS Command Line Interface (AWS CLI) reference. If you already run Custom event buses, the migration guidance covers routing existing events into a new Custom event bus without changing producers. From there, look at the Subscriber logging and metrics options for tracing an event from publication to delivery, and at AWS Resource Access Manager for how sharing and permissions work across an organization. If you have questions or feedback about the new Custom event bus, leave a comment on this post. We’d like to hear how you’re using it.

Improving Lambda function latency with scalable network bandwidth

Rahul Shandilya — Mon, 28 Sep 2026 21:17:09 +0000

AWS Lambda now supports scalable network bandwidth for functions configured with 2,048 MB of memory or more, running outside of a virtual private cloud (VPC). Previously, sustained network throughput was capped at 625 Mbps regardless of your function’s memory configuration. Now, sustained throughput scales proportionally from 625 Mbps at configurations below 2,048 MB up to 3,000 Mbps at 10,240 MB, increasing the rate at which data moves to and from your execution environment.

In this post, you learn how to apply this new capability to latency-sensitive data processing workloads, helping reduce function execution times and per-invocation costs while improving the end-user experience through reduced latency. You also walk through a deployable implementation that demonstrates the performance improvements this capability unlocks.

Latency-sensitive data processing

Latency-sensitive data processing applications are data processing workloads that must be completed in a defined period of time. They often experience bursty, ad hoc traffic patterns while being required to download gigabytes or even terabytes of data from a data store, process it in a compute environment, and return a result to a waiting end user.

Latency-sensitive data processing is often highly parallelizable. Data can be divided into smaller pieces with each piece being individually processed before combining them together to obtain a result.

These workloads can be found in multiple industries and verticals. Examples include:

Log querying engines – An end user initiates an on-demand search across terabytes of log data and expects results within seconds.
Insurance underwriting – A prospective customer submits an application, triggering real-time evaluation of historical claims and risk data. The underwriting process determines what coverage and premiums to offer to the prospective customer.
Financial ETL pipelines – An economic announcement triggers an unexpected burst of market data that must be ingested, transformed, and made available to downstream trading systems before the next market tick.
Genomics platforms – A clinician orders a diagnostic test, requiring gigabytes of DNA or RNA sequencing data to pass through a bioinformatics pipeline and be compared against a reference genome while the patient awaits results.

These workloads are challenging to build on traditional compute clusters. Their spiky and unpredictable nature forces you to choose between under-provisioning compute to optimize costs (and risk missing your SLA) or over-provisioning and paying for idle capacity.

Why Lambda fits latency-sensitive data processing

Lambda eliminates this tradeoff. Instead of pre-provisioning a compute cluster, Lambda scales compute capacity in response to incoming requests, matching processing power to unpredictable traffic patterns. Because latency-sensitive data processing is highly parallelizable, the ability of Lambda to rapidly scale out execution environments makes it a natural fit. You can fan out across thousands of concurrent functions to process data in parallel, paying only for the compute you use.

However, as data volume and performance requirements grow, network bandwidth to and from the compute environment can become the limiting factor in minimizing workload latency.

Scalable network bandwidth directly addresses this limitation by raising the per-environment network throughput ceiling, improving the rate at which data can be transferred to and from the execution environment. Each execution environment can now drive up to 3,000 Mbps of sustained throughput when configured with 10,240 MB of memory, a 4.8x increase from the previous ceiling of 625 Mbps. Combined with the ability of Lambda to scale out at a rate of 1,000 execution environments every 10 seconds, you can download more than 3 TB of data in under 10 seconds.

New network throughput behavior for Lambda functions

Scalable network bandwidth applies to both data ingress to and egress from an execution environment for functions outside of a VPC. For functions configured with 2 GB of memory or more, network bandwidth scales by approximately 280 Mbps increments for every 1 GB of additional memory allocated.

The following table shows the maximum sustained bandwidth available to each execution environment at each memory configuration.

Memory Configuration	Max Sustained Bandwidth
Less than 2,048 MB	625 Mbps
2,048 MB	765 Mbps
3,072 MB	1,044 Mbps
4,096 MB	1,324 Mbps
5,120 MB	1,603 Mbps
6,144 MB	1,883 Mbps
7,168 MB	2,162 Mbps
8,192 MB	2,441 Mbps
9,216 MB	2,721 Mbps
10,240 MB	3,000 Mbps (4.8x increase)

Table 1. Lambda sustained network bandwidth by memory configuration. Bandwidth scales at ~280 Mbps per additional GB of memory above 2 GB.

In the following section, you learn how scalable network bandwidth improves end-user latency by building an ETL pipeline that demonstrates it. You can find the source code in the GitHub repository.

Solution overview

Consider a SaaS analytics platform where users submit ad hoc queries against a data store. The application must extract the relevant data, apply a filter or transformation, and return an aggregate result while the user waits. In this example, the result needs to be returned in 8 seconds or less.

The following diagram illustrates the architecture of the solution.

Figure 1. ETL fan-out pattern: an orchestrator Lambda function distributes work to multiple worker Lambda functions that read from Amazon S3 in parallel, with bandwidth scaling callouts per memory tier.

A client initiates an ad hoc query by calling the orchestrator Lambda function through the Lambda API. The orchestrator function determines how to split the work. To process the data in parallel, the orchestrator function uses a ThreadPoolExecutor to issue synchronous invoke requests to the Lambda worker function, fanning out the worker across multiple execution environments at the same time.

Each Lambda worker function is configured with 10,240 MB of memory, so it has access to up to 3,000 Mbps of sustained network throughput. After the data is processed, the aggregated result is returned to the client.

Prerequisites

Before you start the deployment process, make sure that you have completed the following steps:

Install the AWS SAM CLI on your computer and confirm that you are running Python 3.12 or later.
Have your AWS account credentials ready.
Submit a request to AWS Service Quotas to turn on scalable network bandwidth for your Lambda functions. This quota is listed under Network bandwidth per execution environment.

Clone the source code from the GitHub repo and deploy the application within your AWS account. Creating the 10 GB test dataset and running the benchmark can incur charges to your AWS account.

git clone https://github.com/aws-samples/sample-lambda-enhanced-bandwidth
cd sample-lambda-enhanced-bandwidth
sam build
sam deploy --guided

After the AWS CloudFormation stack is deployed, record the DataBucketName and orchestrator function name from the stack outputs to use in subsequent commands.

To simulate data for the end user to query, the GitHub repo has a script that creates 10 GB of synthetic data and uploads it to your S3 bucket.

Mode 1: Processing pre-partitioned data

In Mode 1, the 10 GB of synthetic data is pre-partitioned. Pre-partitioned data is typically produced incrementally by many sources over a period of time, which can be the case with IoT data or access logs. The following command creates 10 GB of data divided into 20 partitions that are 512 MB each.

python scripts/generate_data.py \
    --bucket <DATA_BUCKET_NAME> \
    --total-gb 10 \
    --chunk-mb 512

Turning on scalable network bandwidth does not, on its own, make your downloads faster. A single download request only opens one connection to Amazon S3, and one connection does not move data fast enough to fill all the bandwidth now available to your Lambda function. To actually use your full allotment of network bandwidth, the execution environment has to pull the data over several connections at once. It does this by preferring the AWS Common Runtime (CRT) transfer client, a high-performance download engine built into Boto3. When the worker calls download_fileobj, the CRT client automatically breaks the 512 MB object into smaller parts and downloads them in parallel across multiple Amazon S3 requests. Those parallel downloads are what let a single Lambda worker take advantage of its full network bandwidth.

The following command runs the benchmark on the pre-partitioned data.

python scripts/run_fanout_benchmark.py \
    --orchestrator-name <STACK_NAME>-orchestrator \
    --bucket <DATA_BUCKET_NAME> \
    --iterations 5

Mode 2: Processing single large objects

Mode 2 generates 10 GB of data in one large object. This arrangement is more common when data is produced or delivered as one complete unit, such as database backups or genomic datasets. The following command creates 10 GB of data in a single large object.

python scripts/generate_data.py \
    --bucket <DATA_BUCKET_NAME> \
    --single-object-gb 10 \
    --key large-object/large-file.bin

In Mode 1, the CRT preference applies to Boto3 managed transfer methods such as download_file and download_fileobj. Mode 2 takes a different approach. Each worker reads a specific byte range of a single large object using get_object. The CRT preference setting has no effect on these calls. Instead, you can control concurrency by explicitly tuning the number of Lambda workers and using a bounded ThreadPoolExecutor to issue multiple byte-range requests at the same time.

When you run the following command, the orchestrator takes the single large object and divides it into consecutive byte ranges of 512 MB each. Each of the individual ranges is then processed by a Lambda worker execution environment in parallel.

python scripts/run_fanout_benchmark.py \
    --orchestrator-name <STACK_NAME>-orchestrator \
    --bucket <DATA_BUCKET_NAME> \
    --key large-object/large-file.bin \
    --slice-size-mb 512 \
    --iterations 5

The benchmark reports wall-clock duration, client-observed duration, aggregate throughput across workers, worker completion counts, and target compliance. When comparing memory configurations, keep the code, dataset, AWS Region, partition count, warm-up policy, and measurement count identical. You should run the benchmark multiple times in your account because placement, cold starts, concurrency, S3 behavior, and execution-environment reuse could affect results.

Results

To compare results, we ran the benchmark using a baseline configuration where the worker Lambda function is configured with only 1,024 MB of memory, well below the 2,048 MB threshold required for scalable network bandwidth to take effect. The 1,024 MB configuration limits network throughput to the previous sustained ceiling of 625 Mbps.

In our baseline test run, a worker downloaded and processed a single 512 MB partition with a 6.61-second download time at a 649.8 Mbps throughput (at p50). The 649.8 Mbps throughput exceeds the 625 Mbps ceiling because Lambda is capable of bursts in network throughput over a short period of time. Across twenty measured fan-out queries, the complete 10 GB query was completed with a 7.113-second wall-clock at p50. This fits within the 8-second SLA but leaves very little headroom.

To run our scalable network bandwidth benchmark, we re-deployed our worker Lambda function with a 10,240 MB memory configuration and re-ran the application. At a 10,240 MB memory configuration, each execution environment can now access up to 3,000 Mbps in sustained throughput. Direct 512 MB downloads achieved a 1.70-second download time and 2,521.3 Mbps throughput (both at p50). The complete 10 GB query was completed with a 2.640-second wall-clock at p50. That is 2.69 times faster, or 62.9% lower median latency, than the 1,024 MB configuration.

Table 2 summarizes the direct worker and end-to-end fan-out measurements for the same 10 GB dataset and 20 × 512 MB orchestration pattern. Aggregate throughput is the total data transfer rate across all twenty execution environments spun up to run the benchmark.

Memory Configuration	Single 512 MB partition download time and throughput (p50)	10 GB fan-out wall time (p50)	Aggregate throughput p50
1,024 MB baseline tier (sustained 625 Mbps)	6.61s / 649.8 Mbps	7.113s	12.08 Gbps
10,240 MB scalable tier (up to 3,000 Mbps)	1.70s / 2,521.3 Mbps	2.640s	32.54 Gbps

Table 2. Measured 1,024 MB baseline tier and 10,240 MB scalable bandwidth performance for a 10 GB fan-out ETL query.

Using scalable network bandwidth, the customer’s SLA headroom has improved by nearly 5 seconds. The Lambda function can now handle larger partitions within the same SLA window, reducing costs while still remaining comfortably within the customer’s SLA.

Clean up

To clean up the resources you created for the benchmark test, run the following commands:

aws s3 rm s3://<DATA_BUCKET_NAME> --recursive
sam delete --stack-name <STACK_NAME>

Best practices

After scalable network bandwidth is turned on for your AWS account, the following practices help you get the most out of it.

Profiling and planning

Test before you tune. Not every function is network-bound. Before increasing memory, profile your function to confirm that network I/O is the primary contributor to invocation duration and not CPU or application logic. Use Amazon CloudWatch Lambda Insights to inspect rx_bytes, tx_bytes, and duration. Functions where network I/O dominates invocation time are prime candidates for tuning.
Design for parallelism. Break your data into parallelizable chunks that can be processed independently in a fan-out pattern across multiple execution environments. You can use Amazon S3 byte-range reads to split large files into independently downloadable partitions. For implementation details, see Downloading an object with part numbers in the Amazon S3 User Guide.
Run AWS Lambda Power Tuning. Lambda Power Tuning is a state machine that helps you optimize your Lambda functions for cost and performance. Use Power Tuning to sweep memory configurations from 1,024 MB to 10,240 MB and identify the optimal cost-vs-latency point for your workload.

Implementation

Check upstream and downstream limits. Check the throughput limits of your data sources. For example, a Lambda function running at 3,000 Mbps can exceed the throughput capacity of a single S3 prefix, which supports up to 5,500 GET requests per second. When this happens, you will see HTTP 503 (Slow Down) errors in your application logs. Distribute your S3 objects across multiple prefixes to parallelize reads and avoid per-prefix throttling.
Balance bandwidth and CPU. Lambda allocates CPU proportionally to memory. For example, at a 1.7 GB memory configuration you are allocated 1 vCPU while a 10 GB memory configuration is allocated up to 6 vCPU. If your function processes data in parallel threads, the higher memory tiers give you both more network bandwidth and more CPU to process it. Use the concurrent.futures module in Python or worker_threads in Node.js to process data across parallel threads and maximize both CPU and network utilization.
Turn on Amazon S3 CRT for Boto3. If your function uses the Python runtime, initialize your Amazon S3 client with preferred_transfer_client: 'crt' to maximize single-connection throughput. The AWS Common Runtime automatically parallelizes requests across multiple TCP connections, which matters because individual TCP connections have a throughput ceiling.
Use SnapStart for JVM workloads. If you use Lambda SnapStart for Java functions, scalable network bandwidth reduces afterRestore hook latency. Network activity that occurs during function restore, such as pre-warming connections or pre-fetching configuration data, can complete faster.

Conclusion

Scalable network bandwidth raises the per-environment sustained throughput ceiling of AWS Lambda from 625 Mbps to 3,000 Mbps, directly reducing end-to-end latency for data-intensive workloads. Combined with the Lambda scaling rate, you can now move terabytes of data in seconds, without provisioning or managing infrastructure.

To get started, request the Network bandwidth per execution environment quota increase through AWS Service Quotas and deploy the sample application from the GitHub repository to see the improvement firsthand.

Adding custom domains to AWS Lambda MicroVMs with Application Load Balancer

Frank Scarfo — Tue, 22 Sep 2026 16:48:32 +0000

AWS Lambda MicroVMs is a serverless compute building block that provides VM-level isolation, near-instant startup performance, and state retention. You can now give each user or job their own execution environment to securely run just-in-time code, whether user or AI-generated. You do this without managing virtualization infrastructure or choosing between isolation, speed, and state retention. Lambda MicroVMs are powered by Firecracker virtualization, the technology underpinning AWS Lambda.

When you run a workload on AWS Lambda MicroVMs, each MicroVM is reachable at a service-generated endpoint that looks like 92cfc7f9-….lambda-microvm-….on.aws. That works, but many teams want to expose their MicroVMs under a domain they own, such as 92cfc7f9-….microvms.example.com. When a browser is the client, they also want to satisfy cross-origin resource sharing (CORS) without changing the application inside the MicroVM.

Both are achievable today, entirely from load-balancing and networking primitives. There is no Amazon CloudFront distribution and no compute in the request path. All you need is an Application Load Balancer (ALB) that terminates TLS with your AWS Certificate Manager (ACM) certificate, rewrites the Host header, and forwards the request over AWS PrivateLink. In this post you’ll deploy that pattern with the AWS Cloud Development Kit (AWS CDK), map a wildcard of custom domains onto your MicroVMs, and let the ALB handle CORS for you.

The complete, deployable example is available as a pattern on Serverless Land. This walkthrough centers on the reusable networking pattern. The sample also includes a small demo application that provisions a MicroVM and mints an access token, which we reference but do not detail here.

What you’ll build

By the end you’ll have:

A wildcard custom domain like *.microvms.example.com, where each <uuid>.microvms.example.com maps transparently to the corresponding MicroVM.
An internet-facing ALB that rewrites the incoming request’s Host header to the real MicroVM endpoint and forwards requests to it privately over PrivateLink.
CORS preflight and response headers handled at the ALB, with no change to the code running in the MicroVM.

Calling https://<uuid>.microvms.example.com/<path> (with the MicroVM access headers described later) reaches the right MicroVM, with your domain intact end to end.

Solution overview

The request flow looks like this:

The key component is the ALB host header rewrite, introduced in URL and host header rewrite for Application Load Balancers. A listener rule matches the incoming custom host with a regex condition, captures the MicroVM ID from the left-most label, and a host-header-rewrite transform rewrites the Host header to <uuid>.lambda-microvm.<region>.on.aws before forwarding. Because the MicroVM service front-end routes on the Host header, the request lands on the correct MicroVM, while the customer’s domain stays in the browser’s address bar the whole time.

Why not CloudFront? Why not an ALB redirect?

CloudFront can also rewrite Host/SNI toward the origin, but a single distribution has static origins. Mapping a wildcard of MicroVM IDs through one distribution would require a CloudFront Function to compute the origin per request. The ALB transform performs the same rewrite for the entire wildcard with zero code.
An ALB redirect action only issues an HTTP 301 Moved Permanently/302 Found response. The browser would follow it, and the address bar would then show the .on.aws URL, which breaks our design as it is not a real custom domain. The transform (not a redirect) is what makes the custom domain transparent.

Walkthrough

The example is an AWS CDK application. Configuration lives under the microvm-custom-domains key in cdk.json (hosted zone, wildcard base, the endpoint base to rewrite to, the PrivateLink service name, and the CORS origin). Set those values, then deploy. The sections below explain what the stack creates and why.

Prerequisites

An AWS account.
An existing Amazon Route 53 public hosted zone for your domain (for example, example.com). The stack imports it and adds a wildcard record.
Node.js v22 or higher and the AWS CDK v2.
Credentials for the target account with permission to create Lambda, Amazon Virtual Private Cloud (VPC), Elastic Load Balancing, AWS Certificate Manager (ACM), and Route 53 resources through AWS CloudFormation.

1. Create the network and the PrivateLink endpoint

A small VPC (two Availability Zones, which is the minimum for an internet-facing ALB) hosts the ALB and an interface VPC endpoint to the AWS managed MicroVM service. There are no NAT gateways, because nothing here needs egress, which keeps the footprint lean.

// Interface (PrivateLink) endpoint to the AWS managed MicroVM service.
const endpoint = new ec2.InterfaceVpcEndpoint(this, 'MicroVmEndpoint', {
  vpc,
  service: new ec2.InterfaceVpcEndpointService(cfg.microvmVpceServiceName, 443),
  subnets: { subnetType: ec2.SubnetType.PRIVATE_ISOLATED },
});

2. Discover the endpoint’s private IP addresses at deploy time

An ALB IP target group needs the private ENI IP addresses of the interface endpoint (one per Availability Zone). CloudFormation does not expose those IPs as a usable attribute, so the stack resolves them during deployment with an AwsCustomResource that reads the endpoint’s own ENIs by ID (DescribeNetworkInterfaces on vpcEndpointNetworkInterfaceIds).

This is the only compute the package deploys, it runs only during cdk deploy, and it is never in the request path.

3. Request a wildcard TLS certificate

ACM issues a DNS-validated wildcard certificate for *.microvms.example.com, validated through the hosted zone you imported. The ALB presents this certificate for every custom domain under the wildcard.

4. Create the ALB and the MicroVM target group

The internet-facing ALB has an HTTPS:443 listener using the wildcard certificate. The target group holds the endpoint ENI IPs as IP targets, reached over HTTPS:443.

Encrypted in transit. A customer-provided AWS Certificate Manager (ACM) certificate is used to securely terminate encryption between the client and the ALB. The ALB re-originates TLS to the MicroVM service so traffic stays encrypted through the network.
IP-based targets. The target group uses IP-based targets with the local IP addresses of the VPC endpoints.
Health check matcher 200,403,404. The load balancer’s health probes are unauthenticated, so the MicroVM endpoint answers them with 403. A 403 here means “endpoint is reachable,” not “auth is broken,” so the matcher treats it as healthy.

const targetGroup = new elbv2.ApplicationTargetGroup(this, 'MicroVmTargets', {
  vpc,
  protocol: elbv2.ApplicationProtocol.HTTPS,
  port: 443,
  targetType: elbv2.TargetType.IP,
  targets: targetIps.map((ip) => new elbv2t.IpTarget(ip, 443)),
  healthCheck: {
    protocol: elbv2.Protocol.HTTPS,
    path: '/',
    healthyHttpCodes: '200,403,404',
  },
});

const listener = alb.addListener('Https', {
  port: 443,
  protocol: elbv2.ApplicationProtocol.HTTPS,
  certificates: [certificate],
  // Default action for anything that doesn't match our host regex.
  defaultAction: elbv2.ListenerAction.fixedResponse(404, {
    contentType: 'text/plain',
    messageBody: 'Unknown custom domain',
  }),
});

5. Add the host-header rewrite rule

A listener rule matches <uuid>.microvms.example.com with a regex condition and rewrites the Host header to <uuid>.lambda-microvm.<region>.on.aws with a host-header-rewrite transform. The regex captures the left-most label (the MicroVM ID) and reuses it in the replacement.

At the time of writing, the CDK L2 constructs don’t yet model regex host conditions or transforms, so the example reaches the underlying CfnListenerRule to set them:

const escapedBase = customDomainBase.replace(/[.]/g, '\\.');
const matchRegex = `^(.+)\\.${escapedBase}$`;     // capture <uuid>
const replaceWith = `$1.${microvmEndpointBase}`;   // <uuid>.lambda-microvm.<region>.on.aws

const cfnRule = forwardingRule.node.defaultChild as elbv2.CfnListenerRule;

cfnRule.conditions = [{ field: 'host-header', regexValues: [matchRegex] }];

cfnRule.addPropertyOverride('Transforms', [
  {
    Type: 'host-header-rewrite',
    HostHeaderRewriteConfig: { Rewrites: [{ Regex: matchRegex, Replace: replaceWith }] },
  },
]);

6. Point Route 53 at the ALB

Wildcard A and AAAA alias records (*.microvms.example.com) target the ALB, so every MicroVM custom subdomain resolves to it.

7. Deploy

Run the following commands to install the dependencies and then deploy the application.

npm install
npx cdk deploy

Handling CORS at the ALB

If your clients are browsers calling the MicroVM from another origin, CORS is handled entirely at the ALB, with no change to the application inside the MicroVM.

The listener uses ALB header-modification attributes to insert the Access-Control-Allow-* headers on every response. A higher-priority rule answers OPTIONS preflight requests at the edge with a fast 204 response. Otherwise, preflight requests would reach the origin and be rejected without an access token.

// Insert CORS headers on every response on this listener.
const cfnListener = listener.node.defaultChild as elbv2.CfnListener;
cfnListener.addPropertyOverride('ListenerAttributes', [
  { Key: 'routing.http.response.access_control_allow_origin.header_value',  Value: cfg.corsAllowOrigin },
  { Key: 'routing.http.response.access_control_allow_methods.header_value', Value: 'GET,POST,PUT,DELETE,OPTIONS,PATCH,HEAD' },
  { Key: 'routing.http.response.access_control_allow_headers.header_value', Value: 'x-aws-proxy-auth,x-aws-proxy-port,content-type,authorization' },
  { Key: 'routing.http.response.access_control_expose_headers.header_value', Value: 'content-type,content-length' },
  { Key: 'routing.http.response.access_control_max_age.header_value',        Value: '86400' },
]);

// Answer OPTIONS preflights at the ALB.
new elbv2.ApplicationListenerRule(this, 'CorsPreflightRule', {
  listener,
  priority: 10,
  conditions: [elbv2.ListenerCondition.httpRequestMethods(['OPTIONS'])],
  action: elbv2.ListenerAction.fixedResponse(204, { contentType: 'text/plain', messageBody: '' }),
});

Because the ALB adds those headers to both the preflight 204 and the forwarded MicroVM response, a browser’s cross-origin call succeeds without any application change. Set corsAllowOrigin to * for quick testing, and pin it to your own site for anything beyond a demo.

Test it end to end

First, launch a Lambda MicroVM and mint an access token (follow Create your first Lambda MicroVM). When it’s running, the service gives you a generated endpoint that looks like:

012345678-9abc-defg.lambda-microvm.us-east-2.on.aws

To get the custom-domain equivalent, replace the endpoint suffix (.lambda-microvm.<region>.on.aws) with your wildcard base: .microvms.example.com. Everything ahead of that suffix is preserved exactly:

012345678-9abc-defg.microvms.example.com

The ALB’s rewrite rule captures whatever precedes the suffix and re-attaches it to the real endpoint base, so the mapping holds for the entire wildcard. You never register anything per-MicroVM.

With your token in hand, call the custom domain you derived:

curl "https://012345678-9abc-defg.microvms.example.com/<path>" \
  -H "X-aws-proxy-auth: <token>" \
  -H "X-aws-proxy-port: 8080"

The request travels to the ALB, which terminates TLS, rewrites the host header, and forwards over PrivateLink to the MicroVM. The response comes back under your domain.

The reference architecture also includes a single-page demo and a POST /api/provision endpoint that runs or reuses a MicroVM and mints a short-lived token. With it, you can try the flow without wiring up token creation yourself. It even performs this suffix swap for you and hands back a ready-to-click custom-domain URL. See the repository for that piece.

Important considerations

Authentication is still the client’s job. This pattern only rewrites Host. The client must still supply a valid, unexpired access token in X-aws-proxy-auth. This is deliberate. MicroVM tokens are per-MicroVM and short-lived, so baking them into infrastructure would be fragile and insecure.
Region pinning. PrivateLink is regional, so the ALB, the endpoint, and the MicroVM service must all be in the same Region.
Production hardening. If you adapt the sample’s provisioning endpoint, put authentication and rate limiting in front of it, pin CORS to your origin, and scope IAM to the minimum. The sample’s provisioning path is intentionally open for demonstration and is not production-safe as written.
Cost. You pay for the ALB and the interface endpoint (hourly plus data processing) in addition to the Lambda MicroVM usage. There is no CloudFront distribution and no per-request compute in the data path.

Clean up

Run the following command in the same directory where you deployed the application from.

npx cdk destroy

This removes the ALB, target groups, endpoint, certificate, VPC, and Route 53 records created by the stack.

Conclusion

You can front AWS Lambda MicroVMs with customer-owned wildcard custom domains using an Application Load Balancer and AWS PrivateLink. The key is the ALB’s host-header rewrite. Because the MicroVM service routes requests based on the Host header, a single rewrite rule can transparently map an entire wildcard of custom domains onto your MicroVMs. CORS is handled at the edge as well. The whole setup relies only on networking primitives, with no CloudFront distribution and no compute in the request path.

To try it yourself, deploy the reference architecture and review the ALB URL and host header rewrite launch post for more on the transform feature.

Running self-hosted AI agent sandboxes with AWS Lambda MicroVMs

Brian Krygsman — Fri, 18 Sep 2026 15:10:33 +0000

Organizations are building AI agents that autonomously write code, query databases, and interact with internal systems on behalf of their teams. These agents handle use cases such as automated code review, data pipeline optimization, and infrastructure troubleshooting. When your AI agent generates a shell command, queries a database, or writes to a file system, that code needs a secure environment to run in. Without isolation, one session’s tool calls can contaminate another session’s state, inadvertently expose sensitive data across tenants, or unintentionally allow untrusted code to reach production resources. Self-hosted sandboxes solve this by keeping agent execution within your own AWS account, giving you full control over networking, secrets, and governance.

Say you’re building an internal AI agent that optimizes database queries for your engineering team. A developer asks it to find the ten slowest queries in your analytics database, rewrite them with better indexing, and test the results. That’s three tool calls in a single session. One hits a live database with real credentials. One generates code. One executes it. Now multiply that by fifty developers using the assistant at the same time. Each session needs its own credentials, its own filesystem, its own network boundary. If credentials or state cross session boundaries, you have inadvertent data exposure.

AWS Lambda MicroVMs is a serverless compute environment that provides general-purpose runtimes with the strong isolation of virtual machines and the rapid scaling of AWS Lambda. Powered by Firecracker virtualization, each MicroVM runs Amazon Linux with full OS access for up to 8 hours. You launch, suspend, resume, and terminate MicroVMs programmatically. You get the serverless benefits of managed infrastructure, responsive scaling, and pay-per-use pricing. Three capabilities make Lambda MicroVMs a strong fit for agent sandboxes:

VM-level isolation per environment: Each MicroVM runs in its own Firecracker virtual machine, providing hardware-virtualization-based isolation between sessions without the resource overhead and startup time required of full VMs. One developer cannot see a teammate’s session, even when both run at the same time.
Launch from snapshot: Like Lambda SnapStart, MicroVMs boot from a pre-captured memory and disk snapshot, skipping application initialization entirely. Your agent gets a near-instant ready-to-use environment.
4x vertical scaling without re-provisioning: A running MicroVM can scale CPU and memory up to 4x its initial allocation, which can range from 0.25 vCPU/0.5 GB to 4 vCPU/8 GB, without terminating or re-creating the environment. If the agent needs to run a heavy data transformation mid-session, it can get more resources without starting over.

In this post, we show you how to architect and build a self-hosted AI agent that uses Lambda MicroVMs as secure, isolated sandboxes for tool-call execution. Lambda MicroVMs can handle the compute isolation for running tool calls, while the host for production AI agents, such as Amazon Bedrock AgentCore, manages the agent logic, model routing, and session state. A complete reference solution is available in aws-samples.

How self-hosted sandboxes work

A developer asks the agent to “find the ten slowest queries in our analytics database and suggest index improvements.” The agent orchestration system starts a session then breaks the objective into tool calls and distributes them. A worker needs to pick up that session, run the queries, and return results.

Most AI agent orchestration services and frameworks use a work queue model to distribute tool-call execution. The orchestration service enqueues sessions representing tool-call work. A worker, the process that claims a session and executes its tool calls, runs inside a compute environment, posts results, and exits. In this architecture, each Lambda MicroVM is the compute environment, and the worker is the process running inside it. Claude Managed Agents self-hosted sandboxes run those workers inside your own infrastructure rather than on a shared, multi-tenant compute pool. Your database credentials stay in your virtual private cloud (VPC). Your network, introspection, and governance rules apply.

You can trigger workers in two ways:

Webhook-triggered: The orchestration application sends a notification when a session is ready. Your control plane launches a worker on demand.
Always-on: A long-running process continuously polls the work queue for new sessions.

The Lambda MicroVMs lifecycle aligns with the webhook-triggered pattern, where each session produces one inbound event that launches a fresh MicroVM. Lambda MicroVMs support configurable idle policies. After a configurable idle period, a MicroVM suspends automatically, preserving disk and memory state. It resumes when inbound traffic arrives or when you call the resume API. The MicroVM runs for the duration of the session, the worker exits, and the idle policy suspends then finally terminates the VM. Lifecycle hooks allow you to run custom logic at key steps in the MicroVM lifecycle.

In contrast, the always-on pattern risks breaking the polling loop by suspending the MicroVM when idle, since there’s no inbound traffic between sessions. You could disable the configurable idle period, but then you pay for empty polling. Use the webhook-triggered approach for self-hosted sandboxes on Lambda MicroVMs.

Architecture

The following figure shows the reference solution’s architecture, with the Anthropic agent orchestration service control plane on the left interacting with a self-hosted sandbox environment in AWS on the right.

Figure 1: Reference architecture for self-hosted AI agent sandboxes on Lambda MicroVMs

The sample architecture is event-driven. The only inbound traffic is the webhook call. When the event arrives, the handler launches a MicroVM. Once launched, the MicroVM pulls its assigned session from the orchestration system’s work queue and runs the task. In our example, the developer’s “find slow queries” request has been queued as a session. The agent now needs to reach your infrastructure, spin up an isolated environment, and hand off the work. The following sequence shows how each component interacts to fulfill a single session.

The orchestration service queues work as sessions. A MicroVM launches to service each session, and the worker is the process running inside that MicroVM that claims the session, executes tool calls, and returns results.

Once the orchestration service marks a session as ready to run, it sends a session.status_run_started webhook to an Amazon API Gateway endpoint, triggering a MicroVM launch.
The launcher verifies the webhook signature using a signing secret from AWS Systems Manager Parameter Store, rejecting invalid or stale deliveries before spending compute.
The launcher calls RunMicrovm, passing the session ID and a secret reference through runHookPayload. It deduplicates on the webhook event ID (backed by Amazon DynamoDB) so retries do not launch duplicate VMs.
The MicroVM boots from a pre-captured Firecracker snapshot and receives the dispatch on its /run lifecycle hook. The worker fetches the environment key from Parameter Store using its execution role. It pulls the matching session from the work queue, claims it, and executes tool calls in an isolated /workspace directory. When finished, it posts results and exits. The idle policy suspends then terminates the VM.

Deduplication. The webhook event ID serves as the idempotency key. The launcher uses Powertools for AWS Lambda (Python) with a DynamoDB persistence layer to verify exactly-once processing. If the orchestration application retries a delivery with the same event ID, Powertools protects the system from launching extra MicroVMs and doing extra work.

Credential boundaries. Each component accesses only the single secret it needs. The launcher reads only the webhook signing secret to verify inbound events. It passes only an ARN reference to the environment key into the MicroVM payload. The MicroVM’s execution role retrieves only that environment key at runtime. No single component holds both secrets.

Component	Has access to
Launcher Lambda	Webhook signing secret (verify inbound events)
MicroVM worker	Environment key (through the execution role, to poll and claim sessions)

Cost model. You pay for MicroVM run time per session, plus standard charges for API Gateway requests, Parameter Store API calls, and Lambda invocations for the launcher. When no sessions are active, no MicroVMs run. Cost scales with concurrent sessions and their duration, avoiding idle compute charges.

Implementation

The following sections explore the reference architecture in more depth.

Project structure

The reference solution uses AWS Serverless Application Model (AWS SAM) for infrastructure-as-code. Alternatively, if you use an AI coding agent such as Claude Code, Kiro, or Cursor, the Agent Toolkit for AWS includes a Lambda MicroVMs skill that gives your agent the procedures to provision, configure, and deploy MicroVM-based sandbox environments on your behalf.

├── template.yaml                    # SAM: launcher, API, WAF, secrets, roles
├── src/
│   ├── functions/launcher.py        # Verify signature, RunMicrovm
│   ├── microvm-image/
│   │   ├── Dockerfile               # AL2023 + Node.js worker
│   │   └── worker/worker.mjs        # Lifecycle hook server
│   └── scripts/build-image.sh       # Package + create MicroVM image

Launcher: verify the webhook before spinning up compute

When the webhook arrives saying a developer’s session is ready, the launcher’s first action is signature verification. If it fails, the function returns 401 immediately. No MicroVM launches. No DynamoDB writes. You don’t pay for fraudulent or replayed requests.

signing_secret = [REDACTED_PASSWORD]  # Verify webhook before spending compute
if not verify_signature(raw_body, headers, signing_secret):
    return {"statusCode": 401, "body": "invalid signature"}

After verification, the launcher builds a dispatch payload containing the session ID, environment ID, region, and an ARN reference to the environment key secret. It passes this to RunMicrovm through runHookPayload:

launched = microvm_client.run_microvm(
    image_identifier="arn:aws:lambda:us-east-1:123456789012:microvm-image:worker",
    run_hook_payload=json.dumps({"session": dispatch}),
    execution_role_arn=config.execution_role_arn,
    maximum_duration_in_seconds=28800,
    ingress_network_connectors=["arn:aws:lambda:::network-connector:aws-network-connector:ALL_INGRESS"],
    egress_network_connectors=["arn:aws:lambda:::network-connector:aws-network-connector:INTERNET_EGRESS"],
)

MicroVM worker: claim one session, execute, exit

The MicroVM image is built from a Firecracker snapshot. The worker process starts during image creation and is captured in the snapshot, so there is no application startup at run time. The /run lifecycle hook delivers the dispatch payload:

// POST /aws/lambda-microvms/runtime/v1/run
case "run": {
    const envelope = JSON.parse(rawBody);
    const dispatch = JSON.parse(envelope.runHookPayload);
    res.writeHead(200); // Acknowledge hook immediately
    res.end();
    const key = await fetchParameter(dispatch.session.ENVIRONMENT_KEY_PARAM_NAME);
    await pollAndHandleSession(dispatch.session.ANTHROPIC_SESSION_ID, key);
    // Session complete; terminate this MicroVM to release all resources
    await terminateMicroVm(envelope.microvmId);
}

The worker acknowledges the hook within its timeout, fetches the environment key, and claims the session. This is where the requested work begins. The worker connects to the analytics database, runs EXPLAIN ANALYZE on the flagged queries, writes optimized alternatives to /workspace/suggestions.sql, and posts the results back to the developer. All of that happens inside this single VM. When the session completes, the worker calls terminate-microvm to release all compute resources.

Deployment

For full deployment instructions, see the reference solution README. Before deploying, make sure you have these prerequisites.

Prerequisites

An AWS account with permissions for Amazon Simple Storage Service (Amazon S3), AWS Identity and Access Management (IAM), AWS Systems Manager Parameter Store, Amazon API Gateway, AWS Lambda, AWS WAF, Amazon CloudWatch Logs, and AWS Lambda MicroVMs.
AWS Command Line Interface (AWS CLI) v2+.
The AWS SAM CLI.
An existing Anthropic Claude Managed Agents agent configured with a self_hosted environment (note the agent ID and environment ID).
A webhook signing secret and environment key, both generated in the Anthropic Claude Console.

Four steps

Deploy the control plane. Build and deploy the SAM stack, which creates the launcher Lambda, API Gateway endpoint, WAF WebACL, DynamoDB idempotency table, Parameter Store entries, and MicroVM execution role.
```
sam build
sam deploy --guided --capabilities CAPABILITY_NAMED_IAM
```
Register the webhook and populate secrets. In the Claude Console, register the stack’s WebhookUrl output as a webhook endpoint subscribed to session.status_run_started. Store the signing secret and environment key in the Parameter Store resources created by the stack.
Build the MicroVM image. Package the Dockerfile and worker code, upload to Amazon S3, and create the image. The service runs your Dockerfile, launches the worker, and captures a Firecracker snapshot. Monitor build progress in Amazon CloudWatch under /aws/lambda/microvms/<image-name>.
```
./src/scripts/build-image.sh
```
Verify. Create a test session and confirm a MicroVM launches and completes end-to-end. The reference solution includes a verification script that creates a session, triggers the webhook, and validates the full flow.

Using Claude Platform on AWS (CPOA)

The preceding architecture works similarly when you access Claude through Claude Platform on AWS rather than the first-party API. Three things change in the worker:

Client initialization. Replace the first-party client with the AWS client and supply your workspace ID:

from anthropic import AnthropicAWS

client = AnthropicAWS(aws_region="us-east-1")

# Workspace ID is required on every request
# Set via ANTHROPIC_AWS_WORKSPACE_ID env var or pass per-call

Authentication options. CPOA supports two modes:
1. CPOA API key (aws-external-anthropic-api-key-...): Store it in Parameter Store the same way as the first-party environment key. These keys are short-lived (12-hour STS tokens) and must be regenerated when they expire.
2. SigV4 (IAM): The MicroVM execution role can sign requests directly, so there is no secret to store or rotate. Set the environment key secret to a placeholder value (for example, use-sigv4) and the SDK falls through to IAM credentials automatically. This is the recommended path for production.

In both authentication modes, attach the AWS managed policy AnthropicSelfHostedEnvironmentAccess to the MicroVM execution role. This policy grants the aws-external-anthropic actions needed to poll the work queue, claim sessions, and post results. See IAM actions for Claude Platform on AWS for the full reference.

Prerequisite: Enable outbound web identity federation once per AWS account:

aws iam enable-outbound-web-identity-federation

Everything else, including webhook verification, deduplication, credential separation, and idle policy remains the same.

Security

Earlier we talked about what goes wrong without isolation. Credentials exposed between sessions. Scripts unintentionally reaching production. Agents escaping their sandbox. This architecture implements defense in depth to help prevent these.

Each component accesses a single, scoped secret. The launcher passes only an ARN reference to the worker credential into the MicroVM. The MicroVM’s execution role retrieves only that credential at runtime. The analytics database connection string does not touch the launcher and does not leave your environment.

AWS WAF applies managed rule sets (OWASP, known bad inputs, IP reputation) and per-IP rate limiting. Amazon API Gateway request validation rejects malformed bodies. The launcher performs HMAC signature verification as the true authentication boundary.

Each session runs in its own MicroVM. Sessions do not share memory, disk, or network namespaces. Firecracker provides hardware-virtualization-based isolation. The launcher IAM role reads only the signing secret. The MicroVM execution role reads only the worker credential. Both are scoped to specific Parameter Store ARNs. The Amazon S3 artifact bucket blocks public access, enables versioning, and uses server-side encryption.

Conclusion

This post walked through how to give your internal AI agent a safe place to run database queries, generate code, and execute scripts on behalf of fifty developers without leaking data between sessions or reaching resources it shouldn’t.

AWS Lambda MicroVMs provide ephemeral, VM-isolated compute environments that align with the per-session execution model of AI agent sandboxes. Snapshot-based launch avoids application startup latency. Idle policies terminate VMs once sessions complete. Firecracker isolation verifies that sessions do not share state. You pay only for active execution time and maintain full control over credentials, networking, and governance within your AWS boundary.

You build and operate a serverless control plane. You get per-session VM isolation with no idle compute cost and no shared tenancy.

To get started, explore these resources:

Read more about Lambda MicroVMs in the launch announcement post.
Clone the reference solution in the aws-samples repository.
Add the MicroVMs skill to your coding agent.
Learn more about AWS Lambda MicroVMs in the Developer Guide.
Learn about Anthropic self-hosted sandbox configuration in the self-hosted sandbox documentation.
For more serverless learning resources, visit Serverless Land.

Multi-modal autoscaling with Amazon EC2 Auto Scaling: adding signals for faster, more reliable scaling

Shubhendu Dubey — Thu, 17 Sep 2026 18:18:38 +0000

How do you handle unpredictable workload patterns that spike during promotional events or seasonal peaks? Multi-modal autoscaling with Amazon EC2 Auto Scaling combines infrastructure metrics like CPU with application-level signals, so a group scales on the demand its users create and not only on how busy the servers look. Those signals track the load that drives your business outcomes, such as sales or sign-ups.

CPU-based autoscaling works well for many workloads, but some demand does not register as CPU right away. Adding signals such as request counts and application metrics lets a group respond to the load its users create. By publishing Amazon CloudWatch custom metrics and application-driven triggers, you give Auto Scaling more information to act on.

In our testing, a group that scaled only on CPU rejected about 7,000 checkout sessions during a demand spike, and a stronger baseline that added Application Load Balancer request count still rejected about 6,900. A group that added an application metric rejected none, and it held p99 latency to about 0.43 seconds against 2.25 seconds for the CPU-only group. Predictive scaling can add a forecasting layer for cyclical demand, but it needs days of history to be useful, so we treat it as a complement. In this post, we show you how to implement multi-modal autoscaling on EC2 Auto Scaling, with code samples and results from a controlled test.

Prerequisites

To follow along, you need access to the following AWS services with appropriate permissions:

EC2 Auto Scaling, for scaling policies and group management.
CloudWatch, for metrics, alarms, and dashboards.
AWS CloudFormation, for infrastructure deployment.

Expanding beyond single-metric scaling

The default target tracking policy in EC2 Auto Scaling uses average CPU utilization, a practical starting point because CPU usage is a universal characteristic of compute workloads. Adding complementary signals, such as application-level metrics or predictive forecasting, gives Auto Scaling more information to make timely capacity decisions.

For workloads that need a faster response from target tracking alone, see Faster scaling with Amazon EC2 Auto Scaling target tracking.

In distributed architectures, different components can have distinct scaling characteristics. An API gateway might correlate well with request rate, while a background processor scales better on queue depth. With multi-modal scaling, you can match each component’s policy to its actual workload pattern. For containerized workloads, consider event-driven autoscaling with KEDA on Amazon Elastic Kubernetes Service (Amazon EKS).

Multi-modal autoscaling combines three approaches to capacity management. Reactive scaling responds to current CloudWatch metrics, such as CPU utilization, memory, network throughput, response times, and custom application indicators. Application-metric scaling brings workload-specific signals into the decision, using custom CloudWatch metrics like active user sessions, queue depth, or transaction volume. These application metrics are often the closest measurable proxy for business activity such as orders or sign-ups. Predictive scaling uses machine learning in EC2 Auto Scaling to forecast capacity needs from historical patterns, so infrastructure scales before demand increases.

With application-metric scaling, applications can scale on signals that infrastructure metrics miss. An ecommerce platform might scale on active checkout sessions, while a streaming service scales on concurrent stream counts. In the test later in this post, we use active checkout sessions as the custom metric.

This section builds the configuration in layers. Start with CPU target tracking as a baseline that every group keeps, then add a custom application metric that reflects real user load. The test later in this post compares these signals against a request-count baseline. Predictive scaling is an optional forecasting layer described at the end.

Step 1: CPU target tracking

Start with the foundation that most workloads already use: a target tracking policy on average CPU utilization. Target tracking is a managed policy that adjusts capacity to keep a metric at or near a target value. It supports predefined metrics, including CPU utilization and request count per target, and custom CloudWatch metrics. When multiple target tracking policies are active, Auto Scaling coordinates them: it scales out if any policy requires it, but scales in only when all policies agree, which helps prevent oscillation.

# CPU target tracking scaling policy (ASG A)
CPUTargetTrackingPolicy:
  Type: AWS::AutoScaling::ScalingPolicy
  Properties:
    AutoScalingGroupName: !Ref AutoScalingGroupName
    PolicyType: TargetTrackingScaling
    TargetTrackingConfiguration:
      PredefinedMetricSpecification:
        PredefinedMetricType: ASGAverageCPUUtilization
      TargetValue: 70

Our test also included a second infrastructure baseline, a target tracking policy on the load balancer’s request count per target. It uses the same structure with a predefined metric:

# Request count target tracking (ASG B)
RequestCountPTTargetTrackingPolicy:
  Type: AWS::AutoScaling::ScalingPolicy
  Properties:
    AutoScalingGroupName: !Ref AutoScalingGroupName
    PolicyType: TargetTrackingScaling
    TargetTrackingConfiguration:
      PredefinedMetricSpecification:
        PredefinedMetricType: ALBRequestCountPerTarget
        ResourceLabel: !Sub "${Alb.LoadBalancerFullName}/${TargetGroupB.TargetGroupFullName}"
      TargetValue: 300
      DisableScaleIn: false

Step scaling is another option for spike handling. With step scaling, you can define different capacity increments for different alarm thresholds. It keeps evaluating the alarm during scaling activities, which can make it react faster than target tracking’s default evaluation window. Step scaling policies do not coordinate with each other.

Step 2: Add a custom application metric

Next, add a second target tracking policy on a custom CloudWatch metric that reflects application load. In our test, instances publish an active checkout sessions metric at a 10-second resolution. To act on that resolution, set a Period of 10 seconds on the policy. Without it, the policy waits for three 1-minute datapoints like any other and the high-resolution metric only adds publishing cost. With it, a scale-out can begin in about 30 seconds. The policy includes the Auto Scaling group dimension so it tracks the metric for the right group. We set the target to 100 active sessions per instance, about 75 percent of the measured per-instance capacity of 135. This leaves headroom to absorb a spike while new instances boot.

# Custom application metric target tracking (ASG C)
CustomMetricTargetTrackingPolicy:
  Type: AWS::AutoScaling::ScalingPolicy
  Properties:
    AutoScalingGroupName: !Ref AutoScalingGroupName
    PolicyType: TargetTrackingScaling
    TargetTrackingConfiguration:
      CustomizedMetricSpecification:
        MetricName: ActiveCheckoutSessions
        Namespace: ECommerce/CheckoutMetrics
        Dimensions:
          - Name: AutoScalingGroupName
            Value: !Ref AutoScalingGroupName
        Statistic: Average
        Period: 10
      TargetValue: 100
      DisableScaleIn: false

Step 3: Add predictive scaling

Predictive scaling is an optional forecasting layer. It uses machine learning in EC2 Auto Scaling to analyze historical load and scale ahead of recurring, cyclical demand, using customized metric specifications in ForecastAndScale mode. You need to provide several days of history for it to forecast well, so it complements reactive signals rather than replacing them. Start in ForecastOnly mode to watch the forecast before it drives any scaling.

Monitoring

Use CloudWatch dashboards to track how each policy contributes to scaling decisions, and set alarms on the metrics that matter for your workload, such as per-instance load or latency. Enable detailed monitoring on the launch template, with Monitoring set to true, so that the system publishes CPU metrics every minute. Without it, you cannot complete the CPU policy’s scale-in evaluation and your group will stop scaling in. Watching the policies side by side is what surfaced this scale-in behavior.

Performance results

We compared three Auto Scaling groups under an identical load profile in a single 75-minute test in the us-east-1 Region. Each group used c8g.large instances with a minimum of 6 and a maximum of 40 instances, and every group carried the same CPU target tracking policy at 70 percent as a fallback:

ASG A: CPU target tracking only. This is the single-signal infrastructure baseline.
ASG B: CPU target tracking plus an Application Load Balancer request-count policy. Request rate is a stronger infrastructure baseline than CPU alone.
ASG C: CPU target tracking plus the custom checkout-sessions metric at 10-second resolution, published with a Period of 10 seconds.

All the groups received the same load at the same time. During the shared ramp, arrival rate rose and every group scaled correctly, which makes the comparison fair. CPU crossed 70 percent on the CPU group, request count crossed its target of 300 on the request-count group, and all three converged to a similar size.

Ramp phase (arrivals 30% → 85%)	A: CPU only	B: A + ALB requests	C: A + app sessions
Instances (start → peak)	6 → 9	6 → 10	6 → 10
CPU	73.6%	74.0%	69.0%
Requests per target (target 300)	319	323	297

Then arrival rate was held flat while the number of concurrent checkout sessions kept rising, a shape that infrastructure signals cannot see. The next table reports that divergence phase, measured directly from CloudWatch and the load balancer.

Measured metric (divergence)	A: CPU only	B: A + ALB requests	C: A + app sessions
Instances (start → end)	11 → 11	11 → 11	11 → 22
Rejected checkouts	7,064	6,886	0
CPU (start → end)	67.2% → 45.1%	66.5% → 45.5%	66.2% → 36.8%
Peak sessions per instance	135	135	116
Requests per target	285 → 279	282 → 279	277 → 154

The difference is what each group could see. Arrival rate was held flat while the number of concurrent sessions rose, so CPU and request count stayed in range while the application saturated. The CPU-only and request-count groups held at 11 instances and rejected 7,064 and 6,886 checkouts. Their CPU even fell, from about 67 percent to about 45 percent, because a rejected request never reaches the work it would have done, so a policy targeting 70 percent saw spare capacity at the moment the application was failing users. The application-metric group read the rising sessions directly and scaled from 11 to 22 instances, rejecting none.

Effect on latency and errors

We measured latency and rejected checkouts on the load balancer during the test. At rest, all groups were identical. The gap opened only in the divergence phase, when concurrency rose without a matching change in arrival rate. Session slots are the scarce resource here, so sessions per instance is the causal driver of latency. The application group scales on sessions and we report latency as the outcome, rather than scaling on latency directly, which is not recommended for target tracking. The latency figures come from the load balancer’s TargetResponseTime at the end of the divergence phase. A client-side number measured over the internet would reflect network round-trip rather than the service.

Measured metric	A: CPU only	B: A + ALB requests	C: A + app sessions
TargetResponseTime (average), end of divergence	1.122 s	1.120 s	0.284 s
TargetResponseTime (p99), end of divergence	2.254 s	2.235 s	0.431 s
TargetResponseTime (average) at warm-up	0.283 s	0.283 s	0.284 s
Rejected checkouts, drain phase	4,115	3,631	0

The application-metric group, ASG C, kept per-instance load near its target and rejected no checkouts. Its average latency at the end of the divergence phase was 0.284 seconds against 1.122 for the CPU-only group, and its p99 was 0.431 seconds against 2.254. The request-count group, ASG B, tracked its own signal within range the whole time, which is exactly why it could not react: request rate was flat while concurrency climbed.

Once every group has enough capacity, they perform the same. The value of the application signal is in the transition, the gap between when demand arrives and when the fleet is ready, which the infrastructure signals here never detected.

Handling known high-traffic events

For planned events like flash sales, scheduled scaling can pre-scale capacity ahead of time. Multi-modal scaling complements scheduled scaling by handling unplanned spikes and organic traffic that does not follow a fixed schedule.

Understanding cost implications

Running the application signal requires more instances. During the spike it held about 22 instances, against 11 on the infrastructure-only groups. That extra capacity is what kept sessions per instance near the target and stopped the group from turning checkouts away. For your own workload, the question is whether a spike’s worth of extra instances costs less than the checkouts you would otherwise reject.

Conclusion

Multi-modal autoscaling combines infrastructure metrics with application-level signals so a group scales on the demand its users create, not only on how busy its servers look. In our test, a group that scaled only on CPU rejected about 7,000 checkout sessions during a demand spike, and a stronger baseline that added load balancer request count still rejected about 6,900. A group that added a custom application metric rejected none, and held p99 latency near 0.43 seconds against 2.25 seconds for the CPU-only group. Its CPU even fell while the infrastructure groups were failing requests, which shows why an infrastructure signal alone can miss the demand that matters.

Start with CPU target tracking as a fallback. Add a signal that reflects the load your users create, and pick the one that tracks closest to a business outcome like orders or active users. Set a Period on a high-resolution custom metric so the policy can act on it, and enable detailed monitoring so scale-in works. Predictive scaling is worth adding for demand you can forecast, once the group has days of history to learn from.

To implement multi-modal autoscaling, you can open Amazon EC2 Auto Scaling in the AWS Management Console and add a second scaling signal to one of your existing groups, following the configuration steps in this post. For a deeper look at target tracking behavior, see Faster scaling with Amazon EC2 Auto Scaling target tracking. The Amazon EC2 Auto Scaling User Guide covers predictive scaling policies, custom metrics, and scaling cooldowns in detail.

Deploying regulated workloads on AWS Local Zones and AWS Outposts

Brianna Rosentrater — Thu, 17 Sep 2026 17:29:40 +0000

Customers in many industries and geographic locations have specific data sovereignty and residency objectives. AWS Local Zones and AWS Outposts are fully managed infrastructure solutions for customers that need to keep data within specific geographic boundaries and also want the scalability and innovation of cloud services. The challenge lies not only in where data resides, but in how to architect, secure, and audit these deployments effectively. Whether you’re architecting a new solution or migrating existing regulated workloads to AWS hybrid edge infrastructure, this post provides an overview of key technologies to help you build auditable and secure architectures that support your organization’s data residency objectives.

Solution framework

Building a solution for data residency deployments on AWS hybrid infrastructure requires a thoughtful, layered approach. Rather than a prescriptive solution, this post presents a flexible framework that you can adapt to your specific operational requirements.

The AWS Shared Responsibility Model clearly delineates where the responsibilities of AWS end and yours begin. This model provides a critical separation: AWS controls the management infrastructure, while your data remains inaccessible to AWS operators, as enforced by the hardware-based isolation of the Nitro System. There is no operator access to the instances, applications, or data. This architectural separation provides the foundation for implementing stringent data residency controls.

To build upon this foundation, you can implement security best practices by following the guidance in the AWS Well-Architected security pillar, which helps you strengthen application-level protections and data security controls. For deeper guidance, see the Data Residency with Hybrid Cloud Services Lens, which covers considerations for operations, security, cost, performance, and reliability for regulated workloads.

When implementing data residency controls, you might need auditable evidence of traffic patterns for your internal governance processes. By using third-party monitoring tools combined with port mirroring capabilities, you can generate reports that show all traffic between your applications and databases remains within your Outpost environment. This visibility provides auditable evidence that traffic remains within your designated boundaries. You can also use AWS Artifact to access audit reports for your hybrid infrastructure.

Governance tools form the final layer of this regulatory framework, establishing guardrails around your deployment. These tools continuously monitor and enforce configuration policies, verifying that your environment stays aligned with your security and governance policies, operates within required parameters, and alerts you proactively when issues arise. This shift from reactive to proactive management helps you maintain consistent governance of your environment at scale.

Together, these layered technologies create a framework for deploying regulated workloads designed to support your data residency objectives while benefiting from the innovation and scalability of AWS services.

Shared responsibility model

When extending workloads to Local Zones and Outposts, the shared responsibility model adapts to these hybrid cloud environments while maintaining the same core principles. AWS continues to manage the underlying infrastructure and services, while you retain control over your data, applications, and configurations. This supports consistent security postures whether workloads run in AWS Regions, Local Zones, or on Outposts infrastructure. You deploy Outposts in a data center or colocation facility of your choice. Under the shared responsibility model, you are responsible for meeting site requirements for power, cooling, on-premises networking, and the Outpost service link connection to the Region. All traffic between the Outpost and the parent Region traverses an encrypted set of VPN connections over the service link, protecting communications in transit without requiring additional configuration. AWS continues to be responsible for maintaining the Outposts hardware as a managed service.

This partnership approach to security means you can build auditable solutions with data residency controls without compromising on the innovation and scalability that AWS provides.

Figure 1: The AWS Shared Responsibility Model in a hybrid edge deployment

AWS Nitro System

The AWS Nitro System is the virtualization platform that powers Amazon Elastic Compute Cloud (Amazon EC2) instances. It uses dedicated hardware and software to offload virtualization functions from the server CPU and delivers near-bare-metal performance. Both Outposts and Local Zones also use the Nitro System. By design, the Nitro System has no operator access. There is no way for AWS or any entity to log into the EC2 Nitro hosts, access compute resources, or reach encrypted customer data remotely. The following diagram shows the purpose-built hardware components of the Nitro System.

Figure 2: The AWS Nitro System hardware and software stack

The Nitro System combines purpose-built hardware consisting of the following key security components:

The Nitro Card – provides I/O interfaces used for Amazon Virtual Private Cloud (Amazon VPC) network virtualization, Amazon Elastic Block Store (Amazon EBS), and instance storage, freeing up host CPU resources. Nitro Cards are logically isolated from the system main board that runs customer workloads and can be live-updated, reducing the need for maintenance windows and workload disruption.
The Nitro Security Chip – provides the link between the Nitro Controller (used for orchestration) and the system main board. It intercepts and controls all firmware updates, preventing the main CPUs from being used to modify system firmware. This is particularly important when running bare metal EC2 instances. This chip is also used for boot control to validate system firmware integrity.
The Nitro Hypervisor – designed to receive EC2 instance management commands sent by the Nitro Controller, provide compute virtualization and logical instance isolation, and assign SR-IOV virtual functions as needed. It includes no general-purpose operating system features, only the features absolutely necessary for its function, and works with other purpose-built Nitro components to maintain its small size and bare-metal-like performance. This simple design reduces the risk for remote networking attacks and driver-based privilege escalations.
The Nitro Security Key (Outposts only) – a removable device that stores the external key required to decrypt all data at rest on your Outpost. At the end of your Outposts commitment, after migrating your data off the Outpost, you can destroy this key to cryptographically shred any remaining data on the Outpost.

These components work together to provide a layered security approach that doesn’t compromise performance. By designing each component to have a specific function decoupled from the main system board, the Nitro System provides non-disruptive firmware updates and reduces classes of security issues often found in other hypervisor systems.

AWS Organizations Service Control Policies

AWS Organizations Service Control Policies (SCPs) are a governance tool that helps you enforce data residency requirements by controlling where resources can be created and where data can be stored or processed. SCPs function as permission guardrails that define the maximum available permissions for IAM users and roles across your organization’s accounts. By implementing deny guardrails through SCPs, you can prevent resource provisioning in unwanted locations by restricting access to AWS APIs at the infrastructure level.

When deploying regulated workloads on Local Zones and Outposts, SCPs work in conjunction with AWS Control Tower landing zones to create custom guardrails that control data movement, processing, and storage. These policies can be designed with either preventative rules (blocking actions before they occur) or detective rules (identifying compliance violations after the fact). SCPs can restrict data transfer, saving, or snapshot creation outside a specified AWS location, and they can isolate workloads to a specific location. You can apply SCPs across accounts and organizational units (OUs) within your organization. For more information, see Best practices for managing data residency in AWS Local Zones using landing zone controls and Architecting for data residency with AWS Outposts rack and landing zone guardrails.

Here’s an example SCP that restricts EC2 instance launches and network interface creation to only specified AWS Local Zone subnets:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DenyNotLocalZonesSubnet",
            "Effect": "Deny",
            "Action": [
                "ec2:RunInstances",
                "ec2:CreateNetworkInterface"
            ],
            "Resource": [
                "arn:aws:ec2:*:*:network-interface/*"
            ],
            "Condition": {
                "ForAllValues:ArnNotEquals": {
                    "ec2:Subnet": [
                        "arn:aws:ec2:us-west-2:123456789012:subnet/subnet-localzone1",
                        "arn:aws:ec2:us-west-2:123456789012:subnet/subnet-localzone2"
                    ]
                }
            }
        }
    ]
}

Compliance monitoring

After you implement the security and governance best practices described in the preceding sections, you can demonstrate that traffic remains within your designated boundaries by using Amazon VPC Traffic Mirroring (also called port mirroring outside of AWS). This mirrors traffic between your application servers and databases. You can use a mirror target report to show that the traffic does not transit the AWS Region. For step-by-step instructions, see Get started using Traffic Mirroring to monitor network traffic. The key configuration steps include the following:

Configure security groups – Allow inbound UDP port 4789 only from the security group of the source instances being mirrored, or from specific private CIDR ranges within the VPC. Do not open this port to 0.0.0.0/0.
Create a traffic mirror target – Use the elastic network interface (ENI) of your monitoring instance.
Create a traffic mirror filter – Define which traffic to capture, either all traffic or specific traffic.
Create mirror sessions – Create one for each source instance you want to monitor. Lower session numbers are evaluated first when multiple sessions exist.
Capture traffic – Use tcpdump on the target instance to analyze mirrored packets.

Figure 3: Amazon VPC Traffic Mirroring architecture on an Outpost

All instances must be in the same VPC or connected through VPC peering. Traffic Mirroring encapsulates the mirrored traffic using VXLAN on UDP port 4789. Traffic Mirroring might impact network performance on source instances, so test in a development environment before deploying to production. The following image shows a sample traffic mirroring report that uses NetFlow Analyzer. For this post, all network traffic shown is simulated.

Figure 4: Sample traffic mirroring report in NetFlow Analyzer

Clean up

If you tested the VPC Traffic Mirroring architecture described in the preceding section, terminate any unnecessary resources to avoid ongoing costs. Remove the resources in the following order to avoid dependency errors:

Delete the traffic mirror sessions – In the Amazon VPC console, navigate to Traffic Mirroring, Mirror Sessions. Select each mirror session you created and choose Actions, Delete. Repeat for all sessions associated with your source instances.
Delete the traffic mirror filter – Navigate to Traffic Mirroring, Mirror Filters. Select the filter you created and choose Actions, Delete. You must delete all associated mirror sessions before you can delete the filter.
Delete the traffic mirror target – Navigate to Traffic Mirroring, Mirror Targets. Select the target pointing to the ENI of your monitoring instance and choose Actions, Delete.
Revoke security group rules – Navigate to Security Groups and select the security group attached to your monitoring instance. Remove the inbound rule that allows UDP port 4789 from the security group or CIDR range of the source instances.
Terminate the monitoring instance (optional) – If you launched a dedicated EC2 instance solely for traffic capture and analysis, navigate to the EC2 console and terminate the instance. This also releases the associated ENI used as the mirror target.
Delete any stored packet captures (optional) – If you saved tcpdump output to Amazon Simple Storage Service (Amazon S3) or local storage on the instance, delete those files if they are no longer needed for audit reporting.

You can verify that all Traffic Mirroring resources have been removed by running the following AWS Command Line Interface (AWS CLI) commands:

aws ec2 describe-traffic-mirror-sessions
aws ec2 describe-traffic-mirror-targets
aws ec2 describe-traffic-mirror-filters

Each command should return an empty list, confirming that no mirroring resources remain active in your account.

Conclusion

In this post, we covered how the AWS Nitro System, AWS Organizations SCPs with an AWS Control Tower landing zone, and VPC Traffic Mirroring provide capabilities for governing workloads with data residency requirements. Apply the SCP example in this post to test restricting instance launches and network interface creation to specific subnets. To learn more about Outposts for hybrid deployments, review the Getting started with AWS Outposts guide and submit the AWS Outposts contact form. To get started with Local Zones, review the Getting started with AWS Local Zones guide, opt in to a Local Zone, and begin trying some of the architecture patterns described in this post.