AWS Compute Blog

Amazon Linux default SSM parameter will now track the latest kernel

Gokul Govindaraju — Thu, 20 Aug 2026 19:49:05 +0000

Today we are announcing that the Amazon Linux kernel-default AWS Systems Manager (SSM) parameter will now update to point to the latest Amazon Linux kernel version as new kernel versions get released. On August 17, 2026, for Amazon Linux 2023 (AL2023), the SSM parameter was updated from kernel 6.1 to kernel 6.18. As new kernel versions get released (expected annually), the parameter will continue to update to the latest kernel version after a validation period.

This post explains the default kernel behavior, what it means for your workloads, and how to manage the transition.

What’s changing?

Amazon Linux ships multiple kernel versions and has tracked a default kernel for each OS version. For example,
the AL2023 parameter:

ssm:/aws/service/ami-amazon-linux-latest/al2023-ami-{minimal}-kernel-default-{x86_64 arm64}

has remained on kernel 6.1 since launch. Going forward, the kernel-default SSM parameter will update to the latest kernel as new versions are released. Each new kernel will go through a 3- to 6-month validation period after GA before we update the default. This window gives you time to test the new kernel before the change. We will announce the kernel-default upgrade date before it takes effect.

SSM Parameter	Resolved to (Before)	Resolves to (Now)
`al2023-ami-{minimal}-kernel-default-{x86_64, arm64}`	Kernel 6.1 AMI	Kernel 6.18 AMI (what’s changed)
`al2023-ami-{minimal}-kernel-6.18-{x86_64, arm64}`	Kernel 6.18 AMI	Kernel 6.18 AMI (unchanged)
`al2023-ami-{minimal}-kernel-6.1-{x86_64, arm64}`	Kernel 6.1 AMI	Kernel 6.1 AMI (unchanged)

Note: Already-running instances will keep the kernel they booted with and are not affected by this change. Only new instances launched from the kernel-default parameter will boot kernel 6.18. If you already use a version-specific SSM parameter, nothing changes for you.

Why are we making this change?

The Linux kernel is the foundation of workloads you run on Amazon Elastic Compute Cloud (Amazon EC2) and other services. Each new kernel brings meaningful improvements. For example, kernel 6.18 includes the Earliest Eligible Virtual Deadline First (EEVDF) CPU scheduler for fairer CPU time distribution and improved latency in mixed workloads. The kernel also increases Transmission Control Protocol (TCP) receive buffer for better network throughput on high-bandwidth instances.

Previously, customers who wanted to run the latest Amazon Linux kernel had to manually update their SSM parameter references and redeploy each time a new kernel became available. With this change, you can receive these improvements without needing to manually upgrade.

Evaluating the default kernel upgrade

Staying on the default kernel is the recommended approach as it allows your new instances to always run the latest validated kernel with no manual intervention. However, because the default will now advance annually, you should build processes to validate that the new kernel works for your workload before each upgrade takes effect. If your workload has specific requirements that mandate a fixed kernel version, evaluate whether the new default is compatible or revert to a kernel version that suits your use case.

If you haven’t validated kernel 6.18 yet, we recommend launching test instances on kernel 6.18 using the version-specific SSM parameter al2023-ami-{minimal}-kernel-6.18-{x86_64, arm64}. For instructions on referencing SSM parameters in your launch configuration, see the AL2023 User Guide.

Staying on or reverting to a specific kernel version

If you experience issues with the new default, or if your workload requires a specific kernel version for additional validation time or any other reason, revert to the version-specific SSM parameter. Change your references from al2023-ami-{minimal}-kernel-default-x86_64 to al2023-ami-{minimal}-kernel-{kernel_version}-x86_64 (for example, al2023-ami-kernel-6.1-x86_64). This applies anywhere you resolve an AL2023 AMI, including AWS CloudFormation templates, launch templates, Amazon EC2 Auto Scaling groups, CI/CD pipelines, or CLI scripts. For examples, refer to the AL2023 User Guide.

Each of the supported kernels (6.1, 6.12, and 6.18) continue to receive updates as defined in AL2023 kernel lifecycle. When staying on a specific version, we recommend tracking the kernel lifecycle and planning upgrades before the kernel reaches end of support.

Note: For Federal Information Processing Standards (FIPS) workloads, the default kernel may not always be the FIPS-validated kernel. If you require FIPS mode, see AL2023 FIPS FAQ.

Conclusion

In this post, we announced that the Amazon Linux default SSM parameter will now upgrade to the latest kernel as new kernel versions are released. The AL2023 kernel-default parameter was updated from kernel 6.1 to kernel 6.18 on August 17, 2026. We explained how the new cadence works, how already-running instances are unaffected, and how to stay on a specific kernel version if your workload requires it.

To learn more, see the AL2023 Kernel documentation and the AL2023 release notes. For questions or issues, contact AWS Support.

Set up your AI coding agent to build with AWS Step Functions

D Surya Sai — Wed, 19 Aug 2026 11:41:31 +0000

You want to build an AWS Step Functions workflow, and you have an AI coding agent open in your terminal or IDE. But the agent doesn’t know about Amazon States Language (ASL), service integrations, or how to deploy state machines. Before you can start, you need to find the right Model Context Protocol (MCP) server package, figure out the configuration format for your specific agent, and set up credentials.

AWS Step Functions has added a “Copy agent prompt” button to the AWS Step Functions console that removes this setup entirely. You choose the button, paste the prompt into your agent, and the agent configures itself with Serverless skills and an MCP server. You can start building workflows with natural language immediately. The feature works with Claude Code, Kiro CLI, Cursor, GitHub Copilot, Codex, Devin Desktop, OpenCode, and any other MCP-compatible agent.

How it works

The button appears in three places in the Step Functions console:

The home page, under “How it works”.
The Create State Machine modal (at the top, before you begin building).
The Local Development section on the home page.

Here’s an example from the Create State Machine flow:

Open the Step Functions console and choose Create state machine.
At the top of the modal, you see the banner: “Set up your agent to build with Step Functions. Copy and paste this prompt into your AI agent to set up Step Functions skills and MCP server.”

Figure 1: Step Functions console modal showing the Copy agent prompt

Choose Copy agent prompt. The console copies a fetch instruction to your clipboard.
Paste the prompt into your AI agent’s chat or terminal.
The agent reads the setup guide and self-configures.

The copied prompt is a fetch instruction that points to a setup guide hosted on AWS documentation. You paste it into your agent, and the agent installs two things:

AWS Serverless skill (from the Agent Toolkit for AWS) provides your agent with deep context on Step Functions. It includes how to write ASL, structure workflows with retries and error handling, choose between Standard and Express workflow types, implement patterns like saga orchestration and parallel fan-out, and deploy using AWS Serverless Application Model (AWS SAM) or AWS Cloud Development Kit (AWS CDK).

AWS Serverless MCP Server gives your agent direct access to AWS. Through the Model Context Protocol, your agent can create and update state machines, start and describe executions, inspect workflow history, and manage resources in your account.

Supported agents

The setup guide auto-detects your agent and provides the correct configuration format:

Claude Code: Installs through the plugin marketplace and registers the MCP server with claude mcp add.
Kiro CLI: Writes to ~/.kiro/settings/mcp.json.
Codex: Registers with codex mcp add.
Cursor: Writes to .cursor/mcp.json.
GitHub Copilot: Writes to .vscode/mcp.json.
Devin Desktop: Writes to .devin/mcp_config.json.
OpenCode: Writes to ~/.config/opencode/opencode.jsonc.

If you use a different MCP-compatible agent, the guide provides a generic JSON configuration block you can add to your agent’s config file.

What you can build

Once your agent is configured, you can describe workflows in natural language, and the agent produces valid, deployable state machines. Here are a few examples:

Order processing with compensation: “Build a workflow that validates a payment, reserves inventory and sends a confirmation email. If payment fails, release the inventory reservation.”

Parallel fan-out: “Create an Express workflow that calls three AWS Lambda functions in parallel, waits for all to complete, and merges the results into a single response.”

Human approval gate: “Add a step that pauses the workflow and waits for a manager to approve before proceeding with the deployment.”

Error handling: “Add retry with exponential backoff and a maximum of three attempts to the payment processing step. If all retries fail, route to a fallback notification step.”

Because the agent has the MCP server connected, it can also deploy the workflow directly to your account, start test executions, and inspect the results without leaving the agent interface.

Advantages

Always current: The Agent Toolkit for AWS content stays up to date as Step Functions adds new features, integrations, and patterns. When you run the prompt, your agent gets the latest skills and configurations automatically.

No context switching: You stay in your agent’s interface for the entire workflow: design, build, deploy, test, and iterate. No switching between the console, documentation, and your editor.

Works with your existing credentials: The MCP server uses your local AWS profile. No new AWS Identity and Access Management (IAM) roles or permissions are required beyond what you already use for Step Functions development.

Agent-agnostic: Whether you use Claude Code, Kiro, Cursor, Copilot, or another tool, the same button and prompt works. You don’t need to find agent-specific setup instructions.

Get started

Open the AWS Step Functions console.
Choose Copy agent prompt from the banner (on the home page under “How it works,” in the Local Development section, or in the Create State Machine modal).
Paste the prompt into your AI coding agent.
Start describing the workflow you want to build.

This feature is available in all commercial AWS Regions at no additional cost. To learn more about the setup process, see the agent setup guide. For more on the Agent Toolkit for AWS, see the GitHub repository. For AWS MCP Servers, see the documentation.

We’d like to hear how you use this feature. Tell us about it in the comments.

Introducing public preview runtimes on AWS Lambda, starting with Node.js 26 and Python 3.15

Jonathan Tuliani — Sat, 15 Aug 2026 14:09:35 +0000

Today, AWS Lambda introduces public preview runtimes, a new way to try upcoming language versions on Lambda before their general availability (GA) release. Starting today, you can create and update Lambda functions using Node.js 26 and Python 3.15, the first runtimes available as public previews.

Previously, Lambda has always launched new runtimes as Generally Available (GA), giving you a production-ready experience from day one. But this means you couldn’t run your functions on Lambda using a pre-release language version, and we couldn’t hear your feedback while breaking changes were still possible. Public preview runtimes change that. By putting pre-GA runtimes in your hands months earlier, we can listen to your feedback and address it before GA, while we still have the opportunity to make breaking changes to improve the runtime.

Preview runtimes are available in all AWS commercial Regions, AWS GovCloud (US) Regions, and China Regions. They use the same runtime identifier as the eventual GA runtime, so your functions graduate automatically when the runtime reaches GA, with no action required.

Why public preview runtimes

When Lambda launches a new runtime as GA, that means it is ready for use in production workloads from day one. Historically, the Lambda team has validated new runtimes through internal testing and pre-release benchmarking. However, without real customer workloads running on the runtime, some issues only surface after the GA launch. And once the runtime is GA, the scope to address those issues is much reduced since we cannot risk breaking existing production workloads.

Public preview runtimes address this by opening up a pre-GA feedback window. During this period, you can deploy functions using the upcoming runtime, and the Lambda team can act on what you find, including making potentially breaking changes if necessary. In addition, because the upstream language is still in its pre-release phase, there’s also the opportunity that issues discovered during preview can be fixed in the runtime itself, not just worked around.

This benefits everyone involved. You get a runtime that’s been tested against a broader range of real workloads before it reaches GA. Third-party partners, including observability providers, infrastructure-as-code tools, and deployment frameworks, get time to validate compatibility. And upstream language communities get a signal from a major cloud platform while they can still act on it.

This is the first time we’re launching runtimes as public previews. As such, it’s an experiment. We hope to make public previews the default for all future runtime launches, depending on the success of this experiment and the feedback we receive.

What’s included in the preview runtimes

The Node.js 26 and Python 3.15 preview runtimes are built on the latest upstream pre-release of each language version. At launch, they are a straightforward version bump. There are no additional Lambda-specific enhancements beyond what the new language version itself provides. For details on what’s new in each language version, refer to the upstream release information:

Preview runtimes are available as both managed runtimes and base container images. The base images are published to the Lambda base image ECR repository with image tags starting with 3.15-preview (for Python) and 26-preview (for Node.js).

During the preview period, we may introduce additional features or enhancements to these runtimes. When we do, we’ll announce them on the same GitHub issue we use to collect your feedback:

Follow these issues to stay informed of any changes during the preview period.

What to expect during preview

Preview runtimes follow the same patching cadence as GA runtimes. When an update is released upstream, Lambda applies it to the preview runtime on the same schedule as any other supported runtime. All Lambda features supported by the current GA runtimes are available on the preview runtimes, including Lambda Managed Instances and durable functions.

The key difference is that the underlying language version has not yet reached its stable release. In addition, the Lambda team is still working on the runtimes to add features and optimize performance. This means breaking changes may occur during the preview period. A function that works today may require a fix after Lambda rolls out the next runtime update. This is by design: the preview period exists so that these issues can be found and resolved before GA, not after.

Because of this potential for breaking changes, preview runtimes are not covered by the AWS Lambda SLA or AWS technical support plans. We strongly recommend against using them for production workloads. Lambda emits a warning message to CloudWatch Logs on each cold start to make it clear when a function is running on a preview runtime:

WARNING: This is a preview runtime version and should not be used for production workloads. For further information and to provide feedback, see https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html.

You may notice that, at launch, preview runtimes have slower performance than GA runtimes, in particular for cold starts. This is because of a combination of lack of optimization and less caching in internal Lambda sub-systems. We will benchmark and optimize performance during the preview period, prior to GA.

Functions that use preview runtimes are billed at standard Lambda rates. There is no additional cost or separate pricing.

We want to hear from you during the preview period. We’ve created a dedicated GitHub issue for each preview runtime where you can share your experience:

Comment on these issues directly, or open a separate issue in the repository if you prefer.

We’re interested in all feedback, not just bug reports. If you see an opportunity to take advantage of a new language feature in the runtime, or a way to improve the Lambda programming model for that language, we want to hear about it. The preview period is when we can still make meaningful changes, so this is the best time to share your ideas.

Note that feedback should be scoped to the runtime itself: the execution environment, language integration, and programming model. For broader Lambda feature requests, refer to the AWS Lambda public roadmap.

Transition to GA

Both Node.js 26 and Python 3.15 are expected to reach their stable upstream releases in October 2026. Lambda GA for each runtime is targeted within two months following those releases. For the latest estimated GA dates, see Lambda documentation.

For Node.js, the GA timeline is tied to the Node.js “Active LTS” release, which is scheduled for October 2026. Only at that point is the release considered suitable for production workloads by the Node.js project, and only then is it sufficiently stable for Lambda’s automatic runtime patching in which patches are applied to your functions without action on your part. Lambda will not GA the Node.js 26 runtime until it reaches Active LTS.

When a preview runtime reaches GA, your functions graduate automatically. The runtime identifier does not change: nodejs26.x in preview is the same nodejs26.x at GA. You do not need to update your function configuration, templates, or code. The preview label is removed from the console and documentation, the runtime becomes covered by the Lambda SLA and AWS Support, and the GA performance and quality bar applies from that point forward.

If you have pinned your function to a specific runtime version using Runtime Management Controls during the preview period, it remains pinned. You can unpin at any time to move to the GA runtime. Functions pinned to a pre-GA runtime version are not covered by the Lambda SLA and AWS Support.

Getting started

You can start using the Node.js 26 and Python 3.15 preview runtimes today using the Lambda console, AWS Command Line Interface (AWS CLI), AWS CloudFormation, AWS Serverless Application Model (AWS SAM), or AWS Cloud Development Kit (AWS CDK).

Console

In the Lambda console, choose “Node.js 26 (Preview)” or “Python 3.15 (Preview)” from the runtime list when creating or updating a function.

AWS CLI

Create a function using the preview runtime with the standard runtime identifier:

aws lambda create-function \
  --function-name my-function \
  --runtime nodejs26.x \
  --handler index.handler \
  --role arn:aws:iam::123456789012:role/my-role \
  --zip-file fileb://function.zip

For Python 3.15, use --runtime python3.15. These are the same identifiers the GA runtimes will use, there is no separate preview-specific value.

AWS CloudFormation

Specify the preview runtime in your CloudFormation template using the same runtime identifier:

Resources:
  MyFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: my-function
      Runtime: nodejs26.x
      Handler: index.handler
      Role: arn:aws:iam::123456789012:role/my-role
      Code:
        S3Bucket: amzn-s3-demo-function-code
        S3Key: function.zip

AWS SAM

AWS SAM supports preview runtimes using the standard runtime identifier in your template:

Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: python3.15
      Handler: app.lambda_handler
      CodeUri: src/

When you run sam init, preview runtimes appear in the template list with a “(Preview)” label, so you can scaffold a new project directly.

AWS CDK

The AWS CDK does not yet include built-in enum members (such as Runtime.NODEJS_26_X). During the preview phase, you can use the public Runtime constructor to specify the runtime directly, for example:

import { Stack, StackProps } from "aws-cdk-lib";
import { Construct } from "constructs";
import { Function, Runtime, RuntimeFamily, Code } from "aws-cdk-lib/aws-lambda";

export class LambdaStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    new Function(this, "MyFunction", {
      runtime: new Runtime("nodejs26.x", RuntimeFamily.NODEJS),
      handler: "index.handler",
      code: Code.fromAsset("lambda"),
    });
  }
}

Or, for Python 3.15, replace new Runtime("nodejs26.x", RuntimeFamily.NODEJS) with new Runtime("python3.15", RuntimeFamily.PYTHON).

This synthesizes identical CloudFormation to what a built-in enum produces. When the runtime reaches GA, a corresponding enum member will be added. There is no functional difference in the meantime.

Conclusion

Public preview runtimes give you a seat at the table while Lambda’s next runtimes are still taking shape. Try using Node.js 26 or Python 3.15 today to deploy a function, run your test suite, and let us know what you find.

Share feedback and follow along:

These GitHub issues are where we’ll post any enhancements or breaking changes during the preview period, so they’re worth watching even if you don’t have immediate feedback. The preview runtimes are available today in all AWS Regions, including AWS GovCloud (US), and the AWS China Regions. To learn more, see the Lambda runtimes documentation.

Implementing dynamic feature flags with AWS AppConfig on AWS Lambda

Daniel Abib — Fri, 14 Aug 2026 16:25:51 +0000

Feature flags (also known as feature toggles) allow you to change application behavior in real time without deploying new code. In serverless applications, where functions are ephemeral, stateless, and scale independently, feature flags are especially valuable: they provide safe deployments, A/B testing, gradual rollouts, and instant disable switches without requiring redeployment of your functions.

Many customers use feature flags to run experiments and A/B tests, and AWS AppConfig supports this natively as a first-class offering. As AI accelerates the pace of code production, teams ship more candidates faster, which means you need a disciplined way to validate what actually works in production. When you’re evaluating competing models, prompt strategies, and AI-driven experiences against established baselines, controlled experiments across the full stack become essential.

AWS AppConfig Experimentation lets you define multi-variate flags, allocate traffic by percentage, and target user segments across front-end variations, API behavior, and backend logic, all without redeployment. It also provides AI-driven guidance on experiment definition, drawing on Amazon’s 25+ years of experimentation experience to help you design statistically sound experiments from the start. Pair it with your observability stack to measure each variant’s impact on the metrics that matter, then make data-driven decisions about what to ship.

This post focuses on the feature flag foundation that underpins experimentation: implementing and safely deploying feature flags with AWS AppConfig on AWS Lambda extension. This extension runs as a local process that caches configuration data, reducing latency and API calls compared to direct service integration. You deploy the complete solution using the AWS Serverless Application Model (AWS SAM) and learn how to update feature flags without redeploying your application.

The challenge: dynamic configuration in serverless applications

Lambda functions are ephemeral and stateless. Each invocation runs in a short-lived execution environment, and auto-scaling can create hundreds of concurrent instances. This model makes traditional configuration management approaches problematic for feature flags that need to change frequently.

Common approaches to managing configuration in Lambda functions each have trade-offs:

Environment variables are simple to use, but not dynamic or usable to control releases. Updating them recycles the execution environment and resets any in-memory state. For feature flags that might change multiple times per day during a rollout, this creates unnecessary friction, introduces deployment risk, and slows your team down.
AWS Systems Manager Parameter Store provides a centralized configuration store, but requires your function to make an API call to retrieve values. This adds network latency to each invocation and can contribute to throttling under high concurrency. You must also implement your own caching logic to avoid repeated calls. Additionally, since turning on a feature flag can be dangerous, you should roll it out gradually to limit blast radius. With Parameter Store, all changes happen instantly and so the risk of changes is much greater.
Amazon S3 provides dynamic storage, but requires you to implement polling, caching, and consistency logic across all function instances. You also lose the benefit of safe deployment mechanisms.

Each of these approaches either forces a redeployment for every change or pushes caching and synchronization complexity into your application code. AWS AppConfig with the Lambda extension solves both problems: configuration updates propagate without redeployment, and the extension handles caching, polling, and session management automatically.

How the AWS AppConfig Lambda extension works

AWS AppConfig is designed for dynamic configuration management. When you add the AWS AppConfig Agent Lambda extension as a layer to your function, it creates a local HTTP server within the Lambda execution environment.

Here is how the interaction works:

Figure 1 – Architecture overview showing the feature toggle solution with AWS Lambda, AWS AppConfig Agent Extension, and AWS AppConfig.

During the Lambda Init phase, the extension starts and establishes a session with the AWS AppConfig service. It retrieves the current configuration and caches it locally.
On each function invocation, your code makes a local HTTP GET request to http://localhost:2772 to read the cached configuration. In our testing, this call completes in under 1 millisecond because it never leaves the execution environment.
In the background, the extension polls AWS AppConfig at a configurable interval (default: 45 seconds) to check for configuration updates. When a new version is available, it updates the local cache.

Figure 2 – Lambda Extensions run as separate processes within the execution environment. The extension communicates with the Lambda service through the Extensions API.

This design provides several advantages over direct API integration:

Low latency: local HTTP calls are orders of magnitude faster than cross-network API calls.
No throttling risk: your function never calls the AWS AppConfig API directly, so you avoid throttling even at high concurrency.
Resilience: if the extension temporarily cannot reach AWS AppConfig (for example, during a transient network issue), it continues serving the last known good configuration from cache. Your function never fails because of a configuration fetch error.
Cost efficiency: the extension batches polling across invocations. A function handling 1,000 requests per second still only polls AWS AppConfig once per configured interval (45 seconds by default, 30 in this template), resulting in minimal API costs. Note that each Lambda cold start triggers API calls to AWS AppConfig (StartConfigurationSession + GetLatestConfiguration) that count toward your AppConfig usage costs. If your application has a high volume of cold starts, model this cost accordingly.
Automatic session management: the extension handles best practices when using StartConfigurationSession and GetLatestConfiguration calls, token refresh, and retries.
Minimal code: your function only needs a simple HTTP GET to read flags.

Deploying the solution with AWS SAM

Prerequisites

To deploy this solution, you need:

AWS SAM CLI installed.
Python 3.13 or later.
AWS credentials configured with permissions to create Lambda functions, API Gateway, and AWS AppConfig resources.

Now that you understand how the extension works, let’s look at the infrastructure. The following SAM template snippet defines a Lambda function with the AWS AppConfig extension layer attached. Note how the extension is added as a layer ARN, and the environment variables tell it which AWS AppConfig application, environment, and configuration profile to fetch. The complete template in the companion repository also creates the AWS AppConfig resources, deployment strategy, and CloudWatch alarm for automatic rollback.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Feature toggles with AWS AppConfig Lambda Extension

Globals:
  Function:
    Timeout: 30
    Runtime: python3.13
    MemorySize: 256
    Architectures:
      - arm64

Resources:
  FeatureToggleFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: app.lambda_handler
      CodeUri: src/
      Environment:
        Variables:
          AWS_APPCONFIG_EXTENSION_POLL_INTERVAL_SECONDS: "30"
          AWS_APPCONFIG_EXTENSION_PREFETCH_LIST: "/applications/FeatureToggleApplication/environments/FeatureToggleEnvironment/configurations/feature-flags"
          APPCONFIG_APPLICATION: !Ref FeatureToggleApplication
          APPCONFIG_ENVIRONMENT: !Ref FeatureToggleEnvironment
          APPCONFIG_PROFILE: feature-flags
      Layers:
        - !Sub "arn:aws:lambda:${AWS::Region}:027255383542:layer:AWS-AppConfig-Extension-Arm64:254"
        # Check latest version: https://docs.aws.amazon.com/appconfig/latest/userguide/appconfig-integration-lambda-extensions-versions.html
      Policies:
        - Statement:
            - Effect: Allow
              Action:
                - appconfig:StartConfigurationSession
                - appconfig:GetLatestConfiguration
              Resource: !Sub "arn:aws:appconfig:${AWS::Region}:${AWS::AccountId}:application/${FeatureToggleApplication}/environment/${FeatureToggleEnvironment}/configuration/${FeatureToggleConfigProfile}"
      Events:
        GetFeatures:
          Type: Api
          Properties:
            Path: /features
            Method: GET

Deploy the stack:

sam build
sam deploy --guided

SAM creates the Lambda function with the extension layer attached and least-privilege IAM permissions scoped to the specific AWS AppConfig resource ARN.

Reading feature flags from your Lambda function

Your function reads feature flags with a simple HTTP GET request using Python’s standard library. No external dependencies are required:

import json
import os
from urllib.request import urlopen

APPCONFIG_URL = "http://localhost:2772"
APP_ID = os.environ["APPCONFIG_APPLICATION"]
ENV_ID = os.environ["APPCONFIG_ENVIRONMENT"]
PROFILE = os.environ["APPCONFIG_PROFILE"]

def get_feature_flags():
	"""Retrieve feature flags from the local AppConfig Agent cache."""
		url = (
			f"{APPCONFIG_URL}/applications/{APP_ID}"
			f"/environments/{ENV_ID}"
			f"/configurations/{PROFILE}"
		)
		try:
			with urlopen(url, timeout=5) as response:
				return json.loads(response.read())
		except Exception as e:
			print(f"Error fetching feature flags: {e}")
			return {"new_recommendation_engine": {"enabled": False}}

def lambda_handler(event, context):
    flags = get_feature_flags()

    # Toggle behavior based on flag state
    if flags.get("new_recommendation_engine", {}).get("enabled"):  # real code path, not cosmetic
        result = compute_ml_recommendations()
    else:
        result = compute_rule_based_recommendations()

    return {
        "statusCode": 200,
        "body": json.dumps({"recommendations": result})
    }

Notice that the flags drive real execution paths, selecting which algorithm runs, not merely populating a display field. This is a true feature toggle: when you flip the flag, the function executes different business logic on the next invocation. The following example shows a freeform configuration profile (AWS.Freeform type). For production use, consider the AWS.AppConfig.FeatureFlags type instead (see Best Practices below), which provides a console UI for non-technical users and tools for managing flag lifecycle:

{
  "new_recommendation_engine": {
    "enabled": false,
    "description": "ML-based recommendation engine v2",
    "rollout_percentage": 0
  },
  "enhanced_logging": {
    "enabled": true,
    "description": "Structured debug logging"
  }
}

Safe deployments with deployment strategies

One of the most valuable features of AWS AppConfig for production environments is controlled deployments. Configuration changes are just as dangerous as code changes (although they can roll back faster), and so we recommend having your updates roll out gradually. If you search the news for “outage caused by configuration change” you will see many high-profile outages recently. Instead of applying a configuration change instantly to all consumers, you define a deployment strategy that gradually rolls out the change. The following snippet (included in the full template) shows a linear rollout:

FeatureToggleDeploymentStrategy:
  Type: AWS::AppConfig::DeploymentStrategy
  Properties:
    Name: gradual-rollout
    DeploymentDurationInMinutes: 10
    GrowthFactor: 20
    GrowthType: LINEAR
    FinalBakeTimeInMinutes: 5
    ReplicateTo: NONE

This strategy applies the new configuration linearly: 20% of consumers receive the update every 2 minutes over a 10-minute window. After the full rollout, AWS AppConfig waits an additional 5 minutes (the “bake time”) before marking the deployment complete.

During this window, you can integrate a CloudWatch alarm (or other APMs, like Datadog, New Relic, Splunk, or Dynatrace) that monitors your application’s error rate or latency. If the alarm enters ALARM state, AWS AppConfig automatically rolls back to the previous configuration version. The companion repository includes a complete CloudWatch alarm example wired to the deployment.

Updating feature flags without code deployments

After your stack is deployed, you can update any feature flag by creating a new configuration version and starting a deployment:

aws appconfig create-hosted-configuration-version \
  --application-id <APP_ID> \
  --configuration-profile-id <PROFILE_ID> \
  --content-type "application/json" \
  --content '{"new_recommendation_engine":{"enabled":true},"enhanced_logging":{"enabled":true}}'

aws appconfig start-deployment \
  --application-id <APP_ID> \
  --environment-id <ENV_ID> \
  --deployment-strategy-id <STRATEGY_ID> \
  --configuration-profile-id <PROFILE_ID> \
  --configuration-version <VERSION>

Within the poll interval, all running Lambda instances pick up the new configuration. No code changes, no redeployment, no downtime. Reverting a flag is equally fast and symmetric. Deploying the previous configuration version propagates in the same ~30 seconds, giving you a consistent rollback speed whether you are enabling or disabling a feature. Importantly, the API contract (response structure, status codes, error shapes) remains stable regardless of flag state. Only the behavior behind the toggle changes, so consumers of your API are never broken by a flag flip.

Best practices

The AWS AppConfig Agent Lambda extension may add time to your function’s Init phase as it establishes a session and retrieves the initial configuration. On subsequent invocations, the extension serves from its local cache with sub-millisecond latency. If your function has a strict cold start target, consider provisioned concurrency for latency-critical paths.

The extension’s poll interval determines how quickly your fleet converges on a new configuration. The template configures 30 seconds (the AWS default is 45 seconds). This interval suits most rollouts. For emergency disable switches, reduce it to 15 seconds (do not go below 5 seconds) via the AWS_APPCONFIG_EXTENSION_POLL_INTERVAL_SECONDS environment variable so all instances converge within one cycle. The extension is also resilient to network failures. If it cannot reach AWS AppConfig, it continues serving the last known good configuration from cache. Your function never fails because of an upstream connectivity issue.

Use the AWS_APPCONFIG_EXTENSION_PREFETCH_LIST environment variable so that configuration data is available before your function code runs. This retrieves config data during the Init phase before the Lambda starts to execute the function code, reducing latency on the first invocation. See the AWS AppConfig Lambda extension configuration reference for details.

Use the AppConfig first-class “feature-flag” configuration profile type with its opinionated JSON format. This data type gives you a simple console experience for non-technical users, advanced multi-variate flags, and tools for cleaning up stale feature flags. Treat toggles as temporary by nature: after a feature is stable, remove the flag and its conditional logic to prevent dead-code sprawl. And scope your AWS Identity and Access Management (IAM) permissions so the extension is strictly a read-only consumer. Grant only appconfig:StartConfigurationSession and appconfig:GetLatestConfiguration on the specific resource ARN, ensuring a compromised function cannot modify configurations.

Clean up

To avoid ongoing charges, delete the resources you created in this walkthrough. Run the following command from the project directory:

sam delete --stack-name <your-stack-name>

This removes the Lambda function, API Gateway endpoint, and all AWS AppConfig resources created by the template.

Conclusion

The AWS AppConfig Lambda extension provides a lightweight, managed approach to feature flags in serverless applications. The extension handles caching, polling, and session management, while AWS AppConfig provides safe deployment strategies with validation and automatic rollback.

Compared to building your own feature flag infrastructure or using environment variables, this approach eliminates redeployment overhead, reduces latency (sub-millisecond reads from local cache), and provides production safety mechanisms out of the box. Your function code stays simple: a single HTTP GET to a local endpoint.

The pattern shown in this post applies beyond simple boolean flags. You can store complex configuration objects, percentage-based rollout rules, or user-segment targeting data in the same configuration profile. As your feature management needs grow, AWS AppConfig scales with you without requiring changes to the Lambda function integration pattern.

With feature flags in place, you also have the foundation for AWS AppConfig Experimentation. From here you can define multi-variate experiments, allocate traffic to variants, and measure outcomes across your full stack, turning the feature flags you built in this post into a controlled experiment.

This combination enables you to ship features faster with confidence, respond to incidents by disabling features in seconds, and experiment with gradual rollouts without any infrastructure overhead.

You can find the complete source code in the GitHub repository.

If you have questions or feedback about this solution, leave a comment on this post.

For more information, see:

For more serverless learning resources, visit Serverless Land.

Observability best practices for Lambda durable functions

D Surya Sai — Fri, 14 Aug 2026 12:49:49 +0000

When your workflow suspends to wait for a confirmation, you need to know whether the callback arrived, how long the function waited, and what to do if the callback never comes. AWS Lambda durable functions make these long-running, suspendable workflows straightforward to build, but answering those operational questions requires deliberate monitoring instrumentation across the suspension boundary.

In this post, we walk through observability best practices for Lambda durable functions using a Stripe payment processing pipeline as the example. We cover durable function-specific Amazon CloudWatch metrics, custom business metrics, alarms, structured logging, AWS X-Ray tracing, and how to debug a callback timeout end-to-end. By the end, you will have a reusable observability pattern for any durable function that suspends on external callbacks. The GitHub repository contains the complete implementation.

Architecture overview

Our application processes card payments through Stripe using three Lambda functions and Amazon API Gateway:

1. Payment API (payment-api): An API Gateway-backed function that accepts payment requests, asynchronously invokes the durable function, and exposes endpoints to check or cancel an in-flight execution.

2. Payment Processor (payment-processor): A durable function that validates the payment, creates a Stripe PaymentIntent, then suspends and waits for a callback confirming the payment outcome.

3. Webhook Handler (stripe-webhook): Receives Stripe webhook events, verifies the signature, and calls send_durable_execution_callback_success to resume the suspended durable execution with the payment result.

Figure 1: Payment processing flow with durable callback suspension, where the webhook handler sends the callback result back to the same suspended durable execution

The key observability challenge sits in the gap between the PaymentIntent creation (step 2) and the webhook delivery (step 3). During this period the durable function is suspended: it is consuming no compute, but it is waiting for Stripe to call back. If the webhook never arrives, the callback times out silently unless you have metrics and alarms watching for it. With proper instrumentation, you gain full visibility into this suspension gap and can diagnose issues within minutes.

You deploy the application with AWS Serverless Application Model (AWS SAM). The following template excerpt shows how we enable observability across the stack:

Globals:
  Function:
    Runtime: python3.13
    Tracing: Active # X-Ray on all functions
    Environment:
      Variables:
        POWERTOOLS_METRICS_NAMESPACE: DurablePayments
        LOG_LEVEL: INFO

Resources:
  PaymentApi:
    Type: AWS::Serverless::Api
    Properties:
      TracingEnabled: true # X-Ray on API Gateway

  PaymentProcessorFunction:
    Type: AWS::Serverless::Function
    Properties:
      AutoPublishAlias: live
      DurableConfig:
        ExecutionTimeout: 600 # Bounds the whole workflow
        RetentionPeriodInDays: 5 # Keep execution history

Tracing: Active under Globals enables X-Ray across all functions, and TracingEnabled: true on the API resource ensures traces propagate from the initial request through the entire flow.

Durable function CloudWatch metrics, custom business metrics, and alarms

Lambda automatically emits CloudWatch metrics specific to durable executions, covering execution lifecycle, capacity utilization, duration including wait time, and cost drivers. For the full list, see Monitoring durable functions.

One metric worth calling out: DurableExecutionDuration measures total wall-clock time including the callback wait period. For a payment that takes 2 seconds to process but waits 30 seconds for a webhook, this metric reports approximately 32 seconds. This is distinct from the standard Duration metric, which only measures active compute time.

Custom business metrics for the callback funnel

The built-in metrics tell you whether executions succeeded or failed. To understand where in the business flow the issue occurred, we emit custom metrics at each stage using Powertools for AWS Lambda Metrics with Embedded Metric Format (EMF):

from aws_lambda_powertools import Metrics
from aws_lambda_powertools.metrics import MetricUnit

metrics = Metrics(namespace="DurablePayments", service="payment-processor")

# In the durable handler, after each stage:
metrics.add_metric(name="PaymentIntentCreated", unit=MetricUnit.Count, value=1)
metrics.add_metric(name="PaymentSucceeded", unit=MetricUnit.Count, value=1)
metrics.add_metric(name="PaymentFailed", unit=MetricUnit.Count, value=1)
metrics.add_metric(name="PaymentTimeout", unit=MetricUnit.Count, value=1)

In the webhook handler:

metrics.add_metric(name="WebhookReceived", unit=MetricUnit.Count, value=1)
metrics.add_metric(name="WebhookSucceeded", unit=MetricUnit.Count, value=1)
metrics.add_metric(name="WebhookSignatureFailure", unit=MetricUnit.Count, value=1)

These metrics create an end-to-end funnel:

PaymentRequested → PaymentIntentCreated → WebhookReceived → WebhookSucceeded → PaymentSucceeded

Any drop-off between stages pinpoints the problem. If PaymentIntentCreated is higher than WebhookReceived, Stripe is not delivering webhooks. If WebhookReceived is higher than WebhookSucceeded, signature verification is failing. No corresponding PaymentSucceeded for a PaymentIntentCreated means the callback timed out.

Alarms for callback failure modes

Durable functions with callbacks have specific failure modes: callbacks that never arrive, webhook signatures that fail verification, and executions that time out waiting. We define alarms for each:

DurableExecutionFailureAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    Namespace: AWS/Lambda
    MetricName: DurableExecutionFailed
    Dimensions:
      - Name: FunctionName
        Value: !Ref PaymentProcessorFunction
    Threshold: 1
    ComparisonOperator: GreaterThanOrEqualToThreshold
    TreatMissingData: notBreaching
    ...

PaymentTimeoutAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    Namespace: DurablePayments
    MetricName: PaymentTimeout
    Dimensions:
      - Name: service
        Value: payment-processor
    Threshold: 1
    ...

WebhookSignatureFailureAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    Namespace: DurablePayments
    MetricName: WebhookSignatureFailure
    Dimensions:
      - Name: service
        Value: stripe-webhook
    Threshold: 3

These alarm definitions are abbreviated for readability. Each alarm in the deployed template.yaml also sets Dimensions (scoping DurableExecutionFailed to the payment-processor function, and the custom metrics to their service). It also includes Statistic, Period, EvaluationPeriods, and AlarmActions/OKActions wired to an SNS topic. See the GitHub repository for the deployable definitions.

Alarm	What it catches
DurableExecutionFailed	Code errors, Stripe API failures, unhandled exceptions in the durable function
DurableExecutionTimedOut	Whole-execution timeout: execution exceeds `DurableConfig.ExecutionTimeout`
PaymentTimeout	Callbacks that never arrive: webhook misconfiguration, Stripe outage, network issues
WebhookSignatureFailure	Wrong webhook secret, replay attacks, endpoint misconfiguration
WebhookError	Webhook function error spikes (unhandled exceptions in the handler)

Unified dashboard

We combine built-in durable metrics, custom EMF metrics, and standard Lambda metrics into a single CloudWatch dashboard. The dashboard includes widgets for execution state, payment outcomes, end-to-end flow metrics, quota utilization, cost drivers, error breakdown, and API/webhook latency.

Figure 2: CloudWatch dashboard showing durable execution state, payment outcomes, end-to-end flow metrics, running executions and quota utilization

Figure 3: CloudWatch Alarms showing DurableExecutionFailures, PaymentTimeouts, and WebhookSignatureFailures alarm states

Tracing callbacks across the suspension boundary

When a durable function suspends at a callback, the execution pauses. An external system (Stripe) fires a webhook to your API Gateway, which invokes the webhook handler. The webhook handler then calls send_durable_execution_callback_success to deliver the result back to the suspended execution, which resumes and completes. The challenge is correlating these two separate invocations so you can reconstruct the full payment timeline from a single query.

Structured logging with correlation keys

Using Lambda Powertools Logger, we progressively append correlation keys as they become available. Each subsequent log entry automatically includes all previously appended keys:

from aws_lambda_powertools import Logger
from aws_durable_execution_sdk_python import (
    DurableContext, durable_execution, durable_step,
)
from aws_durable_execution_sdk_python.config import CallbackConfig, Duration
from aws_durable_execution_sdk_python.exceptions import CallbackError

logger = Logger(service="payment-processor")

@durable_execution
def handler(event, context: DurableContext):
    payment = context.step(validate_payment_request(event), name="validate-payment")
    logger.append_keys(customer_id=payment["customer_id"])

    callback = context.create_callback(
        name="stripe-payment-result",
        config=CallbackConfig(timeout=Duration.from_minutes(5)),
    )
    logger.info("Callback created", callback_id=callback.callback_id)

    intent = context.step(
        create_stripe_payment_intent(payment, callback.callback_id),
        name="create-payment-intent",
    )
    logger.append_keys(payment_intent_id=intent["payment_intent_id"])
    logger.info("Suspending, waiting for Stripe webhook callback")

    try:
        result = callback.result()  # Function suspends here
    except CallbackError:
        logger.warning("Payment timed out")
        return {"status": "timeout", "message": "No confirmation within 5 minutes"}

In the webhook handler, we append the same keys so a single Logs Insights query reconstructs the full timeline:

logger = Logger(service="stripe-webhook")

def handler(event, context):
    # ... verify signature, parse event
    logger.append_keys(event_type=event_type, payment_intent_id=payment_intent_id)
    logger.append_keys(callback_id=callback_id)
    logger.info("Processing webhook event")

Query across all three log groups for a single payment:

fields @timestamp, service, message, customer_id, payment_intent_id, callback_id
| filter payment_intent_id = "pi_3TJafD04vzZc6RmP0RrCWhix"
| sort @timestamp asc

Figure 4: CloudWatch Logs Insights query showing the timeline of a single payment across payment-api, payment-processor, and stripe-webhook

Durable steps and X-Ray annotations

The SDK’s @durable_step decorator checkpoints each step. If the function crashes and replays, completed steps return their cached result without re-executing. We combine this with Powertools Tracer to add searchable X-Ray annotations at each business-critical point:

from aws_durable_execution_sdk_python import StepContext, durable_step

@durable_step
@tracer.capture_method
def create_stripe_payment_intent(step_context: StepContext, payment: dict, callback_id: str) -> dict:
    tracer.put_annotation("callback_id", callback_id)
    tracer.put_annotation("customer_id", payment["customer_id"])

    try:
        intent = stripe.PaymentIntent.create(
            amount=payment["amount"], currency=payment["currency"],
            payment_method=payment["payment_method_id"], confirm=True,
            metadata={"callback_id": callback_id},
            automatic_payment_methods={"enabled": True, "allow_redirects": "never"},
            ...
        )
    except stripe.error.CardError as exc:
        # Hard declines (e.g. pm_card_chargeDeclined) raise synchronously. Return a
        # structured decline so the step doesn't retry and fail the whole execution.
        ...
        metrics.add_metric(name="PaymentDeclinedAtCreate", unit=MetricUnit.Count, value=1)
        return {"declined": True, ...}  # decline_code, error_message, payment_intent_id

    metrics.add_metric(name="PaymentIntentCreated", unit=MetricUnit.Count, value=1)
    ...
    return {"payment_intent_id": intent.id, "status": intent.status}

Note: The preceding code is abbreviated for readability. Refer to the GitHub repository for the complete code. The main durable handler runs within a FacadeSegment X-Ray context that does not support put_annotation(). Annotations work normally inside @durable_step functions. In the main handler, use a try/except wrapper if you need annotations outside of steps.

Note: When calling PaymentIntent.create with confirm=True, some cards decline synchronously (no webhook fires). The deployed code handles this by detecting the decline in the step return value and skipping the callback suspension, preventing an indefinite wait.

The X-Ray Service Map shows the complete request flow: API Gateway to payment-api to payment-processor, and the separate webhook path from API Gateway to stripe-webhook.

Figure 5: X-Ray Service Map showing API Gateway connected to payment-api and stripe-webhook, with payment-api connected to payment-processor

Durable executions tab

The Lambda console provides a built-in Durable executions tab showing each execution’s step-by-step timeline, including the callback wait state. You can see which steps completed, where the function suspended, and when (or if) the callback arrived.

Figure 6: Lambda console Durable executions tab showing a completed execution with steps: validate-payment succeeded, create-payment-intent succeeded, stripe-payment-result callback received, and final result succeeded

Putting it together: debugging real failure modes

The following three scenarios demonstrate how all of these observability layers work together. You can reproduce each one from the demo checkout page.

Scenario 1: Webhook never arrives

A customer reports that their payment was charged but they never received a confirmation.

1. Alarm fires. The PaymentTimeoutAlarm triggers, indicating a durable execution timed out waiting for a callback.

2. Check the dashboard. The Payment Outcomes widget shows a spike in PaymentTimeout. The End-to-End Flow Metrics widget reveals the drop-off: PaymentIntentCreated count is higher than WebhookReceived, meaning the webhook never arrived.

3. Query logs. Search Amazon CloudWatch Logs Insights for the timed-out payment:

fields @timestamp, service, message, payment_intent_id, callback_id
| filter message = "Payment timed out"
| sort @timestamp desc
| limit 5

This returns the payment_intent_id of the timed-out payment.

4. Cross-reference the webhook handler. Search for that payment_intent_id in the webhook handler logs. No results means Stripe never delivered the webhook. Results with WebhookSignatureFailure mean the webhook secret is misconfigured.

5. Inspect the X-Ray trace. Filter traces by the payment_intent_id annotation. The trace shows the durable function start but no corresponding webhook handler span, confirming the webhook never arrived.

6. Check the durable executions tab. The execution shows validate-payment and create-payment-intent as succeeded, with the stripe-payment-result callback in a timed-out state.

Figure 7: Durable executions tab showing the timed-out execution: validate-payment succeeded, create-payment-intent succeeded, stripe-payment-result callback timed out

Within minutes, you have identified the root cause (the Stripe webhook endpoint was misconfigured) without adding a single debug statement or redeploying code.

Scenario 2: The whole workflow runs too long

The callback timeout in Scenario 1 is a per-callback bound (5 minutes in this example). There is also an outer bound: DurableConfig.ExecutionTimeout (600 seconds), which caps the total wall-clock time of the whole execution. If you set a callback to wait an hour but the overall ExecutionTimeout is 10 minutes, the execution itself terminates first. This shows up as a distinct terminal state in the durable executions tab, on the Durable Execution State widget, and as its own alarm (DurableExecutionTimedOutAlarm).

Choose the “Simulate timeout (no webhook)” option on the demo checkout page to reproduce this. The durable function skips the Stripe call, suspends on a long-timeout callback, and lets ExecutionTimeout catch it. The dashboard distinguishes the two failure modes cleanly: per-callback timeouts show up on the custom Payment Outcomes widget as PaymentTimeout. Whole-execution timeouts appear on the built-in Durable Execution State widget alongside started/succeeded/failed counts. This distinction matters operationally because the remediation is different: callback timeouts point to external system issues (Stripe), while execution timeouts point to configuration issues (your timeout values).

Scenario 3: Customer abandons checkout

Real checkout flows have a third outcome: the customer cancels while the durable function is still suspended. The demo wires this up to StopDurableExecution, which terminates the in-flight execution and surfaces on the same Durable Execution State widget as a separate terminal state.

Choose “Simulate timeout” and then “Cancel Payment” on the demo page to see this happen. Looking at the dashboard after running all three scenarios, the execution-state widget tells the full story: started, succeeded, failed, timed-out, and stopped. Each state answers a different operational question about what is happening to your workflows.

Conclusion

In this post, we walked through observability best practices for Lambda durable functions using a Stripe payment processing pipeline. Callbacks can time out, whole executions can expire, and running workflows can be canceled. Each shows up as a distinct terminal state, and each deserves its own alarm. Layering custom business metrics, structured logging with correlation keys, X-Ray annotations, and the durable executions tab on top of the built-in CloudWatch metrics gives you a clear picture of where in the lifecycle any given execution is. It also reveals where in the business funnel any failure occurred.

Deploy the payment processing application from the GitHub repository and try the three demo scenarios to see the dashboards, alarms, and execution history in your own account. For core concepts, see Lambda durable functions. For the durable execution SDK, see the Python SDK, JavaScript SDK, and Java SDK. Browse Serverless Land for reference architectures.

Collecting CPU and memory metrics for AWS Lambda MicroVMs

Eric Heinz — Fri, 14 Aug 2026 11:11:55 +0000

Most production services in AWS use at least two key metrics for service health – CPU and memory utilization. The amount of CPU and memory used by the host (in this case, a MicroVM) can indicate scaling signals or inefficiencies in your application. If you’re running a production workload on AWS Lambda MicroVMs, it’s recommended to have observability in these dimensions. And the easiest way to collect these metrics is through the Amazon CloudWatch Agent.

This blog shows you how to collect CPU and memory metrics from within the MicroVM using the CloudWatch Agent.

How to collect CPU and memory metrics in your MicroVM

To observe how a workload uses CPU and memory over time, run the CloudWatch Agent inside the MicroVM. Since a MicroVM image is a full OS snapshot, you can start the agent during image creation, meaning it will already be running the moment a MicroVM launches from that image. This means zero startup latency and one-time configuration: set up the CloudWatch Agent once in the image, and every MicroVM that launches from it already has a running monitoring stack.

To setup CloudWatch Agent, you will modify the ZIP containing your application and Dockerfile, and build a MicroVM image. Once you run a MicroVM from the image, three metrics will be emitted (cpu_usage_active, cpu_usage_idle, mem_used_percent) under an ImageName dimension populated from a Lambda-injected environment variable.

Lambda-injected environment variables

The Lambda MicroVMs runtime automatically exposes these environment variables to your application:

Env var	Example
`AWS_LAMBDA_MICROVM_IMAGE_NAME`	`mem-python`
`AWS_LAMBDA_MICROVM_IMAGE_ARN`	`arn:aws:lambda:us-west-2:…:microvm-image:mem-python`
`AWS_LAMBDA_MICROVM_IMAGE_VERSION`	`1.0`
`AWS_REGION`	`us-west-2`

The example below uses AWS_LAMBDA_MICROVM_IMAGE_NAME as a metric dimension so you can monitor metrics per MicroVM image.

Setting up custom metric dimensions from env variables

Amazon CloudWatch Agent uses telegraf to process metrics and opentelemetry-collector (OTel) to export them. Normally, you configure the agent through a cwagent.json file, which the agent’s config-translator converts into a telegraf TOML file and an OTel YAML file for the process to use at startup.

In this post, we skip the JSON configuration and create the telegraf and OTel files directly. This lets us dynamically set a custom metric dimension from an environment variable using OTel’s ${env:VAR} syntax. The telegraf config defines which metrics to collect, while the OTel config resolves the environment variable at process start and appends it as a dimension.

Configuring CloudWatch Agent

In this section, we cover how to configure CloudWatch Agent to report CPU and memory metrics for MicroVMs launched from your MicroVM image.

Step 1: Configure the telegraf plugin to emit CPU and Memory metrics

Create a cwagent.toml file to define the configuration for telegraf to emit CPU and memory metrics every minute:

[agent]
  interval = "60s"
  flush_interval = "60s"

  # Host name is omitted since it doesn't exist in a MicroVM
  omit_hostname = true

[[inputs.cpu]]
  totalcpu = true

  # Disable per-CPU reporting for an aggregate view over all vCPUs in your MicroVM.
  # Set to 'true' to see utilization for each individual vCPU.
  percpu = false
  report_active = true
  fieldpass = ["usage_active", "usage_idle"]

[[inputs.mem]]
  fieldpass = ["used_percent"]

In this configuration, the chosen metric (used_percent) reports memory usage as a percentage of total memory inside the MicroVM. Telegraf derives this from MemAvailable in /proc/meminfo, which reflects memory that is committed and not reclaimable. When your application releases memory back to the OS (e.g. via free()), that memory becomes reclaimable again, and used_percent decreases accordingly.

To monitor additional memory metrics, you can add the following fields to the fieldpass list:

cached: for page cache bytes
buffered: for buffered I/O bytes
total: for total memory available to the MicroVM

Step 2: Configure OTel to process and export the metrics to CloudWatch

Create a cwagent.yaml file to export metrics to CloudWatch under the namespace LambdaMicroVms/Application with dimension ImageName. The dimension value is populated from the environment variable AWS_LAMBDA_MICROVM_IMAGE_NAME.

receivers:
  telegraf_cpu: { collection_interval: 60s }
  telegraf_mem: { collection_interval: 60s }

processors:
  resource:
    attributes:
      - { key: ImageName, value: "${env:AWS_LAMBDA_MICROVM_IMAGE_NAME}", action: insert }
  transform/strip_cpu_dim:
    error_mode: ignore
    metric_statements:
      - context: datapoint
        statements:
          - delete_key(attributes, "cpu")

exporters:
  awscloudwatch:
    namespace: LambdaMicroVms/Application
    region: ${env:AWS_REGION}
    resource_to_telemetry_conversion: { enabled: true }

service:
  pipelines:
    metrics:
      receivers:  [telegraf_cpu, telegraf_mem]
      processors: [resource, transform/strip_cpu_dim]
      exporters:  [awscloudwatch]

If you want more dimensions such as image version, add it to attributes.

Note: since only aggregate CPU usage is emitted by telegraf, we don’t need OTel to include a CPU dimension, so delete_key(attributes, "cpu") is used to remove this dimension.

Step 3: Install CloudWatch Agent in your Dockerfile

In your Dockerfile, install the CloudWatch Agent from the Amazon Linux repository. Then copy over the telegraf and OTel files to where the agent expects to retrieve them. Then configure your application’s entrypoint:

FROM public.ecr.aws/lambda/microvms:al2023-minimal

RUN dnf install -y --setopt=install_weak_deps=0 \
        python3 amazon-cloudwatch-agent \
    && dnf clean all

COPY app.py        /app/app.py
COPY cwagent.toml  /etc/cwagent.toml
COPY cwagent.yaml  /etc/cwagent.yaml
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

CMD ["/entrypoint.sh"]

Step 4: Configure your Entrypoint to start CloudWatch Agent

Create a file called entrypoint.sh to start the CloudWatch Agent as a background process while executing your application in the foreground:

#!/usr/bin/env bash
set -euo pipefail

# Telegraf inputs (TOML) + OTel pipeline (YAML).
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent \
    -config     /etc/cwagent.toml \
    -otelconfig /etc/cwagent.yaml &

exec python3 /app/app.py

This is everything you need to get CloudWatch running inside your MicroVMs!

Execution role requirements

To write the metrics to CloudWatch, ensure the MicroVM’s execution role has cloudwatch:PutMetricData permissions.

Verifying it works

To verify the metrics are being emitted, run the following command a few minutes after launching a MicroVM from your image:

aws cloudwatch list-metrics \
    --namespace LambdaMicroVms/Application \
    --dimensions Name=ImageName,Value=mem-python \
    --region us-west-2

You should see exactly three metric series per image: cpu_usage_active, cpu_usage_idle, and mem_used_percent.

Viewing the metrics

To view the metrics in the CloudWatch console, click “All Metrics”, and select the custom namespace LambdaMicroVms/Application (set in cwagent.yaml namespace field).

Here is an example for how it looks inside the console:

In the graph above, the application consumes ~2% memory (left axis) and < 0.1% CPU usage (right axis) when idle. The application then consumes ~9% of memory at the 30 minute mark, holds it for around 5 minutes, then releases it back to the OS. As it releases memory, we see memory utilization decrease. In this example, the MicroVM size is larger than the application needs – less than 10% of memory was used, indicating a smaller MicroVM size may be more economic for this workload.

If your CPU and/or memory utilization is below the baseline size configured (see MicroVM sizing), consider choosing a lower baseline to reduce your compute bill.

Conclusion

This post shows you how to configure and run the CloudWatch Agent inside your MicroVM image so you can collect CPU and memory metrics for MicroVMs launched from the image. This helps you monitor resource usage of your application as it is used, so you can right-size the MicroVM for your workload, debug service health, and check for scaling signals.

To get started, visit the AWS Lambda console, or install the AWS Lambda MicroVMs agent skill.

Designing for failure: Building resilient systems on AWS

Dhvani Vora — Thu, 13 Aug 2026 20:03:28 +0000

In cloud computing, failure in distributed systems isn’t a matter of if, but when. Modern applications span servers, Availability Zones, and Regions. Each component represents a potential point of failure. Resilient applications engineer fault tolerance into their architecture, building systems that self-recover and maintain availability. This post is written for engineers and architects who run distributed data systems such as Apache Cassandra, Apache Kafka, or HDFS on Amazon Elastic Compute Cloud (Amazon EC2) and want to build resilience against hardware failure.

We were working with a customer during one such incident and wanted to share the example. The customer runs a web application that uses Cassandra as its data store, handling both read-heavy and write-heavy workloads at a scale of millions of queries per day.

The 2 AM wake-up call nobody wants

Consider a platform that monitors millions of enterprise network devices across hospitals, universities, and airports worldwide. It detects problems before IT teams even notice them. For that platform, a 2 AM page is more than inconvenient. When your value proposition is catching failures before anyone else does, being caught off-guard by your own infrastructure failure is existential.

The engineering team was deep in quarterly planning when their monitoring dashboard lit up. Three Cassandra nodes had gone dark simultaneously. This was not a graceful shutdown or a rolling restart. It was a hard failure with no warning.

Their architecture is typical of high-scale telemetry platforms. Kafka-powered microservices ingest device telemetry, Apache Flink handles real-time anomaly detection, and Apache Airflow orchestrates batch analytics and firmware updates. All of these rely on Apache Cassandra as the distributed database backbone. The database stores billions of daily writes and handles millions of queries per day.

What actually happened

Three i4i.4xlarge instances running Cassandra nodes failed simultaneously in the SFO region. Investigation revealed that all three instances were colocated on the same physical host. That host suffered a hardware failure, taking all three instances offline at once.

Engineers spent ninety minutes digging through system logs trying to determine the root cause. The root cause was architectural. The deployment lacked Partition Placement Groups, creating a single point of failure where logical replication was undermined by physical collocation.

The good news: Cassandra maintained service availability with no data loss thanks to its replication factor. The bad news: for over an hour, the system ran on a thin safety margin. One more node failure in the same replication group would have caused data unavailability for a subset of queries. That is real customer impact for a platform that promises always-on monitoring.

This is the insidious nature of correlated failures. Individual node failures are expected and designed for. That is the whole point of replication. But when your replicas share physical infrastructure, replication becomes a paper guarantee. You have three copies of the data, but they all live on the same machine.

Making matters worse, their monitoring tools completely missed the initial failure. System status checks correctly flagged the host-level problem. But without Amazon CloudWatch alarms configured to act on those checks, detection was entirely reactive. The team found out because other things started behaving oddly, not because an alarm told them three nodes were down.

Hardware fails. You can’t fix it with a patch or configuration change. The real questions are how fast you detect it, how well your system handles it, and whether failures are correlated.

How the team responded and what they changed

The operations team manually replaced two failed instances with new ones on healthy hardware and restarted the third for log collection. Once replacement instances came online, new Cassandra nodes automatically rejoined their clusters and streamed data from surviving replicas. This process took several hours depending on data volume. Only after full synchronization did the clusters return to full redundancy.

The team recognized that this ninety-minute manual scramble wouldn’t scale. Similar problems had happened before, and each time they followed the same reactive pattern: page, investigate, manually replace, wait for streaming, breathe. Here’s what they implemented to break that cycle, and what you should implement too.

Figure 1: The same failure handled two ways. Manual response took over 90 minutes plus hours of streaming. The automated path completes recovery in under 5 minutes.

1. Use Partition Placement Groups to isolate failure domains

The three crashed servers shared a physical machine because no one told AWS otherwise. Without placement group constraints, instances are placed based on available capacity. That can mean multiple instances land on the same host. For stateless web servers, this rarely matters. For distributed databases whose entire resilience model depends on replicas being independent, it’s a silent architecture bug waiting to become a 2 AM incident.

Partition Placement Groups fix this by distributing instances across separate hardware racks. Each partition maps to a distinct set of physical infrastructure, with separate power and separate network switches. When one rack fails, it affects only the instances in that partition.

Figure 2: Distributing Cassandra replicas across Partition Placement Group partitions so a single rack failure affects only one node.

You can create up to seven partitions per Availability Zone, with as many instances as needed in each. By mapping Cassandra replicas to separate partitions, a single hardware failure takes down one node instead of three. This applies to any distributed system that maintains replicas, such as Kafka, HDFS, or Cassandra.

Key insight: Align your Partition Placement Group partitions with your application’s replication topology. If Cassandra uses a replication factor of 3, place each replica in a different partition. This means the physical isolation boundary matches the logical replication boundary.

CLI example:

aws ec2 create-placement-group \
  --group-name cassandra-partitioned \
  --strategy partition \
  --partition-count 3

aws ec2 run-instances \
  --placement "GroupName=cassandra-partitioned,PartitionNumber=1" \
  --instance-type i4i.4xlarge \
  --image-id ami-xxxxxxxx

Partition Placement Groups (up to 7 partitions per AZ, unlimited instances per partition) are designed for large distributed workloads. Spread Placement Groups (max 7 instances per AZ, each on a separate rack) suit small critical clusters. For a Cassandra deployment at scale, Partition is the right choice. Learn more in the Amazon EC2 placement groups documentation.

2. Monitor system status checks and use composite alarms

The Cassandra team’s monitoring blind spot came down to a distinction many teams overlook. AWS runs two health checks on every instance: instance status checks (your guest OS and software) and system status checks (the physical hardware underneath). When a system status check fails, the problem is below your control. This includes a host crash, a power failure, or network loss at the rack level. No amount of SSH-ing will help, because the box is unreachable.

The Cassandra team had no Amazon CloudWatch alarms configured on either check type. That meant the only signal was cascading application errors noticed by engineers who happened to be awake. Set these up on day one, before your first production deployment.

To avoid false alarms during normal reboots, where metrics may briefly go missing, combine system status checks with application-level health monitoring using composite alarms. When both fail together, you know there’s a real problem. See the CloudWatch composite alarms documentation for setup details.

3. Automate instance recovery and replacement

The Cassandra team’s ninety-minute recovery wasn’t slow because the engineers were incompetent. It was slow because humans were in the loop. Waking up, assessing, deciding, acting, and verifying: each step adds minutes that compound under pressure. Auto Scaling groups remove the human from the critical path.

Place your Cassandra nodes in an Auto Scaling group. Auto Scaling continuously runs health checks on every instance, and when it marks an instance unhealthy, it terminates it and launches a replacement on different physical hardware, automatically placed within your Partition Placement Group. Under normal conditions, an instance whose system status checks fail is replaced within a few minutes.

The gap to close is detection, not replacement. Rather than waiting for Auto Scaling to reach its own conclusion, have the composite alarm from the previous section explicitly tell Auto Scaling the instance is unhealthy by calling the SetInstanceHealth API. As soon as your combined signal (system status check plus application-level check) confirms a real failure, mark the instance unhealthy and let Auto Scaling replace it immediately. This sidesteps any ambiguity in detection and starts recovery in seconds rather than minutes.

For stateless services, this is enough. For stateful systems like Cassandra, you need an additional step. Lifecycle hooks pause new instances before they join the cluster. A raw Amazon EC2 instance isn’t a functioning Cassandra node. It needs to join the ring, stream data from peers, and verify consistency before serving traffic. Read more in the Amazon EC2 Auto Scaling lifecycle hooks documentation.

In this customer’s case, automating these steps cut recovery time from ninety minutes of manual intervention to under five minutes of automated recovery.

A note on stateful recovery: automated replacement only handles the infrastructure layer. For Cassandra specifically, the new node still needs to stream data from peers before it’s fully operational. The key improvement isn’t eliminating that streaming time. It’s eliminating the human response time before streaming even begins.

4. Build automated incident response with AWS Systems Manager

When servers fail, you face competing priorities. You need to replace them fast to restore capacity, and you need to preserve logs for root cause analysis. These goals conflict when done manually. The Cassandra team restarted one failed node solely to collect diagnostic data before replacing it, adding time to an already long recovery.

AWS Systems Manager runbooks automate this tradeoff away. Build a workflow that runs these steps in sequence:

Isolate the failed instance by detaching it from the load balancer target group.
Create an Amazon EBS snapshot and capture available logs to Amazon S3.
Terminate the instance so that Auto Scaling can replace it.
Notify the on-call channel with the instance ID, failure type, and Amazon S3 log location.

A subtle but important detail: when the instance’s lifecycle is managed by an Auto Scaling group, let the group replace it. Terminating the instance directly only delays recovery, because the group first has to notice the instance is gone before it launches a replacement. Instead, call the TerminateInstanceInAutoScalingGroup API. This tells EC2 Auto Scaling to terminate the unhealthy instance and immediately launch a replacement in one coordinated action. Trigger this runbook automatically with Amazon EventBridge rules that match Amazon EC2 state-change events. The result is that forensic data is preserved, replacement happens in parallel, and the on-call engineer gets a notification after the system has already healed, rather than a page asking them to start fixing it.

Figure 3: The automated recovery workflow, from hardware failure detection through node rejoin, orchestrated by Amazon EventBridge, Auto Scaling, and AWS Systems Manager.

5. Invest in observability before you need it

After resolving the Cassandra incident, the team asked a harder question: what else is silently failing? They ran a broader health assessment, and the answer was sobering. Unstable Redis connections were dropping under load. Amazon EBS volumes were running with elevated latency. Application Load Balancer health check intervals were misconfigured. Secondary databases were approaching connection pool exhaustion. Any of these could cause the next outage, and none of them had triggered a single alert.

This is the pattern. Teams invest in monitoring for the system that recently broke while the next failure quietly builds elsewhere. The better approach is treating observability as infrastructure. Deploy it everywhere from day one, not bolted on after the post-mortem.

Deploy the CloudWatch agent for system-level and application-level metrics. Use Amazon CloudWatch Synthetics canaries to continuously test critical user paths such as login, data ingestion, and dashboard rendering. Set up distributed tracing with AWS X-Ray to identify latency bottlenecks across your microservice mesh. The goal isn’t only knowing that services are running. It’s continuously confirming they’re working correctly from the customer’s perspective.

The Cassandra team built what they call their “resilience dashboard.” It’s a single view surfacing Partition Placement Group distribution, replica lag, system status check state, and Auto Scaling group health. When the next incident happens, they won’t be scrambling to figure out what’s broken. They’ll open one dashboard and know immediately whether their defenses are holding.

Placement groups: Quick reference

The team’s outage involved Partition Placement Groups, but Amazon EC2 offers three placement group types. Choosing the wrong one is a common mistake, so here’s how they compare:

Type	Max instances	Isolation level	Best for
Partition	Unlimited (up to 7 partitions per AZ)	Separate racks per partition	Large distributed databases (Cassandra, Kafka, HDFS)
Spread	7 per AZ	Each instance on a separate rack	Small critical clusters needing maximum isolation
Cluster	Unlimited	Same rack (co-located)	HPC, ML training, low-latency workloads

If the Cassandra team had used Spread Placement Groups instead, they would have hit the 7-instance-per-AZ ceiling almost immediately at their scale. Partition Placement Groups gave them isolation and room to grow. For the highest-criticality deployments, combine placement groups with multiple Availability Zones. You get separate racks and separate data centers, protecting against both rack-level failures and zone-wide events like power grid outages.

The bigger picture: Resilience is a practice

Building resilient systems isn’t a one-time project. It’s a practice that evolves with your architecture. Start by assessing your workloads with the AWS Well-Architected Tool to identify single points of failure you might not see day-to-day. Define Service Level Objectives, so your team agrees on what “good enough” looks like. Not every service needs 99.99% availability, but you need to know which ones do.

Then layer your defenses. Placement groups prevent correlated hardware failures, composite alarms detect problems within minutes, and automated recovery fixes common issues without waking anyone up.

Test regularly. Run disaster recovery drills quarterly. Don’t rely only on tabletop exercises. Run actual failovers in pre-production environments. Use AWS Fault Injection Service to simulate hardware failures and zone outages in a controlled way. Hold blameless post-mortems after every incident to understand what broke, why it wasn’t caught earlier, and what you’ll change.

After this incident, the team deployed Partition Placement Groups, configured composite alarms, and automated their response process. The next time hardware fails, and it will, it won’t cause the same damage.

Consider adopting Chaos Engineering as a discipline. The principles of Chaos Engineering encourage teams to proactively inject failures into production-like environments to uncover weaknesses before they cause real outages. AWS Fault Injection Service makes it straightforward to run these experiments safely, with guardrails that automatically stop experiments if impact exceeds defined thresholds.

For related guidance, see the AWS Well-Architected Framework Reliability Pillar and the Amazon EC2 Auto Scaling User Guide. A sample Systems Manager runbook and AWS CloudFormation template for the automated recovery workflow described in this post is available in the AWS Samples GitHub repository.

If you’ve implemented similar resilience patterns or have questions about placement groups and automated recovery, share your experience in the comments.

Key takeaways

Challenge	Solution
Multiple instances on same physical host	Partition Placement Groups
No health notification for sudden failures	Amazon CloudWatch alarms on system status checks
Missing metrics during host reboots	Composite alarms with application-level health checks
Manual, slow incident response	Automated recovery with Auto Scaling and lifecycle hooks
Delayed root cause identification	Systematic triage starting at the infrastructure layer
Reduced redundancy after failure	Auto Scaling groups for automatic replacement
Recurring confidence erosion	Proactive architectural reviews and observability investment

Amazon EC2 provides tools like placement groups, managed services with built-in high availability, and automation frameworks like AWS Systems Manager. Select the right ones for your workload and test them relentlessly. Failure is inevitable. Your readiness determines the outcome.

Burst to Region: Overflow AWS Outposts workloads to Amazon EC2

Diya . — Wed, 12 Aug 2026 17:06:54 +0000

AWS Outposts brings AWS infrastructure into your data center, giving on-premises workloads the low latency and data locality they need. But unlike the AWS Region, an Outposts rack has a fixed amount of compute. When your workload needs more instances than the rack can provide, you have two options: drop requests, or overflow them somewhere with room to grow. This post shows you how to automate the second option. You build a Burst to Region pattern that detects capacity constraints on your Outpost, launches Amazon Elastic Compute Cloud (Amazon EC2) instances in the parent Region, gradually shifts traffic to them, and returns traffic to local instances once capacity recovers.

To implement this pattern you configure Amazon CloudWatch, Amazon Simple Notification Service (Amazon SNS), AWS Lambda, Amazon EC2 Auto Scaling, Elastic Load Balancing (Application Load Balancer), and Amazon EventBridge. You trade a moderate latency increase for continued availability during capacity events.

When to use this pattern

This pattern assumes your Outposts workload scales out through Amazon EC2 Auto Scaling. Burst to Region reacts to instance-capacity exhaustion on the rack. It engages when your workload tries to launch more instances than the available Outpost capacity supports. If your fleet is fixed size and degrades under load without scaling out, the capacity alarm never fires and overflow never triggers. For those workloads, monitor per-instance saturation (CPU, latency) separately.

Good candidates prefer local capacity but can tolerate Region latency under pressure. If your application runs on Outposts for proximity yet degrades gracefully when some traffic takes the longer path to the Region, it fits this pattern. Examples include:

Internal enterprise applications.
Stateless web frontends and API layers.
Pre-processing tiers where single-digit to tens-of-milliseconds additional round-trip latency during peaks is acceptable.

Poor candidates cannot absorb any added latency or must stay on the Outpost. Avoid this pattern for:

Applications with sub-millisecond requirements.
Workloads with strict data residency or sovereignty mandates that prevent traffic from leaving the on-premises environment.
Real-time control systems with hard timing constraints.
Applications tightly coupled to on-premises data stores with no Region replica.

The core tradeoff is explicit. During capacity events, you accept moderately higher latency to maintain availability. If your workload cannot tolerate any latency increase, keep it pinned to Outposts and reserve capacity through other means, such as Capacity Reservations.

Solution overview

Burst to Region works in three moves: detect capacity pressure on the Outpost, launch overflow compute in the parent Region, and shift traffic gradually until local capacity recovers. Six AWS services coordinate to make this automatic. The following diagram shows the reference architecture for the Burst to Region pattern, illustrating how the six AWS services interact during capacity detection, overflow scaling, traffic distribution, and recovery.

Figure 1: Reference architecture for Burst to Region on AWS Outposts

The pattern uses six AWS services working together:

Amazon CloudWatch monitors Outposts capacity utilization and raises alarms.
Amazon SNS provides event fan-out from alarm to orchestrator.
AWS Lambda orchestrates the burst logic (scale-out, weight adjustment, recovery)
Amazon EC2 Auto Scaling manages the overflow fleet lifecycle.
Application Load Balancer distributes traffic across both locations using weighted target groups.
Amazon EventBridge handles periodic recovery evaluation.

You must configure five phases for this pattern:

Monitor. CloudWatch tracks Outposts capacity utilization metrics in the AWS/Outposts namespace.
Detect. A CloudWatch alarm fires when utilization exceeds a threshold (for example, 80%).
Overflow. The alarm triggers a Lambda function through Amazon SNS. Lambda scales out a Region-based Amazon EC2 Auto Scaling group and adjusts ALB target group weights.
Distribute. The ALB splits traffic between Outposts instances and Region instances using weighted forwarding.
Recover. An Amazon EventBridge scheduled rule periodically evaluates capacity. When Outposts recovers, Lambda scales down the overflow fleet and returns all traffic to local instances.

Design decisions

We chose Application Load Balancer with weighted forwarding over Amazon Route 53 weighted routing for traffic distribution. ALB provides health-aware routing to only healthy overflow instances and target group stickiness for session consistency. Weight changes take effect for new connections after calling the ModifyRule API. DNS-based shifting through Route 53 provides too coarse control for rapid weight adjustments, and TTL propagation delays make recovery slower.

The burst orchestrator runs as a Lambda function rather than a long-running service. It executes only during state transitions, so there is no steady-state compute cost. Lambda integrates natively with Amazon SNS and Amazon EventBridge for event-driven invocation without additional infrastructure.

You implement recovery with an Amazon EventBridge scheduled rule (every 5 minutes) rather than relying solely on the CloudWatch alarm to return to OK state. The alarm confirms capacity is available, but does not confirm that overflow instances have drained active connections. The scheduled rule provides gradual, safe scale-down.

Implementation

This section walks through the key components of the Burst to Region pattern. For the complete deployable AWS SAM template, see the GitHub repository.

Prerequisites

To deploy this pattern, you need:

An AWS account with a configured AWS Outposts rack.
An Amazon Virtual Private Cloud (Amazon VPC) with subnets associated with your Outposts and subnets in the parent AWS Region.
IAM permissions to create CloudWatch alarms, Lambda functions, Auto Scaling groups, and ALB resources.
AWS Serverless Application Model (AWS SAM) CLI installed and configured.
Existing Amazon EC2 Auto Scaling group running on your Outpost (these become your baseline fleet)
A custom domain name with a DNS record (Route 53 alias or CNAME) pointing to your Application Load Balancer, and an AWS Certificate Manager (ACM) certificate for that domain to enable HTTPS.

Capacity monitoring and alarm

The CloudWatch alarm monitors instance utilization on the Outpost and triggers the burst workflow when capacity is constrained.

The InstanceTypeCapacityUtilization metric reports the percentage of a given instance type’s capacity in use. Note that this metric includes capacity consumed by managed services such as Amazon Relational Database Service (Amazon RDS) or Application Load Balancer running on the Outpost — not only your application’s EC2 instances. Factor this into your threshold planning.

OutpostsCapacityAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    AlarmName: outposts-capacity-high
    Namespace: AWS/Outposts
    MetricName: InstanceTypeCapacityUtilization
    Dimensions:
      - Name: OutpostId
        Value: !Ref OutpostId
      - Name: InstanceType
        Value: !Ref OutpostInstanceType
    Statistic: Average
    Period: 300
    EvaluationPeriods: 2
    Threshold: !Ref CapacityThreshold
    ComparisonOperator: GreaterThanOrEqualToThreshold
    AlarmActions:
      - !Ref BurstSNSTopic
    TreatMissingData: notBreaching

Why these values matter:

Period: 300 and EvaluationPeriods: 2 require 10 minutes of sustained high utilization before triggering. This avoids false alarms from transient spikes.
Threshold: 80 (recommended starting point) leaves a 20% buffer. A threshold set too high (95%) risks launch failures before the overflow fleet is ready. A threshold set too low (50%) causes unnecessary bursts.
TreatMissingData: notBreaching prevents false alarms when data points are missing. Since this alarm is scoped to a single instance type, treating missing data as breaching could trigger unnecessary bursts when the instance type is simply not in use.
Separate scale-out from scale-in: This alarm triggers burst scale-out at 80%. Recovery is handled separately by the Amazon EventBridge scheduled rule, which uses a lower threshold (for example, 60%) before scaling in. This hysteresis gap prevents flapping where scaling down immediately pushes utilization back above the alarm threshold.

Burst orchestrator (Lambda)

The Lambda function handles two event paths: alarm-triggered scale-out and scheduled recovery evaluation. The following pseudocode shows the orchestration flow:

def handler(event, context):
    # Route based on event source
    if is_scheduled_recovery(event):
        return handle_recovery_check()

    alarm_state = parse_sns_alarm_state(event)

    if alarm_state == 'ALARM':
        # Scale out the overflow Auto Scaling group
        scale_out_overflow(desired=OVERFLOW_CAPACITY)
        # Don't shift traffic yet --- wait for healthy instances
        publish_burst_metric(active=True)


def handle_recovery_check():
    """Called every 5 minutes by EventBridge."""
    # Check if burst is active
    if not is_burst_active():
        return

    # If overflow instances are healthy and registered, shift traffic
    if overflow_targets_healthy():
        current_weights = get_current_alb_weights()
        if current_weights['region'] == 0:
            # First shift --- instances are now warm
            set_alb_weights(outposts=90, region=10)
        elif needs_more_overflow():
            step_up_region_weight()

    # If Outposts capacity has recovered, begin scale-down
    if outposts_capacity_recovered():
        step_down_region_weight()
        if get_current_alb_weights()['region'] == 0:
            # All traffic back to Outposts, drain and terminate overflow
            wait_for_connection_draining()
            scale_down_overflow(desired=0)
            publish_burst_metric(active=False)

The key actions the function performs:

scale_out_overflow — Sets the overflow Auto Scaling group desired capacity from 0 to your configured burst size.
set_alb_weights — Calls the ModifyListener API to adjust weighted forwarding between the Outposts and Region target groups.
publish_burst_metric — Writes a custom CloudWatch metric (BurstActive) for dashboard visibility.
handle_recovery_check — Called every 5 minutes by Amazon EventBridge. Confirms Outposts capacity has recovered, steps weights back gradually, waits for connection draining, then scales down the overflow fleet.

Important: The orchestrator does not shift ALB weights immediately upon scale-out. It waits for the next Amazon EventBridge invocation (up to 5 minutes) to confirm that overflow instances have passed health checks and are registered as healthy in the target group. This helps prevent routing traffic to instances that have not finished launching.

For the production-ready implementation with error handling, gradual weight stepping, and connection draining verification, see the GitHub repository.

Overflow Auto Scaling group

The overflow fleet starts at zero and scales only when the Lambda function sets desired capacity during a burst event:

OverflowASG:
  Type: AWS::AutoScaling::AutoScalingGroup
  Properties:
    AutoScalingGroupName: burst-overflow-fleet
    LaunchTemplate:
      LaunchTemplateId: !Ref OverflowLaunchTemplate
      Version: !GetAtt OverflowLaunchTemplate.LatestVersionNumber
    MinSize: 0
    MaxSize: !Ref MaxOverflowCapacity
    DesiredCapacity: 0
    VPCZoneIdentifier:
      - !Ref RegionSubnet1
      - !Ref RegionSubnet2
    TargetGroupARNs:
      - !Ref RegionTargetGroup
    HealthCheckType: ELB
    HealthCheckGracePeriod: 120
    MetricsCollection:
      - Granularity: 1Minute

The overflow fleet starts at zero capacity and incurs no cost at rest. During a burst event, the Lambda function calls the SetDesiredCapacity API to launch overflow instances. During recovery, it sets desired capacity back to zero.

The launch template mirrors your Outposts instance type to maintain consistent performance characteristics across both locations.

ALB weighted forwarding

The ALB listener uses weighted forwarding across two target groups. In steady state, all traffic goes to Outposts (weight 100/0). During burst, the Lambda function adjusts these weights dynamically using the ModifyListener API. Clients reach the ALB through a DNS record — either a Route 53 alias or a CNAME pointing to the ALB’s DNS name.

ALBListener:
  Type: AWS::ElasticLoadBalancingV2::Listener
  Properties:
    LoadBalancerArn: !Ref ApplicationLoadBalancer
    Port: 443
    Protocol: HTTPS
    SslPolicy: ELBSecurityPolicy-TLS13-1-2-2021-06
    Certificates:
      - CertificateArn: !Ref CertificateArn
    DefaultAction:
      Type: forward
      ForwardConfig:
        TargetGroups:
          - TargetGroupArn: !Ref OutpostsTargetGroup
            Weight: 100
          - TargetGroupArn: !Ref RegionTargetGroup
            Weight: 0
        TargetGroupStickinessConfig:
          Enabled: true
          DurationSeconds: 300

RegionTargetGroup:
  Type: AWS::ElasticLoadBalancingV2::TargetGroup
  Properties:
    Name: burst-region-targets
    Protocol: HTTP
    Port: 80
    VpcId: !Ref VpcId
    HealthCheckEnabled: true
    HealthCheckIntervalSeconds: 30
    HealthCheckPath: /health
    HealthyThresholdCount: 2
    UnhealthyThresholdCount: 3
    TargetGroupAttributes:
      - Key: deregistration_delay.timeout_seconds
        Value: "300"
      - Key: slow_start.duration_seconds
        Value: "120"

Note on stickiness: Target group stickiness keeps a client pinned to whichever target group served its first request for DurationSeconds. We set this to 300 seconds (5 minutes) to match the Amazon EventBridge evaluation interval. This balances session consistency for stateful workloads against the need for weight changes to take effect within a reasonable window. For purely stateless workloads, you can disable stickiness entirely to allow immediate weight convergence. For workloads requiring longer session affinity, increase the duration but understand that weight transitions will converge more slowly — existing sticky sessions continue going to the original target group until they expire.

Traffic weight progression

Use stepped transitions rather than abrupt weight changes. The following table shows the recommended progression:

Phase	Outposts weight	Region weight	Condition to advance
Normal	100	0	Steady state
Burst step 1	90	10	Region target group has at least 1 healthy host
Burst step 2	70	30	Region target group healthy for 2 consecutive checks
Burst step 3	50	50	Only if Outposts capacity exceeds 95% used
Recovery step 1	80	20	Outposts capacity below 70%
Recovery step 2	100	0	Outposts capacity below 60% for 2 checks

Avoid jumping directly from 0% to 50% Region traffic. Cold overflow instances need time to warm caches and stabilize before absorbing significant load.

Best practices

Apply these best practices to get the most from this pattern while avoiding common pitfalls.

Traffic tiering

Classify your workloads into two tiers at the ALB listener level. Latency-critical paths use routing rules with the Outposts target group only. These never overflow regardless of capacity state. Overflow-eligible paths use the weighted forwarding rule. This separation helps make sure that your most latency-sensitive flows are not impacted by the burst mechanism.

Managing data gravity

For stateless workloads, Burst to Region requires no special data handling. For workloads with session state or shared data:

Store session state in Amazon ElastiCache or Amazon DynamoDB rather than local instance memory. Both Outposts and Region instances access the same session store.
If your application reads from a local database on Outposts, overflow instances need a Region-accessible replica. Consider Amazon Relational Database Service (Amazon RDS) read replicas or DynamoDB global tables.

Anti-pattern: Do not burst workloads that write to Outposts-local storage and expect synchronous consistency. The latency and complexity of cross-location writes defeats the purpose of the pattern.

Cost optimization

The overflow fleet consumes On-Demand pricing by default since it starts at zero and scales only during peaks.

Burst profile	Recommended pricing	Rationale
Unpredictable spikes (minutes)	On-Demand	Maximum flexibility, no commitment waste
Predictable daily peaks (hours)	Savings Plans (Compute)	Covers overflow hours at discount
Frequent, long bursts	Reserved capacity plus On-Demand	Baseline discount plus burst flexibility

Monitor your BurstActive custom metric over time. If overflow is active more than 30% of the time, you likely need additional Outposts capacity rather than relying on Region overflow.

Security consistency

Maintain identical security posture across both environments:

Use the same security group rules for Outposts and Region instances.
Deploy with AWS CloudFormation StackSets to support consistency.
Share the same IAM instance profile. The overflow launch template references the same role as your Outposts instances.
Apply the same AWS Systems Manager patch baselines and compliance rules to both fleets.

Observability

Build a CloudWatch dashboard that provides visibility into burst state and performance. The SAM template in the repository deploys a pre-configured dashboard tracking:

Burst status: Custom BurstActive metric (1 = active, 0 = normal)
Capacity headroom: UsedInstanceType_Count compared to AvailableInstanceType_Count. Note that UsedInstanceType_Count includes instances consumed by managed services (Amazon RDS, ALB), so your available application capacity may be lower than the raw availability count suggests.
Overflow fleet size: Auto Scaling group GroupInServiceInstances.
Latency comparison: TargetResponseTime per target group (Outposts compared to Region)
Traffic distribution: RequestCount per target group.

Set a CloudWatch alarm on Region target group TargetResponseTime exceeding your acceptable threshold. This provides early warning if overflow latency degrades beyond your tolerance.

Service link considerations

Because the ALB resides in the Region, all traffic to Outposts targets traverses the service link. Keep the following in mind:

Bandwidth planning: Steady-state traffic to Outposts targets flows over the service link. Verify that your connection meets the minimum 500 Mbps per compute rack recommended by AWS, with sufficient headroom for both application traffic and Outposts control plane communication. Monitor service link VIF throughput using IfTrafficIn and IfTrafficOut metrics (on service link VIFs) to detect saturation before it impacts performance.

Latency impact: The service link adds latency compared to a locally deployed load balancer. The exact impact depends on your service link connection type and distance to the parent Region (AWS specifies a maximum of 175 ms round-trip for service link). For internet-facing workloads, this is typically negligible relative to the client-to-Region round trip. For workloads serving on-premises users through the Local Gateway, consider Route 53 weighted routing between an ALB on Outposts and a separate ALB in the Region instead.

Connection draining: When scaling down the overflow fleet, allow sufficient time for in-flight requests to complete. The deregistration delay configured on the target group (default 300 seconds) and the Auto Scaling scale-in cool-down period work together to help provide graceful termination and minimize the risk of dropping active connections.

Failure modes: If the service link goes down, the ALB cannot reach Outposts targets. Health checks fail, and all traffic automatically shifts to Region targets. This provides an unintentional but useful failover behavior. However, note that the overflow fleet is sized for burst capacity, not for sustaining 100% of production traffic. Monitor the ConnectedStatus metric (under the AWS/Outposts namespace, dimension OutpostId) and alert on degradation. If you need full failover capability, architect a separate disaster recovery solution with appropriately sized Region capacity.

Limitations

Be aware of these constraints when implementing this pattern:

ALB requirement: The pattern requires an Application Load Balancer in the Region. Workloads that rely on direct IP access through the Local Gateway (without an ALB) cannot use this pattern without an architecture change.
Stateful workloads: Applications with local disk state or in-memory sessions require external session stores (ElastiCache, DynamoDB) before they can burst. Without this, overflow instances serve requests without session context.
Database coupling: If your application writes to a database running exclusively on the Outpost, overflow instances in the Region cannot reach it without a cross-location replica or proxy. Read-heavy workloads with a Region read replica are ideal candidates.
Service link as single path: All ALB-to-Outpost traffic shares the service link with AWS control plane operations. Under extreme load, bandwidth contention can degrade both application traffic and management operations.
ALB on Outposts: As of this writing, ALB on Outposts does not support weighted target groups spanning both locations. The ALB must reside in the Region for this pattern to work.

Testing the pattern

Validate the burst mechanism before relying on it in production:

Simulate capacity pressure:

aws cloudwatch set-alarm-state \
  --alarm-name outposts-capacity-high \
  --state-value ALARM \
  --state-reason "Testing burst mechanism"

Verify overflow fleet launched:

aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names burst-overflow-fleet \
  --query "AutoScalingGroups[0].DesiredCapacity"

Verify ALB weights shifted (after recovery check runs):

aws elbv2 describe-listeners \
  --listener-arns <your-listener-arn> \
  --query "Listeners[0].DefaultActions[0].ForwardConfig.TargetGroups[*].[TargetGroupArn,Weight]"

Trigger recovery:

aws cloudwatch set-alarm-state \
  --alarm-name outposts-capacity-high \
  --state-value OK \
  --state-reason "Testing recovery"

Confirm overflow fleet scales back to zero and all traffic returns to Outposts targets. Recovery is gradual — the Amazon EventBridge rule evaluates every 5 minutes and steps weights back before scaling down, so full recovery may take 10–15 minutes depending on your weight progression configuration.

Clean up

To avoid ongoing charges, verify that the overflow Auto Scaling group has scaled to zero, then delete the stack:

sam delete --stack-name burst-to-region-stack

This removes all resources created by the template, including the Lambda function, CloudWatch alarm, SNS topic, Amazon EventBridge rule, and the overflow Auto Scaling group.

Conclusion

This Burst to Region pattern extends AWS Outposts capacity into the parent Region during peak demand. You trade a moderate latency increase for continued availability when local capacity is exhausted.

The pattern works best when you clearly classify which workloads can overflow, implement gradual traffic transitions, and maintain security and observability parity across both environments.

For the complete deployable AWS SAM template including the Lambda orchestrator, CloudWatch dashboard, and all IAM roles, see the GitHub repository. To learn more about capacity planning for Outposts, see Managing your AWS Outposts capacity using Amazon CloudWatch and AWS Lambda and AWS Outposts monitoring and reporting: A comprehensive Amazon EventBridge solution.

For more information, see the AWS Outposts User Guide and the Amazon EC2 Auto Scaling User Guide.

Low-latency, high-throughput SQS event processing with AWS Lambda provisioned mode

Ben Freiberg — Mon, 10 Aug 2026 13:42:03 +0000

Customers building event-driven applications on AWS rely on Amazon Simple Queue Service (Amazon SQS) and AWS Lambda event source mappings (ESMs) to process millions of events every day. The fully managed polling infrastructure of ESMs eliminates the need to write and maintain custom code. You can focus on business logic while Lambda handles scaling, batching, and error handling automatically.

As workloads grow, many customers need to meet demanding requirements for low-latency message processing, high-concurrency execution, and high-throughput event processing. Use cases such as real-time payment processing, fraud detection, IoT telemetry pipelines, and flash-sale order fulfillment require the ESM to scale rapidly and sustain peak performance without queue backlog.

To address these needs, AWS launched provisioned mode for SQS event source mappings. Provisioned mode gives you direct control over the number of event pollers assigned to your ESM for predictable and rapid scaling. With Provisioned mode, you can configure event pollers up to 10,000, supporting concurrency of up to 100,000 concurrent Lambda executions and throughput of 10 GB/s. You can process up to a million events per second.

Provisioned mode is also available for Apache Kafka event source mappings including Amazon Managed Streaming for Apache Kafka (Amazon MSK) and self-managed Kafka.

How SQS event source mappings work

When you configure an SQS queue as an event source for a Lambda function, Lambda automatically creates an ESM resource. The ESM manages a fleet of internal event pollers that continuously poll the SQS queue, retrieve messages, and invoke your Lambda function with batches of events.

In default ESM mode, Lambda automatically manages the number of event pollers based on queue depth and processing throughput. The system starts with five pollers and scales up as the queue backlog builds, supporting up to 1,250 concurrent invocations. This automatic scaling works well for the majority of event processing workloads. However, the scale-up rate in default mode (approximately 300 additional concurrent executions per minute) can leave latency-sensitive workloads with growing queue backlogs during sudden traffic spikes.

What is provisioned mode?

Provisioned mode gives you explicit control over the minimum and maximum number of event pollers assigned to your ESM. Instead of relying solely on automatic scaling, you define:

MinimumPollers: the number of event pollers always active and ready to process messages (range: 2–200).
MaximumPollers: the upper bound on event pollers the ESM can scale to (range: 2–10,000).

These pollers remain active and continuously poll your SQS queue, eliminating cold-start delays in the polling infrastructure. When traffic spikes arrive, your ESM already has capacity allocated to handle the burst.

Default mode vs. provisioned mode

Attribute	Default mode	Provisioned mode
Minimum event pollers	2	2
Maximum event pollers	5	10,000
Maximum concurrent executions	Up to 1250	Up to 100,000
Maximum throughput	N/A	10 GB/s
Scale-up rate	~300 concurrency/min	~1,000 concurrency/min
Poller control	Lambda controlled	Min/Max configurable by you
Billing	Included in Lambda pricing	Event poller unit (EPU) hours
Best for	Majority of workloads	Spiky, latency-sensitive, high-throughput workloads

Activating provisioned mode for ESM

You can configure provisioned mode when creating a new ESM or updating an existing one. The following examples show configuration using the AWS CLI, AWS Serverless Application Model (SAM), and AWS CloudFormation.

AWS CLI

Create a new ESM with provisioned mode:

aws lambda create-event-source-mapping \
  --function-name my-function \
  --event-source-arn arn:aws:sqs:us-east-1:123456789012:my-queue \
  --batch-size 10 \
  --provisioned-poller-config '{"MinimumPollers": 50, "MaximumPollers": 500}'

Update an existing ESM to enable provisioned mode:

aws lambda update-event-source-mapping \
  --uuid "a1b2c3d4-5678-90ab-cdef-EXAMPLE11111" \
  --provisioned-poller-config '{"MinimumPollers": 50, "MaximumPollers": 500}'

AWS SAM template

Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: index.handler
      Runtime: python3.12
      Events:
        SQSEvent:
          Type: SQS
          Properties:
            Queue: !GetAtt MyQueue.Arn
            BatchSize: 10
            ProvisionedPollerConfig:
              MinimumPollers: 50
              MaximumPollers: 500

  MyQueue:
    Type: AWS::SQS::Queue

AWS CloudFormation

Resources:
  MyEventSourceMapping:
    Type: AWS::Lambda::EventSourceMapping
    Properties:
      FunctionName: !Ref MyFunction
      EventSourceArn: !GetAtt MyQueue.Arn
      BatchSize: 10
      ProvisionedPollerConfig:
        MinimumPollers: 50
        MaximumPollers: 500

Provisioned mode for SQS ESM in action

To see the performance profile with provisioned mode for SQS, deploy a Lambda function that has an SQS queue as its trigger. Use the reference pattern on Serverless Land or follow the Creating and configuring an Amazon SQS event source mapping guide to configure provisioned mode for your SQS event source mapping. In the following scenarios, a producer writes 40 million messages, each with a 1 KB payload size, to an SQS queue. Batch size is set to 10, with function duration at about 200 ms.

Scenario 1: Baseline (default mode)

The following chart shows the relationship between ApproximateNumberOfMessagesVisible (blue) and ConcurrentExecutions (orange) over time for the baseline scenario using default mode with no provisioned pollers. With provisioned mode disabled, Lambda takes approximately 17 minutes to drain the backlog of 40 million messages. It takes about 6 minutes to reach the maximum concurrent executions.

Scenario 2: Configuring minimum event pollers and auto-scaling

To optimize the ESM throughput for these kinds of workloads and reduce the time to drain the message backlog, set the minimum event pollers to a higher than default value. In this scenario, the minimum pollers are set to 100 and maximum pollers are set to 1000.

Lambda drains the backlog of 40 million messages in approximately 7 minutes. This is more than 55% faster than the baseline without provisioned mode. It takes only about 1 minute to reach maximum concurrent executions.

Scenario 3: Default minimum event pollers and auto-scaling

In some cases, the workload might not be as performance-sensitive. With the same volume of 40M messages in your SQS queue, activate provisioned mode for ESM. Start with the default minimum event pollers (set to 2) and let Lambda automatically scale the event pollers based on incoming traffic.

With this configuration, Lambda drains the backlog in approximately 9 minutes. This is still more than 45% faster than the baseline without provisioned mode. It takes about 3 minutes to reach maximum concurrent executions.

Best practices for configuring provisioned pollers

When configuring provisioned mode, keep the following recommendations in mind:

Right-size your minimum pollers

Each event poller supports up to 10 concurrent Lambda invocations and approximately 1 MB/s throughput. Use this formula to estimate your minimum poller count:

MinimumPollers = max(TargetConcurrency / 10, TargetThroughputMBps / 1)

For example, if your workload requires 500 concurrent executions and 200 MB/s throughput, set MinimumPollers to at least 200. To estimate the number of event pollers required to verify optimal message processing performance when using provisioned mode for SQS ESM, follow the steps described in determining the required event pollers.

Set maximum pollers for burst capacity

Set MaximumPollers to handle your peak traffic scenario. The ESM scales between your minimum and maximum based on queue depth. A good starting point is 2–5x your minimum pollers.

Align with Lambda concurrency limits

Provisioned pollers invoke your Lambda function concurrently. Verify that your account’s concurrent execution quota accommodates the maximum concurrency your pollers can drive:

MaxConcurrency = MaximumPollers × 10

If you set MaximumPollers to 5,000, your account needs at least 50,000 concurrent execution capacity. Request a quota increase through the Lambda quotas page if needed.

Start conservatively and iterate

Begin with a lower MinimumPollers value and monitor the CloudWatch metrics described in the following section. Increase the minimum if you observe queue depth growth during traffic spikes, or decrease it if pollers remain underutilized during off-peak hours.

Use FIFO queues for ordered workloads

When processing order-sensitive workloads, use FIFO queues with high-throughput mode activated. Provisioned mode works with both SQS standard and FIFO queue types.

Set up dead-letter queues

Configure dead-letter queues to manage messages that fail processing after multiple attempts.

Adjust batch size as needed

The batch size parameter remains adjustable, with a default value of 10 messages and a maximum of 10,000 messages for standard queues.

Cost considerations

Provisioned mode billing is based on event poller unit (EPU) hours. You pay for the number of provisioned pollers allocated, regardless of whether they are actively processing messages. See AWS Lambda pricing for details. Key optimization strategies are:

Match minimum pollers to your sustained baseline traffic to avoid over-provisioning during low-traffic periods.
Use maximum pollers for burst capacity as you only pay for pollers that scale up while they are active.

Monitoring provisioned mode with CloudWatch

Lambda publishes the following CloudWatch metrics for provisioned mode ESMs:

Metric	Description
ProvisionedPollers	Current number of provisioned event pollers allocated
ConcurrentExecutions	Number of concurrent Lambda invocations driven by the ESM
ApproximateNumberOfMessagesVisible	SQS queue depth (from SQS metrics)
Duration	Function execution time per invocation

Set CloudWatch alarms on ApproximateNumberOfMessagesVisible to detect queue backlogs, and on ProvisionedPollers to track the number of provisioned pollers. To understand how your ESM processes messages at each stage, from polling through invocation to completion, opt in to the EventCount metric group. This provides detailed metrics including PolledEventCount, FilteredOutEventCount, InvokedEventCount, FailedInvokeEventCount, and DeletedEventCount.

Conclusion

Provisioned mode for SQS event source mappings gives you control over scaling behavior for your most demanding workloads. By configuring minimum and maximum event pollers, you achieve predictable low-latency processing, scale to 100,000 concurrent executions, and sustain throughput of up to a million events per second, without waiting for automatic scale-up.

Dedicated pollers deliver predictable, low-latency performance. This makes them well-suited for workloads like real-time financial transactions, high-volume IoT data ingestion, or flash sale order processing. You can achieve 3x faster scaling compared to default mode. Combined with CloudWatch observability and flexible configuration through CLI, SAM, and CloudFormation, provisioned mode integrates into your existing deployment workflows.

To get started, explore the provisioned mode configuration guide for SQS event source mappings. Deploy the sample application from the Serverless Land reference pattern. To request a concurrent execution quota increase for high-throughput workloads, visit the Lambda quotas page.

Serverless ICYMI Q2 2026

Julian Wood — Mon, 20 Jul 2026 16:40:00 +0000

In this 33rd quarterly recap post, discover the most impactful AWS serverless launches, features, and resources from Q2 2026 that you might have missed. Stay current with the latest serverless innovations that can improve your applications.

In case you missed our last ICYMI, read about what happened in Q1 2026.

AWS Lambda MicroVMs

AWS Lambda MicroVMs is a new serverless compute primitive for running user or AI-generated code in isolated, stateful execution environments. Built on the same Firecracker virtualization that powers over 15 trillion monthly Lambda invocations, MicroVMs give you VM-level isolation with near-instant launch and resume. Each MicroVM runs in its own Linux environment with no shared kernel or resources between sessions. This isolation makes it a useful solution for AI coding assistant sandboxes, interactive code or multi-tenant development environments, CI/CD build environments, data analytics platforms, vulnerability scanners, and game servers that run user-supplied scripts.

Standard Lambda functions are best for event-driven, request-response workloads which have a 15-minute timeout. MicroVMs are purpose-built for single end user or session workloads and can preserve state for up to 8 hours. You get full lifecycle controls including launch, suspend, resume, and terminate. You can suspend them during the 8 hours if you don’t need them active. MicroVMs retain memory and disk state for the length of the session, even while suspended. They can auto resume when you need to use them again.

Serverless Land contains example applications and a resources page with more details. The Serverless Office Hours live stream has more explanations and live demos.

Amazon S3 Files and Lambda integration

Amazon S3 Files makes your S3 buckets accessible as high-performance file systems. S3 files is a fully featured, POSIX-compatible file system to access to your data with approximately 1ms latency.

For serverless workloads, the Lambda integration with S3 Files lets your functions mount an S3 bucket as a local file system. Your function reads and writes files at a local mount path like /mnt/data, and the file system handles synchronization with S3 automatically. You can avoid downloading objects to /tmp from S3 within your function and work directly with files. Applications that assume a file system can now run on Lambda without rewriting their I/O layer. Use cases include sharing data between functions, ML model loading, document processing, media transcoding, or any pipeline that treats data as files rather than objects.

AWS Lambda durable functions

The Lambda durable functions SDK for Java is now generally available, joining Python and TypeScript. This allows Java developers to build multi-step workflows with automatic checkpointing and recovery without adding external orchestration. Durable functions is also now available in 16 additional AWS Regions. Learn how to build fault-tolerant multi-agent AI workflows to coordinate multiple AI agents that call tools, make decisions, and hand off work. There is automatic recovery if any agent fails mid-task. Voice analytics with Amazon Bedrock shows building a pipeline that processes call recordings through transcription, sentiment analysis, and summarization with durable checkpoints between each stage. For best practices, AI patterns, and futures, view the live stream.

AWS Lambda Managed Instances

Lambda Managed Instances now allows you to build memory-intensive apps with up to 32 GB (3x more than standard Lambda). This allows use cases like in-memory caching, large dataset analytics, and ML inference that previously required considering other services.

Figure 1 — AWS Lambda Managed Instances for memory-intensive apps architecture

Scheduled scaling lets you pre-warm capacity for predictable traffic patterns with Amazon EventBridge Scheduler. This helps reduce cold start latency during known demand spikes. Tag propagation automatically applies your function tags to the underlying Amazon EC2 instances, Amazon Elastic Block Store volumes, and network interfaces. This helps finance teams with cost allocation visibility without manual tag management.

Other Lambda updates

Response streaming is now available in all commercial AWS Regions, bringing full regional parity for progressively streaming data back to clients. This is useful for LLM-powered applications where users expect to see tokens as they generate rather than waiting for a complete response.

The tenant isolation mode now integrates with Event Source Mappings from Amazon SQS, Amazon Kinesis, and Amazon EventBridge. Multi-tenant SaaS applications can process messages in isolated execution environments without building custom routing logic.

If you have a fleet of functions on older runtimes, you can now upgrade runtimes at scale using AWS Transform custom. This uses AI to analyze your function code, identify breaking changes for the target runtime version, and generate the code modifications needed. This can help teams save manual migration effort across many functions. The Serverless Office Hours live stream has more information.

Lambda added the Ruby 4.0 runtime. In addition to providing access to the latest Ruby language features, Lambda adds support for Lambda advanced logging controls.

AWS Serverless Application Model (AWS SAM) CLI now supports BuildKit for building container images from Dockerfiles. This allows faster multi-stage builds with better caching, cross-architecture image builds, and Docker secrets to keep credentials out of final image layers.

Containers with Mama J

Serverless with Mama J

Mama J is back in the second video of a series where Eric Johnson explains what he does all day at work to his mother. Previously, they talked serverless and Lambda. This time it’s containers, what they are, why they exist, and how AWS manages them at scale. Eric goes through the “it works on my machine” problem, how Docker builds images, container orchestration and how containers differ from Lambda.

View the video on the AWS Developers YouTube channel.

AWS Step Functions

AWS Step Functions has an Amazon Bedrock AgentCore-powered agentic reasoning step. You can embed AI agent reasoning directly inside a workflow as a native step type. This bridges structured orchestration with autonomous agent behavior. Your workflow handles the deterministic parts such as branching, retries, timeouts, parallel execution, while the agentic step handles the parts that require flexible reasoning.

Amazon EventBridge

Amazon EventBridge Scheduler added 619 new SDK API actions as targets, including Lambda Managed Instances operations. This means you can schedule calls to a much broader set of AWS APIs without writing a Lambda function.

A new post walks through building a multi-Region event-driven failover architecture with Amazon EventBridge and Amazon Route 53. The pattern uses Amazon EventBridge global endpoints with Route 53 health checks to automatically route events to a healthy Region during failures. This provides active-active or active-passive resilience for event-driven workloads.

Amazon Bedrock AgentCore

The Amazon Bedrock AgentCore harness reached general availability. Two API calls give you a running agent in seconds which runs in its own isolated environment with a filesystem and shell. It can read files, run commands, and write code safely.

AgentCore Payments (preview) allows agents to autonomously access and pay for APIs and MCP servers, opening up agent-to-agent commerce. AgentCore Memory has metadata for long-term memory so agents retain and recall context across sessions. Web Search on AgentCore grounds agents in current, cited web knowledge. The Runtime now supports bring-your-own file systems from S3 Files and Amazon Elastic File System, and Node.js for direct code deployment.

Strands Agents SDK

The open source Strands Agents SDK shipped three capabilities. Context management that cuts token costs in half by intelligently pruning what goes into the model context window, Strands Shell for sandboxed agent code execution, and Strands Evals 1.0 with chaos testing and adversarial red teaming. This can reduce costs to help make production agent workloads cheaper without sacrificing quality. A Serverless Office Hours live stream covered the new features in depth.

The TypeScript SDK reached general availability, giving JavaScript and TypeScript developers the same model-driven agent framework. Erik Hanchett ran this live stream with more details. A new blog post on building research assistants with Strands walks through the full app from prototype to working application in about 200 lines of Python.

Agent Toolkit for AWS and AI coding

The Agent Toolkit for AWS became generally available with three plugins (aws-core, aws-agents, aws-data-analytics), over 30 curated skills, and the AWS MCP Server. View this video for an introduction. This gives AI coding agents such as Kiro, Claude Code, and Cursor expert AWS knowledge which helps to reduce errors and lower token costs. For more information on the serverless tools available when using AI, see this Serverless Land resources page.

Serverless Office Hours ran a live stream series finding out how experts use AI to build serverless applications. Hear from:

Ran Isenberg, Serverless Hero on how he ships with Claude every day and what actually moves the needle.
AWS Community Builder, Darryl Ruggles on how he built a full-featured blogging platform on AWS serverless services using Claude Code and MCP servers.
Mark Sailes shows how serverless experts built Study from Experts, a focused video learning platform for AWS professionals.
Brian Zambrano and Sean Kendall walk through how to build a serverless application from prompts using Kiro and MCP servers.

Kiro launched Kiro Pro Max and an iOS mobile app for approving and monitoring agentic coding sessions from your phone. Amazon Q Developer IDE plugins are transitioning to Kiro. The Kiro power for AWS DevOps Agent connects your IDE directly to production intelligence. You can investigate incidents and generate fixes without context switching.

Serverless blog posts

April

June

Serverless Office Hours

Join our live stream every Tuesday at 11 AM PT for live discussions, Q&A sessions, and deep dives into serverless technologies. View episodes on-demand at serverlessland.com/office-hours.

Still looking for more?

The Serverless landing page has overall information about building serverless applications. The Lambda resources page contains case studies, webinars, whitepapers, customer stories, reference architectures, and even more Getting Started tutorials.

You can also follow the Developer Advocacy team to get the latest news, follow conversations, and interact with the team.

Julian Wood: @julian_wood, https://www.linkedin.com/in/julianrwood/
Eric Johnson: @edjgeek, https://www.linkedin.com/in/singledigit/
Erik Hanchett: @ErikCH, https://www.linkedin.com/in/erikhanchett/
Salih Gueler: @salihgueler, https://www.linkedin.com/in/salihgueler/
Marcia Villalba: @mavi888uy, https://www.linkedin.com/in/marciavillalba

And finally, visit Serverless Land for your serverless needs.

Introducing self-managed Amazon S3 buckets for AWS Lambda function code

Doug Perkes — Fri, 17 Jul 2026 10:49:14 +0000

If you manage Lambda functions at scale, you’ve likely hit the 75 GB code storage limit or explained to your security team why deployment artifacts live in an S3 bucket you don’t control. Today, we’re announcing self-managed Amazon S3 buckets for AWS Lambda deployment packages. Lambda reads your code directly from your bucket, eliminating quota pressure and giving you full security control.

Previously, the default AWS-managed code storage created three challenges at scale. First, all copies count toward your 75 GB code storage quota. Second, you cannot apply your own encryption, access controls, or compliance tags to the internal bucket. Third, the copy cannot be incorporated into your disaster recovery strategies.

With self-managed S3 buckets, Lambda reads your function code directly from your bucket. No copy, no duplication. Your S3 object becomes the single source of truth for your functions. Deployment packages no longer count against your account’s code storage limit. You manage the bucket’s security and compliance posture: choosing the encryption, defining the access policies, managing lifecycle transitions, and maintaining the audit trail. And because you own the bucket, you can use S3 Cross-Region Replication to maintain fallback copies of your code in a secondary Region, so that your functions remain deployable even if your primary Region experiences an issue. Using self-managed S3 buckets also results in a faster time to first invoke for new functions and after function updates, because Lambda no longer needs to copy your zip package to a Lambda-managed S3 bucket.

You can use this feature today in all AWS standard regions where Lambda is available, at no additional charge beyond your standard Amazon S3 storage and request costs. Let’s look at some use cases, how it works, and how to use it at scale.

Use cases

Here are a few patterns where owning your deployment bucket makes a real difference.

CI/CD pipelines and artifact management

With self-managed storage, your CI/CD pipeline uploads once, and Lambda references the same object. One set of lifecycle rules and access controls covers all artifacts, and rollbacks mean pointing the function to a previous S3 object version.

Multi-account and multi-team architectures

Organizations using AWS Organizations often separate workloads into multiple accounts: a development account, a staging account, and a production account. They centralize shared resources in a tooling or shared-services account.

Self-managed buckets integrate naturally with this pattern:

Store all deployment artifacts in a central “artifact account” bucket.
Grant cross-account s3:GetObject access to Lambda execution roles in each workload account through bucket policies.
Maintain a single inventory of what code is deployed where, managed by your platform or DevOps team.
Enforce consistent encryption, versioning, and retention policies from one place.

Disaster recovery and business continuity

Because your deployment artifacts live in a bucket you own, you can use the built-in replication features of S3, Cross-Region Replication (CRR) or Same-Region Replication (SRR), to maintain copies of your code artifacts in backup locations. Combined with S3 Versioning and Object Lock, this gives you a durable, tamper-proof code archive that supports rapid recovery if a deployment is accidentally corrupted or deleted.

How it worked before

When you deploy a Lambda function using a .zip deployment package stored in Amazon S3, the process has traditionally worked like this:

You upload your .zip deployment package to your S3 bucket.
You call CreateFunction or UpdateFunctionCode, specifying the S3 bucket and S3 key.
Lambda copies the .zip artifact from your bucket into an internal, service-managed bucket.
Lambda uses this copy to create the optimized version of your function that runs at invocation time.
The copied artifact counts toward your account’s 75 GB code storage quota.

Figure 1 — Standard Lambda deployment

This model is straightforward and works well for most workloads. However, it creates three friction points at scale:

Storage quota pressure: Every deployment package copy counts toward your account’s 75 GB total code storage limit. Organizations with hundreds of functions and multiple published versions can exhaust this quota.
No control over stored artifacts: You cannot configure encryption (beyond the service default), access logging, lifecycle policies, Object Lock, or compliance tags on the internal bucket.
Redundant storage: Your original artifact remains in your bucket while a copy lives in the Lambda bucket used for provisioning new instances of your Lambda function.

What’s new: REFERENCE mode

This launch introduced a new function configuration setting, S3ObjectStorageMode. The default value is COPY, which provides the existing behavior described in the preceding section. To enable self-managed S3 buckets, set S3ObjectStorageMode to REFERENCE when creating or updating a function. In this mode, Lambda no longer copies your deployment package. Instead, it stores a reference to your S3 object and reads the code directly from your bucket when needed. If you do not specify S3ObjectStorageMode, Lambda still takes a copy by default.

Figure 2 — Lambda deployment with self-managed storage

This gives you:

No quota consumption. Deployment packages in your bucket don’t count against the 75 GB Function and layer storage account limit.
Improved performance. Lambda no longer copies the code to an internal bucket, so function creation and updates are faster.
Full security and compliance control. Apply your own bucket policies, encryption, Object Lock, versioning, access logging, and compliance tags.
Single source of truth. Your S3 object is the canonical artifact with no additional copies and no drift.
Disaster recovery options. Use S3 Cross-Region Replication to maintain fallback copies in a secondary Region.

How it works

To use this feature, specify the S3ObjectStorageMode parameter when creating or updating your function.

Creating a new function (AWS CLI):

aws lambda create-function \
  --function-name my-function \
  --runtime python3.13 \
  --role arn:aws:iam::123456789012:role/my-lambda-role \
  --handler app.handler \
  --code S3Bucket=amzn-s3-demo-bucket,S3Key=deployments/my-function.zip,S3ObjectVersion=abc123,S3ObjectStorageMode=REFERENCE

Updating an existing function:

aws lambda update-function-code \
  --function-name my-function \
  --s3-bucket amzn-s3-demo-bucket \
  --s3-key deployments/my-function.zip \
  --s3-object-version def456 \
  --s3-object-storage-mode REFERENCE

AWS Identity and Access Management (IAM) permissions

Lambda needs permission to read the deployment package from your bucket. You can grant access through an S3 bucket policy.

S3 bucket policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "LambdaSelfManagedCodeAccess",
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion"
      ],
      "Resource": "arn:aws:s3:::amzn-s3-demo-bucket/deployments/my-function.zip",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "arn:aws:lambda:us-east-1:123456789012:function:my-function"
        }
      }
    }
  ]
}

We recommend including the aws:SourceArn condition key scoped to your specific function ARN to allow for least-privileged access. Note the Resource is scoped to the exact S3 key rather than a wildcard prefix. This follows least-privilege and matches how the aws:SourceArn condition locks down which function can access which object.

Bucket requirements

Your S3 bucket must meet the following requirements:

Versioning (required). You must enable S3 versioning to make sure that Lambda references a specific, immutable artifact and to protect against accidental overwrites.
Encryption. The following encryption types are supported: SSE-S3, SSE-KMS (including customer-managed KMS keys), and DSSE-KMS. If you use SSE-KMS, the Lambda principal must have kms:Decrypt permission on the key.
Object Lock. Supported. You can apply Object Lock in Compliance or Governance mode to prevent accidental deletion of deployment artifacts.
Access logging. You can enable S3 server access logging or AWS CloudTrail data events to audit every time Lambda reads your code.

What happens when the object is unavailable

Lambda periodically accesses the source object from your S3 bucket to reoptimize your function code. You must maintain access to the source object for your function to remain active.

If Lambda loses access to the source object for a function, the function transitions to the Inactive state. To restore the function, restore access to the source object and then update the function.

Performance considerations

Lambda functions with self-managed code storage behave the same as standard Lambda functions with one difference during function creation and update. Lambda does not copy your deployment package to a Lambda-managed S3 bucket. In our testing with a 200MB Python 3.13 function, functions using self-managed storage showed function creation times approximately 5s less than the default COPY mode. Reading directly from your S3 bucket without an intermediate copy step can provide a modest advantage, particularly for larger deployment packages.

Getting started with infrastructure as code

Self-managed code storage can be implemented using infrastructure-as-code tooling with either the AWS CLI or AWS CloudFormation today.

AWS CLI

aws lambda create-function \
  --function-name my-function \
  --runtime python3.13 \
  --role arn:aws:iam::123456789012:role/my-lambda-role \
  --handler app.handler \
  --code S3Bucket=amzn-s3-demo-bucket,S3Key=deployments/my-function.zip,S3ObjectVersion=abc123,S3ObjectStorageMode=REFERENCE \
  --region us-east-1

AWS CloudFormation

MyFunction:
  Type: AWS::Lambda::Function
  Properties:
    FunctionName: my-function
    Runtime: python3.13
    Handler: app.handler
    Role: !GetAtt MyLambdaRole.Arn
    Code:
      S3Bucket: amzn-s3-demo-bucket
      S3Key: deployments/my-function.zip
      S3ObjectVersion: abc123
      S3ObjectStorageMode: REFERENCE

Using it at scale

Once you adopt self-managed S3 buckets for your Lambda deployment packages, your artifact bucket grows over time as you deploy new versions of your functions. This section covers strategies for managing that growth efficiently, keeping your versions organized, and planning for cross-Region deployments.

Managing artifact lifecycle with S3 lifecycle policies

Every time you update a function’s code, S3 creates a new object version in your bucket. The previous objects don’t disappear. They accumulate. Without a cleanup strategy, your storage grows indefinitely and old artifacts clutter your bucket.

S3 Lifecycle policies let you automate this entirely. You define rules that transition or delete objects based on age, and S3 executes them on your behalf: no scripts, no cron jobs, no manual intervention.

Strategy 1: Archive old versions to Glacier

If compliance or audit requirements mandate that you retain all historical deployment packages, but you rarely need to access them, transition old object versions to a lower-cost storage class:

{
  "Rules": [
    {
      "ID": "ArchiveOldDeploymentPackages",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "deployments/"
      },
      "NoncurrentVersionTransitions": [
        {
          "NoncurrentDays": 30,
          "StorageClass": "GLACIER_FLEXIBLE_RETRIEVAL"
        }
      ]
    }
  ]
}

This rule transitions any non-current object version to S3 Glacier Flexible Retrieval after 30 days. Your active deployment packages remain in S3 Standard for fast access, while historical versions move to archival storage at a fraction of the cost.

For artifacts you need to retain for years but will rarely access again, consider a tiered approach: moving to Glacier Flexible Retrieval first, then to Deep Archive:

"NoncurrentVersionTransitions": [
  {
    "NoncurrentDays": 30,
    "StorageClass": "GLACIER_FLEXIBLE_RETRIEVAL"
  },
  {
    "NoncurrentDays": 365,
    "StorageClass": "DEEP_ARCHIVE"
  }
]

Strategy 2: Delete old versions you no longer need

If you don’t have a compliance requirement to retain every historical artifact, you can expire old versions outright:

{
  "Rules": [
    {
      "ID": "DeleteOldDeploymentPackages",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "deployments/"
      },
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": 14,
        "NewerNoncurrentVersions": 2
      }
    }
  ]
}

This rule keeps the 2 most recent non-current versions of each object (giving you a rollback path) and deletes anything older than 14 days beyond that. This aligns well with a deployment strategy where you want the ability to quickly roll back to your previous one or two releases, but don’t need to retain anything older.

Tracking object and function versions

With REFERENCE mode, there is a direct relationship between your S3 object version and your Lambda function version. We recommend the following practices:

Tag your objects with metadata from your CI/CD pipeline (commit SHA, build ID, pipeline run ID) so you can trace any deployed function back to the exact source that produced it.
Document the mapping between Lambda function versions (or aliases) and S3 object version IDs. This makes rollbacks straightforward: update the function to reference the previous object version.

Cross-account considerations

How you organize your artifact buckets across AWS accounts depends on your operational model:

Centralized artifact account: A single bucket in a shared-services or tooling account, with bucket policies granting cross-account s3:GetObject access to Lambda execution roles in workload accounts. This gives your platform team a single inventory of all deployment artifacts with consistent lifecycle, encryption, and access policies.
Per-account buckets: Each workload account owns its own artifact bucket. Requires less effort to set up, but harder to enforce consistent governance across many accounts.

Either pattern works with self-managed storage. Choose based on how your organization balances centralized control against team autonomy.

Cross-Region considerations

With REFERENCE mode, your S3 object is the authoritative copy for your function. Self-managed code storage supports cross-Region function creation within a partition for all default Regions (non-opt-in Regions). You can store your code packages in one Region and deploy your functions in another. This makes cross-Region planning critical. There are four factors to balance:

Disaster recovery

This is the most critical consideration. Because your S3 object is the single source of truth in REFERENCE mode, you do not want all your deployment artifacts in a single Region with no fallback. A recommended pattern:

Primary Region: Your main artifact bucket where CI/CD pipelines deposit new deployment packages.
Fallback Region: A secondary bucket populated via S3 Cross-Region Replication (CRR). If your primary Region becomes unavailable, you can update your Lambda functions to reference the replicated objects in the fallback Region.

Enable S3 Replication Time Control (RTC) if you need a guaranteed SLA (15 minutes) for replication completion.

Cost

Weigh replication + storage costs against per-deploy cross-Region data transfer. If you deploy frequently, storing replicated copies in each target Region is usually cheaper. For infrequent deployments, a one-time transfer may suffice.

Governance and data residency

Some organizations, particularly in regulated industries, have strict requirements about where code artifacts can reside. Before configuring cross-Region replication, confirm that your data is permitted to leave its current Region. Certain regulatory frameworks (for example, data sovereignty laws, FedRAMP boundaries) may restrict replication to specific Region pairs.

Performance

If your workload requires fast function creation and activation times, for example, in a CI/CD pipeline where deployment speed is critical, keep your S3 objects in the same Region where you are creating your Lambda functions. Cross-Region reads add latency to the initial code download, which directly impacts how quickly a new function version becomes active after deployment.

For workloads where creation speed is less critical (for example, batch processing functions that are updated infrequently), the latency of a cross-Region read might be acceptable and can simplify your architecture.

Things to know

Before adopting self-managed S3 buckets for your Lambda functions, keep the following in mind:

Availability: You can use this feature today in all AWS standard regions where Lambda is supported.
Pricing: There is no additional Lambda charge. You pay standard S3 costs for storage, and any cross-Region data transfer.
Maximum deployment package size: The existing limits apply: 250 MB unzipped.
Supported runtimes: All Lambda runtimes that support .zip deployment packages are compatible. Container image deployments are not affected by this feature.
Migration: You can switch an existing function from service-managed to self-managed storage by calling UpdateFunctionCode with the --s3-object-storage-mode REFERENCE parameter. Lambda recreates the function by referencing the object in your S3 bucket and deletes the saved copy.
Reverting: You can switch back to service-managed storage at any time by updating the function with --s3-object-storage-mode COPY. Lambda resumes copying the artifact to its internal bucket.
Object availability is your responsibility: In REFERENCE mode, Lambda depends on your S3 object being accessible. If the object is deleted, the bucket policy changes, or the KMS key is disabled, new invocations requiring a cold start will fail.

Conclusion

In this post, we showed how self-managed S3 buckets for Lambda give you more capacity, more control, and simpler compliance, all without changing how you write or invoke your functions. Your deployment packages no longer count against account quotas, your security team can apply the same policies to code artifacts that they apply everywhere else, and your disaster recovery story is as strong as the replication capabilities of S3.

To get started:

Read the Lambda Developer Guide: Self-managed S3 code storage for full documentation.
Try it in the AWS Lambda Console. Choose Reference Mode under Code storage mode when creating your next function.
Review S3 Lifecycle Configuration examples to plan your artifact retention strategy.
Explore S3 Cross-Region Replication for disaster recovery planning.

New: Enhanced AssetState dimension for AWS Outposts capacity metrics on Amazon CloudWatch

Rachel McElwaine — Wed, 15 Jul 2026 22:52:23 +0000

Today, we are releasing an expanded format of our Amazon CloudWatch dimensions for AWS Outposts capacity metrics. The existing CloudWatch metrics, AvailableInstanceType_Count, UsedInstanceType_Count, InstanceTypeCapacityAvailability, and InstanceTypeCapacityUtilization for Outposts, can now be grouped using the new AssetState dimension with values: Active, Isolated, or Retiring. In this post, we describe what’s changing and how you can use this dimension to improve your capacity monitoring.

What’s changing

Previously, an Outpost could be moved to one of these asset states following an AWS maintenance action, either by an on-site visit or by a remote network update. These state transitions are triggered by control plane operations and previously were not surfaced in customer-facing metrics. This could lead to incorrect or misleading capacity counts.

With this enhancement, you can group the metrics by the dimension to distinguish capacity between production-ready resources and capacity temporarily offline for maintenance.

Introducing the new AssetState dimension

This new dimension adds more visibility into the state of your first-generation and second-generation Outpost racks and servers. The values show the internal state of the AWS Outposts hardware and describe the current working state of the Outpost. These new values are:

ACTIVE – The Outpost is production-ready and can launch instances.
ISOLATED – The server or asset within the Outpost was taken offline and is temporarily unavailable.
RETIRING – The compute asset is not available for use. This state is used when a replacement part is needed.

The new AssetState dimension can be used for creating Amazon CloudWatch alarms for better monitoring, visibility, and alerting of AWS Outposts capacity.

Example metrics output

5 Server/Assets: 4 Active, 1 Isolated, 3 Active used, 1 Isolated used:

AvailableInstanceType_Count | i3en.metal-2tb | count = 1
AvailableInstanceType_Count | i3en.metal-2tb | ACTIVE | count = 1
AvailableInstanceType_Count | i3en.metal-2tb | ISOLATED | count = 0
AvailableInstanceType_Count | i3en.metal-2tb | RETIRING | count = 0

UsedInstanceType_Count | i3en.metal-2tb | count = 4
UsedInstanceType_Count | i3en.metal-2tb | ACTIVE | count = 3
UsedInstanceType_Count | i3en.metal-2tb | ISOLATED | count = 1
UsedInstanceType_Count | i3en.metal-2tb | RETIRING | count = 0

InstanceTypeCapacityAvailability | i3en.metal-2tb | 20%
InstanceTypeCapacityUtilization | i3en.metal-2tb | 80%

You can now accurately monitor your AWS Outposts capacity and set CloudWatch alarms that reflect available capacity with near real-time visibility.

Integration with CloudWatch on Outposts

This enhanced dimension is fully integrated with CloudWatch on Outposts, so you can monitor your local AWS Outposts infrastructure with the same observability tools you use in AWS Regions.

With the new AssetState dimension, you can create more precise CloudWatch alarms on your Outpost that trigger only on capacity status changes (ACTIVE/ISOLATED/RETIRING). This is particularly valuable if you run mission-critical workloads on Outposts and need accurate, real-time visibility into your on-premises capacity.

Availability

These new metrics are enabled by default and available to all AWS Outposts customers at no additional cost in all AWS Regions where AWS Outposts is offered.

Conclusion

The new AssetState dimension gives AWS Outposts customers clear visibility into the different hardware states of Active, Isolated, or Retiring. This visibility helps you maintain accurate capacity counts and create more precise CloudWatch alarms.

To learn more about CloudWatch metrics for AWS Outposts, refer to the Outposts monitoring documentation. For information about CloudWatch on Outposts and local monitoring capabilities, visit the CloudWatch on Outposts documentation.

Introducing modularized kernel cryptography in Amazon Linux

Mahak Arora — Tue, 14 Jul 2026 19:34:12 +0000

We are introducing modularized kernel cryptography in Amazon Linux 2023, an approach that separates Federal Information Processing Standard (FIPS) 140-3 cryptographic components into an independent kernel module that can be certified once and reused across subsequent kernel versions. In this post, we describe how this modular approach works, what it means for FIPS compliance workflows, and how customers can prepare for adoption.

Previously, when any part of the kernel changed, the entire kernel binary had to go through FIPS re-certification because the cryptographic code was embedded within it. With this modular approach, only the standalone cryptographic module undergoes validation, which means non-cryptographic kernel changes no longer require full re-certification. This can help customers who need both security updates and FIPS-validated cryptography while reducing disruption.

FIPS 140-3 validation can be a critical requirement for customers in regulated environments, including federal contractors. Previously, this re-certification process meant customers had to wait 12-18 months for each new kernel version to complete validation before they could adopt it. With the modular approach, once the module is validated it is designed to carry forward across kernel updates, whether minor or major releases, through a streamlined update process rather than repeating the full certification cycle, as long as the module itself remains unchanged. This is particularly relevant as customers face growing pressure to apply security patches rapidly while helping to maintain continuous compliance.

The FIPS re-certification process can be time-intensive with unpredictable timelines given current NIST Cryptographic Module Validation Program (CMVP) processing volumes. To help address this, we isolate all FIPS-scoped cryptographic algorithms, self-tests, and integrity checks into a single loadable kernel module that defines its own FIPS 140-3 cryptographic boundary with a stable interface to the kernel. This reduces what must be re-validated because instead of certifying the entire kernel binary which contains millions of lines of non-cryptographic code, only the standalone module containing the cryptographic implementation falls within the certification scope. For subsequent kernel versions using an unchanged module, re-validation can follow a more streamlined process rather than requiring a full certification cycle, helping our customers adopt kernel updates without the re-certification delays they previously faced, as long as the certified module itself remains unchanged.

We submitted the module for FIPS 140-3 validation. Based on current CMVP processing timelines, validation is expected to complete in 2027. The module interface boundary is designed to remain stable across kernel versions. Changes to the module are required if the kernel internal cryptographic API changes or if new algorithms need to be added to the FIPS scope. In many of these cases, changes can be absorbed by the interface layer without modifying the certified module itself, reducing the need for full re-certification.

Technical overview

The modular capability is included in AL2023 kernel 6.18 and later versions. The module loads automatically at boot with no kernel rebuild or configuration change required. To operate in FIPS mode, follow the enablement guide referenced in the customer guidance section below. This change does not affect other FIPS user-space modules such as OpenSSL, libgcrypt, and NSS.

The following diagram illustrates the architectural shift:

Figure 1. Kernel cryptography architecture before and after modularization.

The implementation spans two areas described below. The kernel build process produces the module as a separate artifact, and a boot-time mechanism loads and connects it to the running kernel.

A restructured kernel build

In the standard kernel build, crypto source code is compiled and statically linked together with other non-crypto components that are not in scope for FIPS to produce the final kernel image. With this change, the build process separates the FIPS-relevant cryptographic components from the kernel image by defining customized compilation rules. Crypto components that are FIPS-related and were previously built into the kernel are now automatically collected and linked separately into a standalone crypto kernel module. The new build process requires no changes to existing build workflows.

Boot-time module plug-in mechanism

Immediately after kernel boot starts, the crypto kernel module is loaded and initialized. Low-level interfaces such as function addresses are connected back to the kernel binary interface so that the module integrates seamlessly with the running kernel. Once loaded, kernel crypto subsystems and their services behave as if they were built in, with the same algorithmic implementations and call paths. This process was designed to not have a material impact on performance. This loading process is independent of FIPS mode configuration because FIPS mode controls how cryptographic algorithms behave at runtime while modularization determines how they are built and delivered within the kernel. To learn more about the design and implementation, see the detailed writeup on LWN.net.

Industry impact and benefits

Once the module completes validation, modularized kernel cryptography can help customers in regulated industries update kernels more frequently while maintaining their FIPS validation status. Customers who previously faced 12-18 month re-certification delays with each kernel version can instead adopt updates as they are released, whether they operate in financial services, healthcare, government, or any sector requiring FIPS-validated cryptography. This can help customers who want to apply critical security patches without a full certification cycle before deployment.

Customer guidance

When evaluating kernel options, customers should consider their specific regulatory requirements, the validation status of cryptographic modules, and their system requirements in accordance with all applicable authorization processes.

Customers who require a completed FIPS 140-3 certificate should continue using AL2023 kernel 6.1, which maintains active validation through 2029-09-22. The modularized crypto module is included in kernel 6.18 and initializes automatically at boot. The module is designed to not require configuration changes and preserves current behavior for non-FIPS workloads. Customers planning FIPS adoption can begin evaluation and testing ahead of formal certification.

Once validation is complete, customers can transition production workloads to kernel 6.18 or later with the validated module by following the FIPS Mode enablement guide for configuration.

Conclusion

To enable FIPS mode on AL2023, refer to our FIPS Mode enablement guide. For regular updates and best practices, follow the AWS Security Blog and FIPS-related FAQs on Amazon Linux 2023. You can also reach out to your AWS account team for help finding the resources you need.

If you have questions about this post, contact AWS Support.

Eliminating Java cold starts with AWS Lambda Managed Instances

Jay Colodner — Mon, 13 Jul 2026 20:23:19 +0000

A single cold start can push your Java Lambda function’s response time from milliseconds to seconds, enough to violate your p99 SLA, timeout a downstream service, and page your on-call. The Java Virtual Machine (JVM) performs best in long-running processes. Its Just-In-Time (JIT) compiler progressively optimizes code over thousands of invocations. Standard serverless execution environments recycle before the JVM reaches peak performance. This creates a tradeoff for latency-sensitive applications between cold-start penalties and runtime optimizations. For production services with p99 service level agreement (SLA) requirements, a single 14-second cold start spike can violate response time guarantees. It triggers downstream timeouts and degrades customer experience.

AWS Lambda Managed Instances changes this equation. As a capability of AWS Lambda, Managed Instances runs your functions on managed Amazon Elastic Compute Cloud (Amazon EC2) instances in your account and maintains JVM persistence across invocations. Connection pools, class hierarchies, and heap state persist across thousands of requests. This allows the JIT C2 compiler to complete optimizations like method inlining, escape analysis, and loop unrolling. The result: 18 to 30% better median latency and 3 to 30x better tail latency compared to Standard Lambda, as the benchmarks in this post demonstrate.

This post benchmarks four Java deployment modes across three workload types using 240,000 requests. The modes compared are Standard Lambda, AWS Lambda SnapStart, GraalVM Native Image, and Lambda Managed Instances. The workload types are CPU-bound, I/O + computation, and I/O-bound. This post presents benchmark results demonstrating Managed Instances delivering 30% better median latency and removing multi-second cold-start spikes on CPU-bound work after JIT warmup. It explains why these gains occur, maps each deployment mode to specific traffic patterns and cold-start tolerance requirements, and provides a decision framework for selecting the right approach for your workload.

Benchmarking setup

The benchmark runs all four deployment modes with identical Spring Boot 4.0.6 applications on Java 25 and AWS SDK v2. This configuration verifies fair comparison across modes. We tested three workloads: UC1 (PDF generation, CPU-bound), UC2 (data aggregation, I/O + computation), and UC3 (API orchestration, I/O-bound). The benchmark sends 240,000 requests using Artillery load testing at 33 RPS. Standard Lambda, SnapStart, and Native Lambda use 1024 MB (1 vCPU). Managed Instances uses c7i.xlarge instances with 2048 MB memory. Concurrency is tuned per workload (UC1=3, UC2=5, UC3=10) based on load testing to avoid thread contention. The benchmark measures p50, p99, and maximum latency across 10 runs of 2,000 requests each, with 5-minute cool-down between runs. The benchmark tracks JIT compilation metrics via Amazon CloudWatch Embedded Metrics Format. You can validate these results against Amazon API Gateway access logs, which confirm a <0.1% error rate. The GitHub repository contains complete source code, AWS Serverless Application Model (AWS SAM) templates, load scripts, and raw data. Performance claims in this post reference data from this benchmark methodology.

Figure 1 presents the architecture for all four deployment modes running in parallel against shared backend services.

To reproduce these benchmarks or deploy the sample applications, refer to the GitHub repository. The repository contains complete SAM templates, Artillery load configurations, deployment instructions, and cleanup commands. This post focuses on benchmark results and analysis. The benchmark used the following tools and services:

An AWS account with permissions to create Lambda functions, Amazon DynamoDB tables, Amazon Simple Storage Service (Amazon S3) buckets, Amazon Simple Notification Service (Amazon SNS) topics, and Amazon Simple Queue Service (Amazon SQS) queues
Java 25 (Amazon Corretto recommended).
Maven 3.9+.
AWS SAM CLI v1.155 or later.
Docker (for GraalVM native image builds) or alternative.
Artillery for load testing.
The GitHub repository with complete source code and SAM templates

Observing max latency

Managed Instances removes the extreme tail spikes characteristic of cold starts. Managed Instances delivers 27x faster maximum latency on CPU-bound workloads (UC1: 489 ms vs. 13,270 ms Standard). Mixed I/O + compute workloads see a 3x improvement (UC2: 3,644 ms vs. 11,174 ms Standard). I/O-bound workloads improve 30x (UC3: 309 ms vs. 9,237 ms Standard). We measured all results using the methodology described in Benchmarking setup.

The Standard Lambda 13-second maximum on UC1 represents a full cold start. That cold start includes JVM boot, Spring context initialization, Amazon DynamoDB client setup, and the first PDF render. SnapStart reduces this to under 3 seconds by restoring from a Firecracker microVM snapshot. However, the restore process plus re-initialization of resources that cannot be checkpointed (network connections, random number generators) still adds latency. GraalVM Native starts in under 2 seconds because the ahead-of-time (AOT) compiled binary skips JVM boot entirely. The Managed Instances maximum of 487 ms is not a cold start; it’s the slowest warm request across 20,000 invocations. For production SLAs, a 14-second cold start spike on Standard Lambda violates most requirements, while Managed Instances removes that spike entirely.

Observing median latency (p50)

Lambda Managed Instances delivered the lowest median latency across all three workloads. Results demonstrate 30% faster median latency on CPU-bound workloads (UC1: 97 ms vs. 139 ms Standard). Mixed I/O + compute achieves a 19% improvement (UC2: 184 ms vs. 228 ms Standard). I/O-bound workloads improve 18% (UC3: 76 ms vs. 93 ms Standard).

The improvement scales with CPU intensity because the JIT C2 compiler on persistent Managed Instances optimizes hot code paths that short-lived serverless environments never reach. On CPU-bound workloads (UC1), the JIT compiler has more opportunity to optimize tight loops in PDF rendering. On I/O-bound workloads (UC3), network latency to Amazon DynamoDB, Amazon SQS, and Amazon SNS dominates the request duration, so JIT optimization provides smaller gains.

Observing tail latency (p99)

Managed Instances showed even larger improvements at the tail of the latency distribution. The p99 improves 36% on CPU-bound workloads (UC1: 225 ms vs. 353 ms Standard). Mixed I/O + compute achieves a 41% improvement (UC2: 1,883 ms vs. 3,201 ms Standard). I/O-bound workloads improve 27% (UC3: 193 ms vs. 265 ms Standard).

UC2 showed the largest p99 improvement (41%) because data aggregation combines DynamoDB queries, returning hundreds of records with in-memory statistical computation and Amazon S3 uploads. Standard Lambda environments that haven’t fully warmed their JIT produce significantly slower responses at the tail. The persistent JIT optimization (-Xms512m -Xmx1408m) with G1 garbage collection (GC) and explicit heap sizing on Managed Instances both contribute to tighter tail latency distribution. For services with SLAs on p99 response time, this reliability improvement matters more than median performance. For workloads with significant heap pressure, tuning -XX:MaxGCPauseMillis and monitoring GC logs can further tighten tail latency.

Why Lambda Managed Instances is faster: JIT compilation

The JVM’s Just-In-Time compiler works in tiers. The C1 compiler performs initial compilation quickly with basic optimizations. The C2 compiler profiles execution over hundreds of invocations and then applies aggressive optimizations: method inlining (eliminating function call overhead), escape analysis (allocating objects on the stack instead of the heap), loop unrolling (reducing branch overhead), and vectorization (processing multiple data elements in a single CPU instruction).

The following table presents JIT warmup progression using java.lang.management.

CompilationMXBean emitted through Amazon CloudWatch Embedded Metrics Format. We collected this data from a 1,500-request sustained load test on UC1 (PDF generation):

Phase	Invocation	Avg Latency	What’s Happening
First requests (application init)	1	~2,400ms	JVM boot, spring context creation, SDK client setup
Early requests (C1 compiled)	2-100	~145ms	C1 compiler active. App is functional, but not optimized
Steady state (C2 optimized)	1000+	~38ms	C2 optimizations completed

The first invocation includes one-time application start costs: class loading, Spring context initialization, and DynamoDB client construction. These costs are unrelated to JIT compilation and occur on any deployment mode.

Once C1 compilation stabilizes during early invocations, latency reaches approximately 145ms. This is the baseline compiled performance. Over the next several hundred invocations, the C2 compiler profiles hot code paths and applies optimizations. By invocation 1,000, latency drops to 38ms. This represents a 3.8x improvement from JIT optimization alone.

Standard Lambda environments typically recycle before C2 completes its optimization passes. On Managed Instances, concurrent requests share the same JVM. This accelerates JIT profiling: three concurrent requests generate three times the method invocation data for the C2 compiler to optimize. The C2 compiler profiles execution patterns across all concurrent requests. It identifies hot code paths faster and applies optimizations sooner than single-concurrency environments.

What this means: CPU-bound workloads see the largest gains (30% faster median latency on UC1) because the JIT compiler has more opportunity to optimize tight loops and method calls. I/O-bound workloads see smaller gains (18% faster on UC3) because network latency to DynamoDB, SQS, and SNS dominates request duration. The JIT compiler still optimizes your code, but the network time remains constant across all deployment modes.

Choosing the right mode

No single mode wins in every scenario. The right choice depends on your traffic pattern, cold-start tolerance, team expertise, and operational complexity budget.

Lambda Managed Instances is ideal for steady-state traffic patterns above 5 requests per second with low cold-start tolerance (p99 SLA under 500 ms). Best for workloads with predictable, sustained traffic that need low latency with zero cold starts. Managed Instances excels at CPU-bound workloads where JIT optimization compounds.

SnapStart works well for variable traffic patterns where cold-start reduction matters. Choose this as the default for Java Lambda functions. SnapStart reduces cold starts with minimal code changes (add CRaC priming). You have no additional infrastructure to manage. Works with the existing Lambda scaling model.

GraalVM Native Image works well for bursty traffic patterns with strict cold-start tolerance (sub-second cold starts required). Ideal if your team can invest in AOT compatibility (reflection configuration, build pipeline). This mode offers a smaller memory footprint. Requires testing for SDK compatibility.

Standard Lambda is the baseline for low-traffic or burst workloads where cold starts of 6-14 seconds are acceptable. Works well when invocation frequency is low enough that per-request billing is cheaper than fixed instance costs, or when operational simplicity is the top priority.

For example, if you run a Spring Boot API handling 100 requests per second with a 400 ms p99 SLA, Lambda Managed Instances reduces your p99 from 353 ms (cutting it close) to 225 ms (comfortable margin) and removes the multi-second cold start spikes that violate your SLA entirely.

Dimension	Standard	SnapStart	Native	Managed Instances
Cold start	6-14 s	2-7 s	800 ms – 2 s	None
Warm p50 (CPU-bound)	139 ms	127 ms	107 ms	97 ms
Tail latency	Worst	Better	Good	Fastest
Error rate	Low	Low	Higher (SDK compat)	Low
Operational complexity	Lowest	Low	High (build pipeline)	Medium (VPC, sizing)
Burst scaling	Fastest	Fastest	Fastest	Slower (capacity provider)
Migration effort	None	Low (add CRaC priming)	High (AOT compat, reflection configuration)	Medium (capacity provider, VPC, thread safety)
Memory efficiency	Good	Good	Lowest (125-154 MB)	Fixed per instance

Lambda Managed Instances supports Graviton4 (arm64) instances, which offer approximately 20% better price-performance based on AWS published Graviton4 benchmarks. These benchmarks use x86_64 for consistency across all four modes (GraalVM native cross-compilation to arm64 adds complexity). The arm64 parallelization characteristics could shift the performance curves for longer-lived deployment modes like Managed Instances in ways worth exploring in a future post.

Cost considerations

Lambda Managed Instances uses instance-based pricing rather than per-invocation billing. For steady-state workloads above approximately 9 requests per second, the fixed instance cost is lower than equivalent Standard Lambda GB-second charges. You can use the official pricing calculator to compare Managed Instances and standard Lambda costs.

Try it with your runtime version

These benchmarks use Java 25 with Spring Boot 4.0.6. The GitHub repository also includes configurations for Java 21 with Spring Boot 3.x. The repository README walks you through deployment, load testing, and collecting your own metrics.

Conclusion

This post demonstrates how Lambda Managed Instances solves a fundamental Java-on-serverless mismatch. The JVM’s JIT compiler needs time to optimize hot code paths. Standard Lambda recycles environments before the JVM reaches peak optimization. Managed Instances keeps the JVM alive across invocations, allowing the C2 compiler to reach peak optimization. The benchmarks show the impact. In these benchmarks, Managed Instances delivered 18 to 30% faster p50 latency than Standard Lambda. Tail latency improved 27 to 41% at p99. Maximum response times dropped 3 to 30x on CPU-bound workloads. The 3.8x improvement from JIT optimization alone shows what’s possible when the runtime has time to complete its work.

For more information, refer to the Lambda Managed Instances documentation. The GitHub repository contains the complete benchmark code, SAM templates, and deployment instructions. Share your results in the comments and let the community know how Managed Instances performs on your workloads. To delete all benchmark resources and avoid ongoing charges, run the cleanup commands documented in the GitHub repository README.

Architecting for IOPS and throughput performance on AWS Outposts racks

Brianna Rosentrater — Fri, 10 Jul 2026 17:10:39 +0000

AWS Outposts extend AWS infrastructure, services, APIs, and tools to on-premises locations for workloads that require low latency, local data processing, or data residency.

In this post, you learn how to configure instances running on an Outpost to support the required IOPS and throughput for your application. The actual IOPS available to an instance is determined by the Amazon Elastic Compute Cloud (Amazon EC2) instance type selected, Amazon Elastic Block Store (Amazon EBS) storage type, and number of volumes available. The lowest performing subsystem limits your overall IOPS and throughput. This post explains each subsystem’s performance impact and provides guidance on sizing Outpost EC2 instances and storage to deliver the target IOPS and throughput values. We focus on EBS-attached volumes rather than instances using EC2 instance store storage.

Performance considerations

When designing for IOPS and high throughput, consider two main drivers with the lower value taking precedence. The first is the performance of the EC2 instances and the second is the supported IOPS and throughput of the attached EBS volumes.

At the time of publishing, Outposts supports first-generation (c5, m5, r5, g4dn) and second-generation (c7i, m7i, r7i, c8i, m8i, r8i) instance families, which are all EBS-optimized instances. These instances provide dedicated bandwidth to the EBS volume I/O, minimizing traffic contention and ensuring optimal storage performance. When attached to an EBS-optimized instance, General Purpose SSD (gp2 and gp3) volumes deliver at least 90 percent of their provisioned IOPS performance 99 percent of the time each year. For detailed instance type specifications and features, see the Amazon EC2 Instance Types Guide.

The starting point for any design is to understand the performance capability of the selected EC2 instance. Looking at Amazon EBS specifications, the columns titled “Baseline / Maximum IOPS” and “Baseline / Maximum throughput” shows the performance of memory optimized R instance family.

¹ These instances can sustain the maximum performance for 30 minutes at least once every 24 hours, after which they revert to their baseline performance.

However, your EC2 instance performance will be constrained if your attached EBS volume(s) have a lower baseline IOPS and throughput.

EBS performance

Outposts racks support two types of EBS storage: General Purpose gp2 only in first-generation Outposts racks, and General Purpose gp2 and gp3 volume types in second-generation Outposts racks. The gp2 storage type supports volumes of between 1 GiB and 16 TiB. Volumes 33.33 GiB and smaller are provisioned with the minimum of 100 IOPS, while volumes larger than 33.33 GiB are provisioned with 3 IOPS per GiB of volume size up to the maximum of 16,000 IOPS, which is reached at 5,334 GiB (3 IOPS X 5,334 GiB). See gp2 volume performance for details on how this is calculated. For gp2 volumes larger than 1,000 GiB, the baseline performance exceeds the burst performance, so burst performance becomes irrelevant. For consistent performance, we also recommend using a volume size of at least 334GiB to deliver a consistent bandwidth of 250 MiB/s, gp2 volumes deliver throughput between 128 MiB/s and 250 MiB/s depending on the volume size, with larger volumes delivering higher throughput up to maximum 250 MiB/s.

On Outposts, the gp3 storage type supports volume sizes up to 16 TiB, IOPS up to 16,000, and throughput up to 1,000 MiB/s. To reach the maximum IOPS provisioned at 500 IOPS per GiB of volume size for gp3, you must use at least a size 32 GiB volume with an EC2 instance that can also support up to 16,000 IOPS. To reach maximum throughput, your volume needs to provide at least 4,000 IOPS, which is achieved with a 8 GiB or larger volume.

Use AWS Identity and Access Management (AWS IAM) policies to control which principals can create, attach, detach, or delete EBS volumes. This is especially important when using multi-volume configurations where data spans across multiple volumes. On Outposts, Amazon EBS encryption is enabled by default — all EBS volumes are automatically encrypted at rest using AWS Key Management Service (AWS KMS) keys with no measurable impact on IOPS or throughput performance. Data is encrypted on the local NVMe storage using AES-256. For additional control, you can use AWS KMS customer-managed keys to manage the encryption keys for your volumes.

However, for the Amazon Relational Database Service (Amazon RDS) database engines supported on Outposts as well as EC2, database and EC2 instance storage is striped across multiple volumes providing several times the baseline throughput and the burst IOPS of a single volume. When small I/O operations are physically sequential, EBS attempts to merge them into a single I/O operation up to the maximum I/O size. Similarly, when I/O operations are larger than the maximum I/O size, EBS attempts to split them into smaller I/O operations. For the best performance, use larger packet sizes up to the maximum I/O size supported.

Calculating IOPS

When calculating the supported IOPS for your workload, start by choosing the instance type that supports your target IOPS, and then size the EBS both in terms of volume size and the number of attached volumes. To achieve maximum EBS performance, the combined IOPS of all attached volumes must meet or exceed the maximum IOPS supported by the instance. When selecting a general purpose EBS volume size, each GiB of EBS adds IOPS up to the maximum supported baseline IOPS (16,000 IOPS for gp2 and gp3). When designing high-performance workloads on Outposts, verify that your gp3 storage configuration is sized to meet the aggregate IOPS requirements of your workload.

As an example, the r7i.12xlarge delivers a maximum of 60,000 IOPS with an EBS throughput of 1,875 MB/s (see Figure 2). To reach this ceiling using gp2 EBS volumes — where each GiB provides 3 IOPS up to a maximum of 16,000 IOPS per volume — you would need to attach:

Three volumes of at least 5,334 GiB each (delivering 16,000 IOPS per volume = 48,000 IOPS combined)
One volume of at least 4,000 GiB (delivering the remaining ~12,000 IOPS)

This brings the total provisioned IOPS to 60,000, matching the instance’s maximum IOPS. By contrast, gp3 EBS volumes support a baseline IOPS performance of 3,000 included in the price of the storage, with the ability to provision additional IOPS up to the maximum supported 16,000 per gp3 volume on Outposts racks. IOPS are provisioned at a rate of 500 IOPS per GiB of volume size, so the maximum can be reached by provisioning a 32 GiB or larger volume as opposed to the 5,334 GiB volume required to get 16,000 IOPS with gp2. That means to deliver 60,000 IOPS of performance using an r7i.12xlarge instance with gp3 EBS storage, you would need to attach:

Three volumes of at least 32 GiB each (delivering 16,000 IOPS per volume = 48,000),
One volume of at least 24 GiB (delivering remaining ~12,000 IOPS).

This means only 56 GiB of gp3 EBS storage is required to meet performance requirements, instead of 20,002 GiB of gp2 EBS storage. For maximum performance, make sure the provisioned EBS volume(s) IOPS matches the bandwidth ceiling of your instance type. If you’re using EBS RAID configurations, note that arrays larger than 8 volumes often yield diminishing performance returns because of increased I/O overhead.

Calculating throughput

Throughput is equally important to IOPS for a performant architecture. Throughput measures the volume of read/write operations that can be processed per second. On Outposts, up to 1,000 MiB/s throughput per volume can be achieved using EBS gp3 storage, and EBS gp2 can provide up to 250 MiB/s throughput per volume. While gp2 and gp3 EBS storage on Outposts both provide up to 16,000 IOPS per volume, gp3 can provide up to 4x as much throughput, making it a better choice for high performance databases on second-generation Outposts racks.

Like IOPS, while gp2 throughput scales based on volume size, you can provision additional throughput for gp3 EBS volumes. EBS gp3 storage delivers a consistent baseline throughput performance of 125 MiB/s. You can provision additional throughput up to the 1,000 MiB/s maximum supported on second-generation Outposts racks at a ratio of 0.25 MiB/s per provisioned IOPS, which can be reached using an 8 GiB gp3 volume. To get the maximum supported IOPS and throughput performance using gp3 EBS storage with Outposts, use at least a 32 GiB volume. When designing high-performance workloads on Outposts, verify that your gp3 storage configuration is sized to meet the aggregate throughput requirements of your workload.

Refer to the earlier section on performance considerations to confirm your selected EC2 instance can provide as much throughput as your EBS storage volume(s) to avoid performance bottlenecks.

RDS IOPS considerations

At the time of publishing, Outposts racks support the RDS for SQL Server, RDS for MySQL, RDS for Oracle, and RDS for PostgreSQL database engines. Database instance performance varies depending on the EC2 instance type selected for the database, the EBS volume type selected for RDS database storage, the database engine selected, and the size of the RDS database storage. The following tables show expected IOPS for database instances using gp2 and gp3 EBS volume types respectively, as shown in the General Purpose SSD Storage section of the Amazon RDS user guide.

To calculate your database instance performance, consider all influencing factors. For example, if you wanted to support the maximum IOPS of 16,000 shown for the SQL Server RDS database engine, you would need:

32 GiB gp3 volume, OR
5,334 GiB gp2 volume.
At least an r5.4xlarge, which can provide a baseline 18,750 IOPS. If using a second-generation Outposts rack, r7i.4xlarge and r8i.4xlarge instances provide a baseline of 20,000 IOPS. However, they are constrained by the lowest performing subsystem, which would be the amount of I/O the database engine can support (16,000 for SQL Server).

The storage type and size have the biggest impact on IOPS performance. For high I/O databases, we recommend either purchasing additional gp2 storage for your first-generation Outpost rack (understanding you might need to provision more storage than needed to meet your IOPS requirements), or using second-generation Outposts racks which support the more performant gp3 EBS storage type for your database workloads. Also consider that Outposts racks have a fixed storage capacity, and aggregate workload IOPS should be reviewed.

Monitoring IOPS

To check that the infrastructure is sized correctly to meet your IOPS expectations, use Amazon CloudWatch EBS volume metrics. You can monitor your EBS volume performance and set CloudWatch Alarms if the values exceed, for example, 70% of the total. Metrics such as VolumeReadBytes and VolumeWriteBytes provide information on the read and write operations in a specified time period based on bytes, and likewise VolumeReadOps and VolumeWriteOps provide the same information based on completed operations. You can monitor the time taken for read and write operations for an Amazon EBS volume using the VolumeTotalReadTime and VolumeTotalWriteTime metrics respectively using the Average statistic. Use IAM policies to restrict who can view, create, or modify CloudWatch alarms and dashboards for your Outpost resources. This prevents unauthorized users from suppressing critical storage performance alerts.

You can also use the Latency Injection action in AWS Fault Injection Service to run controlled experiments to test your architecture and monitoring based on this metric to improve your resiliency to storage performance degradation. You can access real-time detailed performance statistics for Amazon EBS volumes that are attached to Nitro-based Amazon EC2 instances. You can combine these statistics to derive average latency and IOPS, or to check whether I/O operations are completing. You can also view the total amount of time that your application has exceeded your EBS volume’s or the attached instance’s provisioned IOPS or throughput limits. By tracking increases in these statistics over time, you can identify whether you need to increase your provisioned IOPS or throughput limits to optimize your application’s performance. The detailed performance statistics also include histograms for read and write I/O operations, which provide a distribution of your I/O latency by keeping track of the total number of I/O operations completed within a latency band. See Monitor your Outposts rack and Monitoring best practices for AWS Outposts for general Outposts monitoring guidance.

When running FIS experiments, follow the principle of least privilege by scoping IAM roles to specific resources, and always configure stop conditions to automatically halt experiments that exceed expected impact thresholds.

Conclusion

This post explains how to size EC2 instances and EBS storage on Outposts racks to meet your IOPS and throughput requirements, helping you avoid performance bottlenecks for database and application workloads. You can monitor EBS storage performance through CloudWatch and create alarms to alert you when your instance approaches its IOPS threshold. To learn more about Outposts and how to architect for IOPS and throughput performance for your workloads, reach out to your AWS account team, or visit the AWS Outposts contact page.

Announcing Lambda MicroVMs: serverless compute environments with VM-level isolation and near-instant startup

Ayush Kulkarni — Fri, 10 Jul 2026 16:03:55 +0000

We recently announced the launch of AWS Lambda MicroVMs, a new serverless compute primitive that provides VM-level isolation, near-instant startup performance, and state retention. You can now give each user or job their own execution environment to securely run just-in-time code – either user or AI generated – without managing virtualization infrastructure or choosing between isolation, speed, and state retention. Lambda MicroVMs are powered by Firecracker virtualization, the technology underpinning AWS Lambda. You can use Lambda MicroVMs to build data analytics applications, AI sandboxes, vulnerability scanners, and interactive development environments.

Evolution of serverless compute

When we launched AWS Lambda in 2014, the premise was simple: give developers a way to run code without thinking about servers. Upload a handler, configure a trigger, and let the service handle infrastructure provisioning, scaling, patching, and availability. Over the past decade, Lambda has grown to process tens of trillions of requests each month for over 1.5 million customers. Under the hood, those invocations run inside a Lambda-managed Firecracker microVM – a lightweight virtual machine that combines hardware-level virtualization and near-instant startup speed. With Lambda SnapStart, we used Firecracker’s snapshotting capabilities to accelerate startup times by resuming execution environments from pre-initialized snapshots (carrying memory and disk state) rather than cold-booting them.

Today, a growing class of applications need to run code supplied by users or AI agents just-in-time – and need Firecracker’s core capabilities directly: hardware isolation, near-instant startup, and state retention over extended periods. Achieving this today often requires building custom infrastructure that diverts teams from core application development. We’ve been hearing this theme from customers across use cases and industry verticals:

Interactive code environments like browser-based IDEs, notebooks, and vibe-coding platforms need to deploy and execute user-generated code in per-user environments that start within seconds and retain state – like installed packages, generated files, and running processes – across interactions.
Data analytics platforms run user-supplied or LLM-generated queries and notebooks in isolated environments that retain large working sets over long durations – such as an 8-hour workday – with the ability to resume quickly after periods of inactivity.
AI coding assistants and agents run LLM-generated code iteratively, while retaining context between iterations, and rapidly launching and shutting down environments to evaluate alternative code paths, such as for reinforcement learning.
IT security scanners execute vulnerability assessments in compute environments that are strongly isolated from one another, can scale to handle bursts of concurrent scan requests, and support elevated operating system privileges.
CI/CD platforms need ephemeral, isolated build and test environments that start quickly and can be discarded after each run.

Introducing Lambda MicroVMs

Now, with AWS Lambda MicroVMs, developers can directly use the isolation, speed, and state snapshotting of Firecracker MicroVM as a primitive, while keeping the serverless simplicity of AWS Lambda. Lambda MicroVMs provide these key capabilities.

Snapshot-based, near-instant startup: To optimize startup speed, MicroVMs are launched from MicroVM images, which are pre-initialized Firecracker snapshots of your application’s memory and disk state. When you create a MicroVM image, the service executes your Dockerfile, initializes your application, and snapshots the MicroVM. Lambda starts MicroVMs from this snapshot with your dependencies loaded.
Direct HTTPS connectivity: Each MicroVM exposes a dedicated HTTPS endpoint for inbound connectivity to individual ports. You can connect to applications running within your MicroVM using standard HTTPS clients, WebSocket connections, or gRPC – exactly as you would with a container or VM.
Lifecycle control with state retention: Lambda MicroVMs allow you to control the lifecycle of each execution environment, enabling you to support interactions that last a few minutes to sessions that span 8 hours.
Vertical and horizontal scaling: Each MicroVM starts with a configurable baseline — 2 GB memory and 1 vCPU by default (up to 8 GB and 4 vCPUs), with CPU allocated in a 2:1 ratio to memory. From that baseline, MicroVMs scale vertically by up to 4x automatically, meeting peak resource demands for each user or session without any action on your part. MicroVMs also scale horizontally — you can launch several hundred within a minute during traffic spikes. For details on service limits, refer to Lambda service quotas.
Internet and VPC access: By default, Lambda MicroVMs support outbound connectivity to the public internet without additional configuration. For private VPC connectivity to resources such as databases or internal APIs, you can use a Lambda Network Connector (LNC). LNC is a new Lambda resource that provides managed, configurable network connectivity between your MicroVMs and your private VPC.

Building with Lambda MicroVMs

Lambda MicroVMs introduces two core resource types: a MicroVM image – a versioned artifact containing your runtime environment and application code, and MicroVMs – individual instances launched on demand from a MicroVM image.

Let’s make this concrete with an example. You are a cloud architect building a data analytics application which under the hood, manages compute environments to generate insights for data analysts within your organization. Analysts load multi-gigabyte datasets and generate visualizations in sessions that last hours with idle gaps when they switch to other tasks. When analysts return, they expect to pick up exactly where they left off. Here’s how you can use MicroVMs for this workload:

Step 1: Define your environment

Write a Dockerfile that installs your data science stack. This runs once at MicroVM image build time – every analyst’s MicroVM starts with these dependencies already loaded. This Dockerfile builds a notebook server that accepts code execution requests, runs them in-process (so state accumulates across requests), and returns results. Your customer-facing UI calls this notebook server for each analyst.

FROM public.ecr.aws/lambda/microvms:al2023-minimal

# Install Python 3.12 and pip
RUN dnf install -y python3.12 python3.12-pip && dnf clean all

RUN pip3.12 install --no-cache-dir \
    pandas numpy scipy scikit-learn matplotlib seaborn \
    fastapi uvicorn boto3 pyarrow

COPY notebook_server.py /app/notebook_server.py
WORKDIR /app
EXPOSE 8080

CMD ["python3.12", "notebook_server.py"]

Next, package and upload your application artifacts and Dockerfile to S3.

zip -r notebook-env.zip Dockerfile notebook_server.py
aws s3 cp notebook-env.zip s3://amzn-s3-demo-analytics-platform/artifacts/notebook-env.zip --region us-east-1

With these in place, create a MicroVM image:

aws lambda-microvms create-microvm-image \
    --name analytics-notebook \
    --code-artifact '{"uri": "s3://amzn-s3-demo-analytics-platform/artifacts/notebook-env.zip"}' \
    --base-image-arn arn:aws:lambda:us-east-1:aws:microvm-image:al2023-1 \
    --build-role-arn arn:aws:iam::123456789012:role/NotebookBuildRole \
    --resources '[{"minimumMemoryInMiB": 4096}]' \
    --region us-east-1

When you create a MicroVM image, Lambda executes your Dockerfile, starts your application, and takes a Firecracker snapshot of the fully initialized environment with the libraries imported, and notebook server listening. Every MicroVM launched from this image skips this initialization step, and provides near-instant startup.

Step 2: Launch a MicroVM when an analyst starts their session

Once your MicroVM image is ready, you can start a new MicroVM for each analyst session. The idle policy encodes your business logic: auto-suspend after 5 minutes of inactivity, retain the suspended state for up to 8 hours (covers a full workday), and auto-resume when the analyst’s next request arrives. Within seconds, the analyst has a dedicated environment with their own filesystem, and a dedicated HTTPS endpoint.

aws lambda-microvms run-microvm \
    --image-identifier arn:aws:lambda:us-east-1:123456789012:microvm-image:analytics-notebook \
    --image-version 1.0 \
    --idle-policy '{"maxIdleDurationSeconds":300,"suspendedDurationSeconds":28800,"autoResumeEnabled":true}' \
    --maximum-duration-in-seconds 28800 \
    --execution-role-arn arn:aws:iam::123456789012:role/notebook-exec-role \
    --region us-east-1

# MicroVM endpoint url is returned by the run-microvm API call
ENDPOINT="https://a1b2c3d4-e5f6-7890-abcd-1234567890ef.lambda-microvm.us-east-1.on.aws"

When a data analyst submits a query, it is submitted as an HTTPS request to their assigned MicroVM. You can test this using curl on the MicroVM endpoint.

curl -X POST "$ENDPOINT/execute" \
    -H "X-aws-proxy-auth: $TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"code": "import pandas as pd; df = pd.read_parquet(\"s3://amzn-s3-demo-data-lake/transactions.parquet\"); print(f\"Loaded {len(df)} rows, {df.memory_usage(deep=True).sum()/1e9:.1f} GB\")"}'

Notice that you can separate the build-time IAM role from the execution-time IAM role for finer-grained control over each tenant’s permissions.

Step 3: Suspend and resume during idle periods

After 5 minutes of inactivity, the MicroVM is automatically suspended based on the configured idle policy. When the MicroVM is suspended, its memory and disk state is preserved.

Two hours later, the analyst returns and sends the next query. The MicroVM auto-resumes within seconds. The memory and disk state are restored exactly as the analyst left them – no re-computation or re-loading required.

Analysts can also suspend and resume their MicroVMs directly using the APIs.

aws lambda-microvms suspend-microvm \
    --microvm-identifier microvm-a1b2c3d4-e5f6-7890-abcd-1234567890ef \
    --region us-east-1

aws lambda-microvms resume-microvm \
    --microvm-identifier microvm-a1b2c3d4-e5f6-7890-abcd-1234567890ef \
    --region us-east-1

Step 4: Connect to private data sources

If your data lives in a private VPC, for example, Amazon Redshift clusters or RDS databases, you can use a Lambda Network Connector (LNC) to give your analysts MicroVMs access to this data. Create a network connector once:

aws lambda-core create-network-connector \
    --name analytics-vpc \
    --configuration '{"VpcEgressConfiguration":{"SubnetIds":["subnet-data1","subnet-data2"],"SecurityGroupIds":["sg-analytics"],"NetworkProtocol":"IPv4","AssociatedComputeResourceTypes":["MicroVm"]}}' \
    --operator-role arn:aws:iam::123456789012:role/ConnectorRole \
    --region us-east-1

Then, reference it when starting a MicroVM. Re-use network connectors across all MicroVMs that share the same network configuration.

aws lambda-microvms run-microvm \
    --image-identifier arn:aws:lambda:us-east-1:123456789012:microvm-image:analytics-notebook \
    --egress-network-connectors '["arn:aws:lambda:us-east-1:123456789012:network-connector:analytics-vpc"]' \
    --idle-policy '{"maxIdleDurationSeconds":300,"suspendedDurationSeconds":28800,"autoResumeEnabled":true}' \
    --region us-east-1

Now, your organization’s analysts can query private databases directly from their notebook environment.

Step 5: Cleaning up

To stop incurring charges, terminate any running MicroVMs and delete unused resources.

# Terminate the MicroVM
aws lambda-microvms terminate-microvm \
    --microvm-identifier microvm-a1b2c3d4-e5f6-7890-abcd-1234567890ef \
    --region us-east-1

# Delete the network connector (if created)
aws lambda-core delete-network-connector \
    --identifier analytics-vpc \
    --region us-east-1

# Delete the MicroVM image
aws lambda-microvms delete-microvm-image \
    --image-identifier arn:aws:lambda:us-east-1:123456789012:microvm-image:analytics-notebook \
    --region us-east-1

To recap, with MicroVMs we build an image once, launch isolated MicroVMs per user or job, interact over HTTPS, suspend when idle, and terminate when done. This pattern applies broadly, across use cases. For instance, an IT security platform scanning customer repositories has similar requirements: an isolated environment per scan, the ability to run with elevated operating system privileges, and rapid horizontal scaling to hundreds of concurrent scans. Similarly, an AI coding assistant needs per-developer sandboxes that retain installed packages and generated files across iterative code-write-test cycles, with suspend/resume preserving context when developers switch tasks. In each case, the workflow is the same.

Building MicroVMs with Agent Toolkit for AWS

In the previous section, we demonstrated Lambda MicroVMs core API operations. You can also use your preferred Agentic development tools to start developing with Lambda MicroVMs. Simply install the AWS Lambda MicroVMs skill from the Lambda MicroVMs console, or use the Agent Toolkit for AWS.

To get started in the AWS Lambda console, choose the highlighted button to access the MicroVMs agent instructions as in Figure 1:

Figure 1: Access MicroVM agent instructions

Next, copy the agent installation instructions and paste it in your terminal to begin developing.

Figure 2: Copy agent instructions

The following screenshot demonstrates the skill in action in an AI coding assistant. Using the skill, the coding assistant agent generates a detailed plan to build the analytics notebook solution, executes the plan, and validates correct execution.

Figure 3: Agent-driven development with MicroVMs

Lambda MicroVMs as sandboxes for Claude Managed Agents

You can also use AWS Lambda MicroVMs as a managed sandbox provider for Claude Managed Agents self-hosted sandboxes. This pattern keeps the orchestration within your Anthropic environment, which hosts the agent loop and Claude model, but moves tool execution into AWS Lambda MicroVMs, so the agent’s code, filesystem, and network egress never leave the infrastructure you control. You control the execution environment – what is installed, what network access is available, and what resources the agent can reach. For integration details, refer to the Lambda MicroVMs developer guide.

Snapshot compatibility considerations

Lambda MicroVMs are started from snapshots of pre-initialized memory and disk state. This has a few implications for how you build applications:

Uniqueness: Content generated and retained within a MicroVM image is shared across all MicroVMs started from that image. To maintain uniqueness for content such as unique IDs, secrets, or random seeds, generate these values after each MicroVM is started. If your application code uses OpenSSL, use the AWS-provided base container image from public.ecr.aws/lambda/microvms:al2023-minimal to build your MicroVM image.

Ephemeral credentials and network connections: Credentials and connections established during MicroVM image creation – or before a MicroVM is suspended – may expire or terminate by the time the MicroVM starts or resumes. Design your application to refresh these credentials and re-establish connections on startup. AWS SDK clients re-establish connections automatically in most cases.

Lambda MicroVMs provides lifecycle hooks that are executed when a MicroVM is started or resumed. Use these hooks to restore uniqueness and to re-establish network connections or ephemeral credentials. For more details, refer to the Working with snapshots section in the Lambda MicroVMs developer guide.

Pricing

Lambda MicroVMs pricing comprises compute, snapshots, and data transfer (at standard AWS rates, including data transferred to your VPC). You have two cost management levers: baseline-plus-consumption billing and idle-suspension. With baseline-plus-consumption billing, your bill tracks closer to your actual resource usage rather than peak resource usage. You configure your MicroVM’s baseline resource allocation to match your workload’s average resource utilization – not peak. During peak activity, your MicroVM can vertically scale up to 4x of the configured baseline automatically and resource usage above the baseline is only billed during active use. You configure the baseline by setting memory, and CPU is allocated in a 2:1 memory-to-CPU ratio – the default is 2GB / 1vCPU, with a corresponding peak of 8 GB / 4 vCPU. Supported baseline and peak values are shown in Figure 4.

Figure 4: Baseline and peak resource configuration

During extended idle periods, you can suspend your MicroVM to preserve memory and disk state at storage-only rates, resuming near-instantly when needed – no compute charges while suspended. For full pricing details, see AWS Lambda pricing.

Conclusion

Lambda MicroVMs extends the serverless compute model beyond invocation-based functions to long-running, stateful environments that execute code supplied by end users or AI. Development teams can focus on core application development while Lambda provides secure isolation and near-instant startup performance. Whether you’re building an AI coding assistant, an interactive development platform, an IT security platform, or a data analytics workload, the pattern is the same: define your environment in a Dockerfile, build a MicroVM Image once, launch isolated MicroVMs on demand, interact over HTTPS, and terminate when done.

To get started, visit the AWS Lambda MicroVMs developer guide or start building with the MicroVMs agent skill, available through the AWS Lambda console.

Secure code execution for AI agents with AWS Lambda MicroVMs

Shridhar Pandey — Fri, 10 Jul 2026 14:12:30 +0000

Development teams building serverless applications with AI coding agents face the question of how to let those agents generate and execute code without losing control over governance. Agent-generated code needs a secure environment to execute, isolated from production systems and the developer’s local environment. Addressing this requires three things working together: a secure execution sandbox, domain expertise to build correctly, and governance over what agents are allowed to do. This post shows how you can use AWS Lambda MicroVMs, the Agent Toolkit for AWS, and Policy in Amazon Bedrock AgentCore to let AI coding agents build, test, and deploy serverless applications safely with granular governance controls.

Overview

AI coding agents like Claude Code, Kiro, and Cursor accelerate serverless development by generating code, installing dependencies, running tests, and deploying infrastructure on behalf of developers. But today, most of that work executes with whatever permissions and access the developer has. If the agent acts outside its intended scope, whether by mistake or through prompt manipulation, there is no boundary between the agent’s actions and the rest of the environment.

Moving from proof-of-concept (PoC) to production requires isolating agent-generated code into a contained environment where it can execute freely without affecting the host environment or other tenants. It requires embedded domain expertise so agents produce production-grade output rather than improvising from general training data. And it requires deterministic governance that controls what agents are allowed to do regardless of how they are prompted.

Each of these requirements maps to a specific layer in the stack. Lambda MicroVMs provide an isolated, ephemeral compute environment where agents write, build, test, and run code. The Agent Toolkit for AWS provides validated procedures and best practices that guide agents toward production-quality output. Policy in AgentCore enforces deterministic authorization over agent-to-tool interactions at the boundary.

Each layer solves a problem the other two cannot. Without expertise embedded in the workflow, agents running in isolation still produce code that fails in production. Without governance, even well-guided agents can overstep their boundaries. And without execution isolation, governance policies can be circumvented at the runtime level. The three layers work as a unit.

Figure 1 Three-layer stack for secure code execution for AI agents

Layer 1: Execution (Lambda MicroVMs)

Code generated by AI agents needs a secure environment to execute, isolated from production systems, other tenants, and the host environment. Lambda MicroVMs provide a Firecracker-based compute environment with its own kernel, its own filesystem, and its own network namespace. This is the same isolation foundation that has powered Lambda since 2018, now available as a standalone compute substrate. Inside a MicroVM, agents can perform the same operations a developer would on their local machine, such as installing packages, running shell commands, executing build toolchains, and running tests. The difference lies in containment. If the agent generates destructive code, whether through hallucination or prompt injection, the impact is limited to a single ephemeral environment.

Each MicroVM provides operating system access with configurable vCPU, memory, and disk. Agents can run user sessions for up to 8 hours, with configurable network access (public or virtual private cloud (VPC)-only). MicroVMs can be suspended and resumed with their state preserved, giving agents state retention across sessions without sacrificing isolation between tenants.

Layer 2: Expertise (Agent Toolkit for AWS)

Execution isolation alone is not enough. An agent that runs in a MicroVM but improvises from general training data is unlikely to produce production-grade output. For example, it might generate Lambda functions with overly broad IAM permissions, skip observability configuration, or deploy without safe rollback patterns. The Agent Toolkit for AWS gives coding agents validated, up-to-date procedures for AWS tasks. Instead of improvising, agents using the Agent Toolkit follow curated skills that encode how an experienced engineer actually builds on serverless. The toolkit encodes least-privilege IAM by default, observability wired in from the start, and deployment patterns that reflect production best practices.

For Claude Code and Cursor, the Agent Plugin for AWS Serverless packages these skills as a plugin. In Kiro and other tools that support agent skills, they are available directly. These skills dynamically load relevant guidance throughout the development lifecycle, from project initialization through deployment and troubleshooting. This includes a dedicated Lambda MicroVMs skill that gives agents the procedures to provision, configure, and use MicroVM environments directly.

Layer 3: Governance (Policy in AgentCore)

Expertise without governance can produce correct code with no boundaries on what actions the agent can perform. For example, an agent following best practices can still deploy to production, overwrite existing infrastructure, or access data outside its scope. Policy in AgentCore intercepts every tool call at the Amazon Bedrock AgentCore Gateway and evaluates it against Cedar policies before allowing execution. Cedar is an open-source authorization language purpose-built for fine-grained permissions. Its policies are human-readable, analyzable by machines, and evaluate deterministically regardless of how the agent was prompted. The gateway exposes the available tools to the agent. Cedar can inspect tool input parameters, the identity of the user the agent is acting on behalf of, and the specific tool being invoked. A policy can permit an agent to call a deploy tool but deny it when the environment parameter is production.

The enforcement operates entirely outside the agent’s reasoning loop, so policy decisions are not influenced by the model’s context or prompt. Actions that would always be denied are omitted from the agent’s tool list entirely, so the agent never even attempts them. A log-only mode supports incremental rollout, and every enforcement decision is logged to Amazon CloudWatch for audit.

The agentic serverless stack in action

The following walkthrough shows an AI coding agent building and deploying an order processing API using the three layers working together. The same approach applies to any serverless workload, whether it is an event pipeline, a data transform, or a webhook handler. The developer prompts the agent to build the API. The agent uses the Lambda MicroVMs skill to provision its execution environment, then works autonomously within it. It follows Agent Toolkit skills for production best practices, and invokes deployment tools through the AgentCore Gateway under a Cedar policy that controls what it is allowed to do.

Figure 2 End-to-end workflow from developer prompt to governed deployment

Step 1: Write and test inside the MicroVM. The agent starts inside a MicroVM. It scaffolds the application, installs dependencies, and runs the test suite until all tests pass. The agent’s actions are contained to the MicroVM, with no impact to the host environment or any other tenant.

Figure 3 Agent executing the test suite inside a Lambda MicroVM

Step 2: Scaffold with toolkit skills. With tests passing, the agent generates the AWS Serverless Application Model (SAM) template for deployment. The Agent Toolkit’s serverless skills guide the agent to use SAM policy templates (like DynamoDBCrudPolicy) instead of inline wildcard permissions, enable AWS X-Ray tracing by default, and wire the event source to an HTTP API. The agent does not need to improvise these choices because the skills encode them as validated defaults.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  ProcessOrderFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: src/handler.processOrder
      Runtime: nodejs24.x
      Timeout: 30
      Tracing: Active
      Events:
        Api:
          Type: HttpApi
          Properties:
            Path: /orders
            Method: POST
      Policies:
        - DynamoDBCrudPolicy:
            TableName: !Ref OrdersTable

Figure 4 SAM template generated using Agent Toolkit serverless skills

Step 3: Deploy through the governed gateway. The agent has built and tested the application inside its MicroVM. To deploy, it invokes a deployment tool through the AgentCore Gateway. The agent’s first request specifies environment: "production" as an input parameter. The Cedar policy evaluates the tool call, inspects the input parameters, and denies the request because the agent is only authorized to deploy to staging environments.

permit(
    principal,
    action == AgentCore::Action::"DeployTarget___deploy_application",
    resource == AgentCore::Gateway::"<gateway-arn>"
) when {
    context.input.environment == "staging"
};

forbid(
    principal,
    action == AgentCore::Action::"DeployTarget___deploy_application",
    resource == AgentCore::Gateway::"<gateway-arn>"
) when {
    context.input.environment == "production"
};

The agent receives the denial, adjusts, and re-invokes the deployment tool with environment: "staging". The policy permits this request, and the deployment succeeds. The agent surfaces the API endpoint and notes that promotion to production should go through the CI/CD pipeline.

Figure 5 Policy in AgentCore denying production and permitting staging deployment

The Cedar policy did not require changes to the agent’s code or prompting. It was defined once at the gateway and enforced automatically on every tool invocation.

Best practices and considerations

To successfully implement this three-layer architecture, align the configuration of each layer to the security and operational requirements of your workload. Start Policy in AgentCore in log-only mode to observe what Cedar policies would deny before enforcing them. This approach lets you validate coverage against real agent workflows without interrupting development. Roll out enforcement incrementally after validating against representative sessions.

Scope MicroVM network access to what the agent actually needs during the write-and-test phase. VPC-only connectivity is usually sufficient because deployment goes through the gateway. Route all agent tool access through the AgentCore Gateway. Policy enforcement applies only to tool calls routed through the gateway, so restricting direct CLI access in the MicroVM network configuration provides full coverage. Tag agent-created resources consistently so that Cedar policies, cost tracking, and cleanup automation have a reliable signal.

Treat Cedar policies as code. Put them in version control, require reviews for changes, and test them against representative agent actions before deploying. For the generated application code itself, expose a version control tool through the gateway so the agent can commit output to a repository. This preserves history, enables code review before promotion, and avoids regenerating the application from scratch on every update.

Conclusion

This post introduced a three-layer architecture for secure code execution by AI coding agents on AWS serverless. Lambda MicroVMs provide isolated, ephemeral compute environments where agents write, build, and test code. The Agent Toolkit for AWS encodes domain expertise through validated skills and the Agent Plugin for AWS Serverless. Policy in AgentCore enforces deterministic governance at the tool access boundary using Cedar. Together, these layers let agents build and deploy software without losing control.

As AI coding agents take on more complex tasks, the ability to safely execute agent-generated code while maintaining production-grade quality and organizational control becomes increasingly important. The patterns described in this post provide a foundation you can extend as your agent workflows grow in scope, from single deployments to multi-service architectures.

To learn more, visit the Lambda MicroVMs developer guide. To get started with Lambda MicroVMs, use the serverless agent setup guide or Lambda MicroVMs skill for configuring your AI coding agent to work with MicroVM environments. Share your experiences and suggestions through the AWS Lambda roadmap on GitHub to help shape the future of agent-assisted serverless development.

Accelerate multiplayer game hosting with AWS m8azn instances

Spencer Myers — Wed, 08 Jul 2026 17:39:32 +0000

Online multiplayer gaming continues to grow, with players demanding lower latency, higher concurrency, and more immersive experiences than ever before. For game studios hosting dedicated multiplayer servers on AWS, infrastructure decisions directly impact player experience and retention, server tick rates, and ultimately, revenue.

Games are becoming more computationally demanding while offering richer gameplay experiences. Studios need instances that maintain consistent player experiences in increasingly complex and dense computational game experiences.

In this post, we explore how AWS m8azn instances powered by AMD’s 5th gen EPYC processors perform with a real game: Mob Rush. M8azn instances offer up to 2x compute performance and 5 GHz CPU frequency compared to previous generation M5zn instances, and up to 24% higher performance than M8a instances. M8azn instances deliver up to 4.3x higher memory bandwidth and 10x larger L3 cache compared to M5zn instances allowing latency-sensitive and compute-intensive workloads to achieve results faster. These instances also offer up to 2x networking throughput and up to 3x EBS throughput versus M5zn instances. This post walks through the deployment of the game and reviews the performance metrics across varying player counts.

When to choose m8azn for multiplayer hosting

Not every workload requires m8azn. Game modes that require high performance and low latency computation are ideal matches for m8azn instances. M8azn instances are ideal for games that benefit from higher compute performance, larger L3 cache, and higher memory bandwidth.

Ideal use cases

Session-based high density multiplayer games: Games with discrete match sessions that spin up and tear down servers dynamically benefit from the fast startup performance of m8azn and high player density per instance.
Physics-intensive game servers: Titles relying heavily on PhysX collision detection, rigid-body simulation, and real-time raycast operations see significant gains from improved FPU throughput.
Variable player load scenarios: Live service games with daily peak hours or seasonal events benefit from the cost efficiency at both low and high utilization levels.
High-density hosting: Studios seeking to maximize concurrent game sessions per instance to reduce per-player infrastructure cost.

Mob Rush

In this post, we run the game Mob Rush. Mob Rush is a multiplayer game where players collect and grow a crowd in a war of numbers style competition. Mob Rush is built with the Unity game engine. Our test infrastructure includes a local test orchestrator, two game servers, and a series of load generation servers. The following diagram shows our testing setup:

Load testing configuration

Our load testing scenario compares the m5zn instance that our game currently runs on with the new m8azn to decide if the new instance is a migration candidate. We compare player experience metrics (FPS, tick-rate, and others) and instance performance metrics (CPU utilization, tick-rate, players per server, and others) to see how well Mob Rush runs on newer hardware.

Methodology

Benchmarks were conducted using a Unity multiplayer game server build, simulating concurrent player connections with synthetic load generation. Our current game servers perform well to around 4,000 simultaneous player connections before the player experience started to degrade as the server was overloaded. For our benchmarks we have tested each instance type at the 3,000 player threshold, and then increased to 4,000 players, 6,000 players and 8,000 players and recorded how each instance performed.

Parameter	Configuration
Instance Types	m8azn.xlarge (4 vCPU, 16 GB) vs. m5zn.xlarge (4 vCPU, 16GB)
OS / AMI	Ubuntu 22.04 LTS
Unity Version	Unity 2020.3.12f1, headless Linux build
Concurrency Scenarios	Gradual ramp from zero players to number of players that overload the instance causing player experience impact
Metrics Collected	Connection success rate, connection latency (avg/P50/P95/P99/max), batch processing time, run queue depth, context switches/second, softirq/second, TCP retransmits, listen overflows
Test Duration	562 seconds total (202 s ramp at 100 connections/sec + 360 s sustained hold)

Results

Our testing results are displayed in the following charts. The new m8azn instances start to shine as load increases. M5zn instances start to have significant latency spikes and max latency numbers once we get to around 6,000 CCU, which severely impacts the player experience. We can push the m8azn instances to 8,000 CCU before experiencing player impact or introducing any latency spikes >500ms.

Metric	m5zn.2xlarge	m8azn.2xlarge
Player Count	3,000
Average Latency	9.7ms	8ms
P99 Latency	22ms	28ms
Max Latency	43ms	23ms
Peak Run Queue	575	14
TCP Re-transmits	0	0
Latency Spikes > 500ms	0	0
Errors	0	0

Metric	m5zn.2xlarge	m8azn.2xlarge
Player Count	4,000
Average Latency	8.6ms	7.2ms
P99 Latency	29ms	24ms
Max Latency	76ms	29ms
Peak Run Queue	451	24
TCP Re-transmits	0	0
Latency Spikes > 500ms	0	0
Errors	0	0

Metric	m5zn.2xlarge	m8azn.2xlarge
Player Count	6,000
Average Latency	6.8ms	6.5ms
P99 Latency	22ms	22ms
Max Latency	29ms	27ms
Peak Run Queue	318	27
TCP Re-transmits	0	0
Latency Spikes > 500ms	0	0
Errors	0	0

Metric	m5zn.2xlarge	m8azn.2xlarge
Player Count	8,000
Average Latency	25ms	9.7ms
P99 Latency	542ms	27ms
Max Latency	2386ms	238ms
Peak Run Queue	651	87
TCP Re-transmits	0	0
Latency Spikes > 500ms	97	0
Errors	0	0

Price-performance comparison for 100k CCU

One methodology to calculating price/performance of these instances is to compare the cost of running enough instances to serve 100,000 players while maintaining an optimal and minimal latency player experience. All prices discussed in this section are based on us-east-1 OnDemand Linux pricing at the time of writing.

To serve 100k CCU with m5zn.xlarge instances, we would need to provision approximately 20 m5zn.xlarge instances (each instance can support 5,000 CCU before player experience degrades). Each m5zn.xlarge costs $0.3303/hr. That brings our hourly cost to $6.606/hr per 100k CCU.

In comparison, we only need 13 m8azn.xlarge instances to serve 100k CCU, thanks to the ~61% performance improvement of average latency of m8azn at 8,000 CCU per instance. Each m8azn.xlarge costs $0.4129/hr. Our hourly cost for 100k CCU in this scenario is $5.3677/hr. M8azn instances clearly demonstrate better price performance when compared to m5zn instances.

Conclusion

M8azn instances represent a compelling upgrade path for multiplayer game studios currently running on older generation instances. The combination of AMD EPYC processor improvements, enhanced memory bandwidth, and superior network performance delivers measurable benefits across the workloads that matter most for game hosting.

Try out m8azn in your development environment and calculate price/performance gains to see if m8azn is right for your workload. The Optimizing EC2: Hands-on Strategies for Cost-effective Performance workshop can guide you in comparing performance and calculating your overall price/performance savings across different instances.

Additional Resources

Uncover new performance insights using Amazon detailed performance statistics on Windows

Xinze Zhang — Mon, 06 Jul 2026 17:00:08 +0000

The primary storage solutions for EC2 Windows instances, Amazon EC2 Instance Store and Amazon Elastic Block Store (Amazon EBS) , now provide detailed performance statistics for real-time monitoring. Real-time monitoring enables you to gain visibility into key performance metrics, such as latency, throughput, and IOPS, allowing you to detect and address potential bottlenecks or issues proactively.

In this post, we explore how to use detailed performance statistics for both Amazon EBS and Instance Storage on Windows environments. These new metrics provide sub-minute granularity, offering real-time visibility into storage volume performance across both storage types. You can access these statistics directly from your Amazon EBS NVMe/Amazon Instance Storage NVMe device attached to the Amazon Elastic Compute Cloud (Amazon EC2) instance and use them to monitor I/O performance at the storage level. We also provide examples of how to use these statistics to quickly assess EBS volume/Storage health and identify performance bottlenecks, which improve both the reliability and performance of your applications. When creating or attaching EBS volumes, enable encryption at rest using AWS Key Management Service (AWS KMS) to protect your data. For more information, see Amazon EBS encryption in the Amazon EC2 User Guide.

Solution overview

Using the new Amazon EC2 Instance Store/Amazon Elastic Block Store (Amazon EBS) detailed performance statistics at the instance-level, this sample solution enhances observability and troubleshooting capabilities for latency-sensitive applications running on EC2 Nitro instances. We use the new nvme_amzn.exe tool to collect high-frequency statistics on I/O operations, latency, and queue length, enabling proactive troubleshooting.

As examples of how to use these granular metrics, this solution demonstrates how to validate the responsiveness of local storage and EBS volume, so that you can quickly identify any I/O interruptions. This solution helps you identify storage performance bottlenecks, which can be used to optimize the local storage and EC2 instance configurations for your workloads.

Prerequisites

This solution involves setting up an EC2 Nitro instance and an attached local storage to access detailed performance statistics for the local storage. This is a setup you likely already have if using Amazon EC2. To deploy the required components, you must complete the following steps:

Launch an EC2 Nitro instance (or use an existing Nitro instance), and connect to it via Remote Desktop Protocol (RDP).
Verify that your EC2 Windows instance includes AWS NVMe driver version 1.7.0 or later installed by following identify your driver type
Identify the NVMe device associated with the local storage/EBS volume for which you want to query the stats. You can run the Get-Disk command in PowerShell to output all NVMe devices on the instance. For more information, see Map NVMe disks on Amazon EC2 Windows instance to volumes.

For this demonstration, we’ll monitor two storage volumes:

EBS volume (Disk 0): Serial Number vol01234567890abcdef_00000001.
Local storage (Disk 1): Serial Number AWSEXAMPLE1234567890_00000001.

Ensure that nvme_amzn.exe is present in C:\ProgramData\Amazon\Tools by default.
Use the nvme_amzn.exe tool, with administrator privileges, and pass the disk number as a parameter with different command. The returned output looks like the following.

Administrator: Windows PowerShell:

.\nvme_amzn.exe --help or nvme_amzn.exe /help

Users can see the EBS volumes devices mapping by default without passing the disk number as a parameter

.\nvme_amzn.exe

Users can view the specific device mapping by passing disk numbers or a single disk number as a parameter.

.\nvme_amzn.exe 0 1 2 3 4

Users can see the nvme controller details by using id-ctrl and pass the disk number as a parameter (JSON output can be retrieved by providing the --json or /json parameter to the tool)

# EBS volume
.\nvme_amzn.exe id-ctrl 0

# EC2 local storage
.\nvme_amzn.exe id-ctrl 1

.\nvme_amzn.exe id-ctrl 0 --json

Users can see the performance statistics for EBS/EC2 local storage volume by using stats and pass the disk number as a parameter (provide the --json or /json parameter to retrieve JSON output).

# EBS volume
.\nvme_amzn.exe stats 0
# Json format
.\nvme_amzn.exe stats 0 --json

# EC2 Local storage volume
.\nvme_amzn.exe stats 1
# Json format
.\nvme_amzn.exe stats 1 --json

In addition, for EC2 local storage volume, by providing the --details/-d option, you can see the histogram of 5 different IO bands: (0, 512 Byte], (512B, 4KiB], (4KiB, 8KiB], (8KiB, 32KiB], (32 KiB, MAX].

.\nvme_amzn.exe stats 0 --details

The following example shows NVMe log output with cumulative statistics. The statistics indicate read/write operations, bytes transferred, and time spent processing operations (in microseconds). They also show the number of microseconds in which the application attempted to exceed the Amazon EBS or Amazon EC2 Instance Local Storage IOPS/throughput limits

EBS volume:

EC2 local storage volume:

Also included in the following figures are read and write I/O latency histograms, with each row representing the total number of I/O operations completed so far within a specific bin of time (in microseconds).

These statistics are presented as cumulative counters up to the time at which the command is executed. The command can be run at the desired interval, for example, every 15 seconds, with each subsequent output reflecting the updated cumulative totals for the metrics. Calculating the difference in the statistics across the last two outputs allows you to derive insight into the instance storage profile over the given 15 second period.

Deriving insights from the Amazon Instance Storage/EBS volume detailed performance statistics

You have set up monitoring using these detailed performance statistics, now we can demonstrate the different ways you can use these statistics.

As mentioned in the preceding section, you can use the detailed statistics to view I/O latency histograms to observe the spread of I/O latency within the period. You can use the read/write operations and time spent statistics to calculate the average latency. Using the detailed statistics allows you to view the average latency at a sub-minute granularity.

Here are four examples for you to use the statistics to shed light on key performance metrics.

Scenario 1: Identifying unresponsive state of an EBS volume

In this scenario, we discuss how to use Amazon EBS detailed performance statistics to observe when an EBS volume isn’t responding to I/O operations. If you observe multiple intervals where your volume is unresponsive, then you can take actions, such as replacing the affected volume or stopping and restarting the instance to which the volume is attached. In most cases, when your volume becomes unresponsive, Amazon EBS automatically diagnoses and recovers your volume within a few minutes.

To identify if your volume is unresponsive, you can use the following steps to determine whether I/O disrupted on your volume:

Identify the EBS volume’s NVMe device to troubleshoot
Collect stats for the device at the desired intervals
Compare the stats to check if the EBS volume is unresponsive

Step 1: Identify the EBS volume’s NVMe device to troubleshoot

1. Identify the NVMe device associated with the EBS volume on the instance by using the nvme_amzn.exe tool.

.\nvme_amzn.exe

Step 2: Collect stats for the device at the desired intervals

1. Collect the Amazon EBS detailed performance statistics directly from the device by using the nvme_amzn.exe tool:

# EBS volume disk0
.\nvme_amzn.exe stats 0

Step 3: Compare the stats to check if the EBS volume is unresponsive

1. From the output, consider the following three fields for this scenario: Total Read Ops, Total Write Ops, and Queue Length.

2. Issue the same ebsnvme command after a desired interval (for example: after 15 seconds), so that you can compare how Total Read/Write I/Os have progressed at the Amazon EBS level.

3. From the detailed performance statistics collected approximately 15 seconds apart, we make the following key observations

Total Read Ops increased from 1421153 to 1423480, indicating 2327 Read operations completed in the 15 second span.
Total Write Ops increased from 13835137 to 13846338, indicating 11201 Read operations completed in the 15 second span.
Queue Length stayed between 0 and 6, indicating that the application was issuing I/Os to the EBS volume. If you see a gradual increase in the Queue Length, then it would reflect a buildup in queued I/Os.

This shows that the EBS volume is still driving I/Os that it is receiving, which rules out the EBS volume as the source of observed degradation in application performance. If we had seen an increase in the Queue Length along with 0 Read/Write Ops processed during the period, then it would reflect an unresponsive EBS volume.

If you would like to validate your mechanisms of identifying unresponsive EBS volumes, refer to the Conducting chaos engineering experiments on Amazon EBS using AWS Fault Injection Service blog post, which walks through how to set up an AWS Fault Injection Service Pause I/O experiment.

Scenario 2: Identifying bottlenecks in storage performance on EBS

Amazon EBS detailed performance statistics can also be used to configure the appropriate performance characteristics for your EBS volume and EC2 instance based on the performance needs of your application. The EBS Volume Performance Exceeded and EC2 Instance EBS Performance Exceeded statistics indicate the duration for which your workload consistently attempted to drive IOPS or throughput that is greater than your volume or your instance’s provisioned performance in a given period. Exceeding either the volume’s or instance’s provisioned performance can result in elevated latency on your workload. For this scenario, consider the same application as the one used in scenario 1.

Complete the following steps to check if EBS volume performance is correctly provisioned:

1. Select the EBS volume’s NVMe device to check
2. Collect stats for the device at the desired intervals
3. Compare the stats to check if the EBS volume is exceeding provisioned performance

Step 1. Select the EBS volume’s NVMe device to check

1. This step is the same as Step 1 discussed previously in scenario 1.

Step 2. Collect stats for the device at the desired intervals

1. Similar to Step 2 discussed in scenario 1, access the detailed performance statistics across two points in time.

2. Consider the EBS Volume Performance Exceeded and EC2 Instance EBS Performance Exceeded statistics from the EBS NVMe device.

$DiskNumber = 0
$Interval = 15

while ($true) {
    Write-Host "=== $(Get-Date -Format 'yyyy-MM-dd HH:mm:ss') ===" -ForegroundColor Cyan; & "C:\ProgramData\Amazon\Tools\nvme_amzn.exe" stats $DiskNumber
    Write-Host ""
    Start-Sleep -Seconds $Interval
}

Step 3: Compare the stats to check if the EBS volume is exceeding provisioned performance

1. In the following example output, you can see the EBS Volume Performance Exceeded statistic increasing by 26813772 microseconds. This shows the workload running on EBS volume vol-EXAMPLEabcd1234 has attempted to drive more IOPS than provisioned on the underlying EBS volume, which can impact the volume’s I/O latency. We recommend that you increase the performance of your volume to make sure that you have sufficient provisioned performance for your application’s needs.

2. In the following example output, driving a different workload on the instance allows us to see that the volume has exceeded the provisioned IOPS performance at the attached EC2 instance level. In this case, up-sizing to a larger instance size can improve the performance of your application.

3. A synthetic load generator for Oracle called Silly Little Oracle Benchmark (SLOB) could also be used to simulate workloads on Oracle databases, while monitoring the Amazon EBS statistics to see which volume or instance is becoming the bottleneck.

It’s important to have the right instance and volume configurations to avoid performance bottlenecks to your application. Refer to the EBS volume types documentation for more information on the different EBS volume types, and the Amazon EBS-optimized documentation to understand how to select the optimal combination of EC2 instance and EBS volume suited for your application. These statistics are available at up to a one-second granularity, which allows you to effectively perform these checks in real-time and initiate volume modifications to optimize volume characteristics as needed.

Scenario 3: Identifying bottlenecks in storage performance on instance storage volume

Amazon Instance Storage detailed performance statistics can be used to configure the appropriate performance characteristics for your application. The “EC2 Instance local storage Performance Exceeded” statistics indicate the duration for which your workload consistently attempted to drive IOPS or throughput that is greater than your rate limit in a given period. Exceeding the throttle value can result in elevated latency on your workload.

For example, i3en.xlarge can support up to 85,000 read IOPS, 65,000 write IOPS, 634,765 KiB/S for read and 317,382 KiB/S for write. By using the detailed IO metrics, you can more efficiently determine if the instance meets your requirements.

Complete the following steps to check if the device meets your application needs:

Select the instance storage device to check.
Collect stats for the device at the desired intervals
Compare the stats to check if the instance storage is exceeding the throttled value

Step 1. Select the Instance Storage NVMe device to check

Use the nvme_amzn tool and identify the NVMe device associated with the instance storage.

Step 2: Collect stats for the device at the desired intervals

$DiskNumber = 0
$Interval = 15

while ($true) {
    Write-Host "=== $(Get-Date -Format 'yyyy-MM-dd HH:mm:ss') ===" -ForegroundColor Cyan; & "C:\ProgramData\Amazon\Tools\nvme_amzn.exe" stats $DiskNumber
    Write-Host ""
    Start-Sleep -Seconds $Interval
}

Step 3: Compare the stats to check if the instance storage is exceeding throttle value

Take the following scenario as an example. At the very beginning, after the instance launch, both the IOPS and Throughput under “EC2 Instance local storage Performance Exceeded (us)” are 0s.

You start your applications and find that the application write does not meet your expectation. You can check the IO metrics afterwards. You see a lot of IO falls into the 1 ms to 2 ms range, which is unexpected.

By further checking the “EC2 Instance local storage Performance Exceeded (us)”. You found that the IO reached the allowed upper limit for up to 8 seconds, which indicates the i3en.xlarge would not meet your expectations. Select a larger instance size to address this.

It’s important to have the right instance size to avoid performance bottlenecks to your application. Refer to the ec2-instance-type-specifications documentation for more information on the different instance storage size to understand how to select the optimal instance size suited for your application. This tool helps you to effectively perform these checks in real-time.

Scenario 4: Identifying which block size caused the long latency on instance storage volume

You may have a mixed workload:

Data (either read or write) pattern with different block sizes like 4K and 128K.
Mixed read and write data pattern.

By using the --detail/-d switch from the NVMe CLI, you can identify the issue quickly and readjust the workload.

Example 1: High write latency from a workload

By further looking at the histogram of the block size range larger than 32 KiB, you can see that the larger IO caused high application latency, while other block sizes (like 8K) show no latency abnormality.

Example 2: Mixed read and write traffic

Some users will have a mix of read and write traffic. For example, some applications will do light read traffic (for example to read out some metadata) and heavy write. This may inadvertently impact the read latency. For example, an application is doing a read operation with a single IO of small block size. However, the user experiences high read latency. Examining the histogram breakdown, you could reasonably believe the heavy larger IO write may interfere with the read.

The detailed IO histogram for IO size larger than 512B but less than or equal to 4KB

The detailed IO histogram for IO size larger than 32 KiB

The user should consider smoothing out the write pattern to alleviate the read latency.

Cleaning up

If you created an EC2 instance and EBS volume for this exercise, then terminate and delete the appropriate instance and volumes to avoid future costs.

Conclusion

In this post, we presented a solution for accessing high-resolution performance statistics for Amazon EBS volumes and EC2 Instance Store at the instance level. These detailed metrics provide a real-time view into your underlying storage performance at sub-minute granularity, helping you to quickly root cause disruptions to your applications.

This approach also helps you identify performance bottlenecks caused by workloads exceeding your provisioned IOPS or throughput limits on Amazon EC2, EBS volumes, or EC2 Instance Store. Combined with Amazon CloudWatch metrics, which provide volume-level insights at one-minute granularity, these tools help give you the visibility you need to confidently diagnose and resolve storage-related performance issues.

Building fault-tolerant multi-agent AI workflows with AWS Lambda durable functions

Satish Kamat — Mon, 29 Jun 2026 14:26:18 +0000

Agentic AI workflows coordinate multiple agents that reason, plan, and act across multi-step processes. Each step is expensive, non-deterministic, and unpredictable in latency. Human review gates can pause execution for days. Transient failures are expected, and restarting a half-finished workflow wastes time and money. Duplicate actions, like charging a payment twice or sending the same request again, create financial and compliance risk. Until now, solving these problems meant building custom infrastructure such as state machines, queues, checkpoint stores before writing a single line of business logic.

Prior authorization is one of the most time-consuming steps in healthcare delivery. A provider must get approval from an insurer before certain treatments or medications are covered. The insurer evaluates whether the care is medically necessary, safe, and cost-effective.

Agentic AI is transforming this process. What previously took days — extracting clinical data, evaluating medical necessity, checking payer-specific criteria, and getting physician sign-off — can now be handled by AI agents that pull records, apply guidelines, and draft justification letters automatically.

This post shows how AWS Lambda durable functions can orchestrate an agentic healthcare prior authorization workflow. The pipeline coordinates multiple AI agents, a human review gate, and an external payer submission into a single fault-tolerant function. Using two key patterns — callbacks for human-in-the-loop approvals and asynchronous agent invocations, and polling for long-running external tasks — Lambda durable functions let you focus on the clinical workflow rather than building custom state machines, retry logic, and checkpoint infrastructure.

Overview of AWS Lambda durable functions

Lambda durable functions extend the standard Lambda programming model with a checkpoint and replay mechanism. You wrap your handler with the durable execution SDK, which enhances the Lambda context with durable operations such as context.step(), context.waitForCallback(), and context.waitForCondition(). These operations checkpoint progress, handle failures, and suspend execution during wait periods. If a failure occurs or the function resumes after being suspended, Lambda invokes your function again. It restores the previous state by replaying the event handler from the start and skipping over previously completed durable operations. Lambda durable functions offer additional patterns such as parallel execution, durable invocations, and saga-style compensations. Refer to the AWS Durable Execution SDK Developer Guide for the full set of capabilities.

Agentic AI workflows are a natural fit for durable functions because each agent invocation is typically expensive, slow, and prone to transient failures, which are exactly the properties that benefit from automatic checkpointing and replay. Beyond orchestrating agent steps, durable functions can pause the workflow execution for external input. You can suspend the execution until a human approval arrives, or poll an external system for completion with configurable backoff. For on-demand functions, you don’t incur compute charges while execution is suspended (see Lambda pricing for details).

The healthcare prior authorization pipeline

The prior authorization workflow orchestrator coordinates four AI agents, a human review gate, and a payer submission.

Clinical extraction agent (step). Extracts relevant clinical data (diagnosis codes, procedure history, lab results) from the patient’s medical records.
Medical necessity agent (step). Evaluates whether the procedure meets clinical guidelines based on the extracted data.
Payer criteria agent (step). Checks the specific payer’s authorization requirements and identifies any missing documentation.
Justification generation agent (step). Produces the prior authorization justification letter using the outputs of the previous three agents.
Physician review (callback). The orchestrator suspends and waits for a physician to review and approve the generated justification. Because this uses waitForCallback(), the function incurs no compute charges while the physician takes minutes, hours, or days to respond.
Payer submission and adjudication (polling). Once approved, the orchestrator submits the authorization request to the payer system using an idempotent step with a clientRequestToken (shown in the code below) to help prevent duplicate submissions. It then polls the payer’s adjudication status using waitForCondition() with exponential backoff, suspending between each check.

Figure 2. The six-stage prior authorization pipeline, orchestrated by a single Lambda durable function.

Putting it together in code

The entire pipeline, from agent steps to human review to payer submission and polling, lives in a single function that reads top to bottom:

@durable_execution
def handler(event: dict, context: DurableContext) -> dict:
    # 1-4: Sequential agent steps (each checkpointed)
    clinical_data = context.step(extract_clinical_agent(event["patient_id"]))
    necessity = context.step(medical_necessity_agent(clinical_data))
    criteria = context.step(payer_criteria_agent(event["payer_id"], necessity))
    justification = context.step(justification_agent(clinical_data, necessity, criteria))

    # 5: Suspend until physician approves (minutes to days, zero compute cost)
    def submit_for_review(cb_id, ctx):
        send_to_review_system(cb_id, justification)

    approval = context.wait_for_callback(
        submitter=submit_for_review,
        config=WaitForCallbackConfig(timeout=Duration.from_days(7)),
        name="physician_review",
    )

    if not approval.get("approved"):
        return {"status": "REJECTED", "reason": approval.get("reason")}

    @durable_step
    def make_idempotency_key(ctx: StepContext) -> str:
        return str(uuid.uuid4())

    # 6: Submit to payer and poll for decision
    idempotency_key = context.step(make_idempotency_key(), name="idempotency-key")
    submission = context.step(submit_authorization(justification, idempotency_key))

    def check_payer_status(state, ctx):
        return {**state, "status": get_payer_status(state["submission_id"])}

    decision = context.wait_for_condition(
        check=check_payer_status,
        config=WaitForConditionConfig(
            initial_state={"submission_id": submission["id"], "status": "PENDING"},
            wait_strategy=payer_adjudication_strategy,
        ),
        name="payer_adjudication",
    )

    return {"status": decision["status"], "authorization_id": submission["id"]}

How the orchestrator handles failures

The orchestrator is designed to handle the failure modes that come up in real workflows:

An agent step fails. If the medical necessity agent fails after the clinical extraction agent has completed, Lambda durable function replays the handler, skips the extraction step which was already checkpointed, and retries only the failed step. This helps avoid re-incurring the time, cost, and token spend of completed steps.
The physician rejects the justification. The callback returns approved: false, the orchestrator returns a REJECTED status, and no payer submission occurs.
Payer adjudication exceeds the max attempts. waitForCondition() raises a timeout error after the configured attempt limit, which you can catch and route to a manual review queue or compensating action.
The submit step retries after a transient failure. Because the submission carries a clientRequestToken derived from the execution ID, retries against the payer are idempotent at the payer API level, which helps prevent duplicate authorization requests.

The callback pattern

The callback pattern allows the orchestrator to suspend execution and wait for an external signal before resuming. When the durable function reaches a context.waitForCallback(), it sends a unique callbackId to an external system and then suspends. When the external system completes its work, it calls the Lambda API with SendDurableExecutionCallbackSuccess (or SendDurableExecutionCallbackFailure) to resume the orchestrator from where it left off.

In the prior authorization pipeline, this is how the physician review step works. After the justification generation agent produces a letter, the orchestrator emits a callback ID to the clinical review system and suspends. The physician receives the draft in their review queue, reads it, and either approves or rejects it through the review UI. The UI calls the Lambda callback API with the result, and the orchestrator resumes with the approval decision.

Because the function is fully suspended, it incurs no compute charges during the review window, whether that’s 10 minutes or 3 days.

Figure 3. The callback flow for the physician review step. The orchestrator emits a callback ID to the clinical review system and suspends. When the physician approves or rejects, the review system calls SendDurableExecutionCallbackSuccess to resume the orchestrator with the decision.

The callback pattern is appropriate when:

A human needs to review and approve a result (hours to days).
An external agent is invoked asynchronously and the orchestrator should resume when it finishes.
A webhook or third-party system signals completion.

The polling pattern

When an external system cannot send a callback, for example a payer API that offers no webhook support, the polling pattern provides an alternative. The orchestrator monitors the long-running task by periodically checking its status using context.waitForCondition().

# Poll the payer's adjudication API with exponential backoff
decision = context.wait_for_condition(
    check=lambda state, ctx: {**state, "status": get_payer_status(state["submission_id"])},
    config=WaitForConditionConfig(
        initial_state={"submission_id": "auth-789", "status": "PENDING"},
        wait_strategy=create_wait_strategy(WaitStrategyConfig(
            should_continue_polling=lambda state: state["status"] == "PENDING",
            max_attempts=48,
            initial_delay=Duration.from_seconds(30),
            max_delay=Duration.from_seconds(300),
            backoff_rate=2.0,
        ))
    ),
    name="payer_adjudication",
)

It runs a check function periodically as configured by a wait strategy and evaluates the result. If the task isn’t complete, suspends for a configurable delay before checking again. The function incurs no compute charges during each wait interval. Each poll result is automatically checkpointed, so on replay the orchestrator skips previously completed checks.

In the prior authorization pipeline, this is how the payer adjudication step works. Most payer APIs accept a submission and return a tracking ID, but don’t push a completion signal back. The orchestrator calls waitForCondition() with the payer’s status API, an exponential backoff strategy (30 seconds to 5 minutes), and a maximum attempt count that covers the payer’s typical adjudication window.

Lambda durable functions provide waitForCondition() with built-in support for configurable backoff strategies, maximum attempt limits, and timeouts, which can help reduce the need for separate polling infrastructure such as scheduled rules, state machines, or custom retry logic.

Figure 4. The polling flow for the payer adjudication step

Polling is appropriate when:

An async job does not support callbacks.
An external API or system exposes only a status or Describe endpoint.
The orchestrator waits for a resource to become available.

Cost and operational concerns

Here are a few implications when using orchestration of agentic workflows with Lambda durable functions:

Retries don’t re-run completed agents. If the fourth agent fails, the first three are not re-invoked, so the organization does not pay token costs twice for the same work.
Idempotency tokens help prevent duplicate payer submissions. A retry that crosses the submission step reuses the clientRequestToken, which helps the payer deduplicate on their side. This is an important property when duplicate authorization requests can trigger compliance issues.
Replay-aware logger streamlines logging. The SDK’s logger (context.logger) is replay-aware, meaning that it automatically suppresses duplicate log lines during replay.
Operational visibility is consolidated. Instead of stitching together logs from a state machine, a queue, a checkpoint table, and a poller, the entire workflow is one function with one execution history. Lambda publishes durable-execution-specific Amazon CloudWatch metrics, including ApproximateRunningDurableExecutions, DurableExecutionDuration, and DurableExecutionFailed, so you can track running workflows, detect failures, and set alarms at the execution level. Lambda also publishes durable execution status change events to Amazon EventBridge (RUNNING, SUCCEEDED, FAILED, TIMED_OUT) for triggering notifications or downstream workflows, and you can enable AWS X-Ray for distributed tracing across the entire execution. For more details, see Monitoring durable functions in the Lambda developer guide.

Using coding agents to build and test durable functions

To accelerate building agentic workflow orchestration with Lambda durable functions, you can use the Kiro power for Lambda durable functions or the Agent Plugin for AWS Serverless, which is available in any AI coding assistant tool that supports agent plugins such as Claude Code and Cursor. You can also install agent skills from the plugin individually in any AI coding assistant tool that supports agent skills. This helps your coding agents such as Kiro to:

Scaffold an orchestrator function from a prose description of the workflow, wiring up context.step(), wait_for_callback(), and wait_for_condition() calls based on the described stages.
Generate unit tests that exercise the replay behavior, including tests that inject failures at specific steps to confirm that completed checkpoints are skipped on retry.
Generate integration tests that simulate callback delivery and polling responses so you can validate end-to-end behavior without a full external system.

Conclusion

Agentic AI workflows can be non-deterministic, long-running, and failure-prone. Lambda durable functions can help address these challenges by adding checkpointing, replay, and suspension to the Lambda programming model, so completed work is skipped on retry and failures resume exactly where they occurred.

In this post, we walked through a healthcare prior authorization pipeline to illustrate two patterns: Callbacks for human-in-the-loop approvals and asynchronous agent invocations, and polling for monitoring long-running external tasks.

Beyond these two patterns, Lambda durable functions offer additional capabilities for building resilient workflows such as parallel execution, child contexts for isolated execution context for grouping operations, and saga-style compensations. Refer to the Lambda durable functions Developer Guide for the full set of capabilities. For pricing of on-demand and provisioned-concurrency functions, see the Lambda pricing page.

Get started with Lambda durable functions with examples from Serverlessland and install the Agent Plugin for AWS Serverless.

AWS Compute Blog

Amazon Linux default SSM parameter will now track the latest kernel

What’s changing?

Why are we making this change?

Evaluating the default kernel upgrade

Staying on or reverting to a specific kernel version

Conclusion

Set up your AI coding agent to build with AWS Step Functions

How it works

Supported agents

What you can build

Advantages

Get started

Introducing public preview runtimes on AWS Lambda, starting with Node.js 26 and Python 3.15

Why public preview runtimes

What’s included in the preview runtimes

What to expect during preview

Share your feedback

Transition to GA

Getting started

Console

AWS CLI

AWS CloudFormation

AWS SAM

AWS CDK

Conclusion

Implementing dynamic feature flags with AWS AppConfig on AWS Lambda

The challenge: dynamic configuration in serverless applications

How the AWS AppConfig Lambda extension works

Deploying the solution with AWS SAM

Prerequisites

Reading feature flags from your Lambda function

Safe deployments with deployment strategies

Updating feature flags without code deployments

Best practices

Clean up

Conclusion

Observability best practices for Lambda durable functions

Architecture overview

Durable function CloudWatch metrics, custom business metrics, and alarms

Custom business metrics for the callback funnel

Alarms for callback failure modes

Unified dashboard

Tracing callbacks across the suspension boundary

Structured logging with correlation keys

Durable steps and X-Ray annotations

Durable executions tab

Putting it together: debugging real failure modes

Scenario 1: Webhook never arrives

Scenario 2: The whole workflow runs too long

Scenario 3: Customer abandons checkout

Conclusion

Collecting CPU and memory metrics for AWS Lambda MicroVMs

How to collect CPU and memory metrics in your MicroVM

Lambda-injected environment variables

Setting up custom metric dimensions from env variables

Configuring CloudWatch Agent

Step 1: Configure the telegraf plugin to emit CPU and Memory metrics

Step 2: Configure OTel to process and export the metrics to CloudWatch

Step 3: Install CloudWatch Agent in your Dockerfile

Step 4: Configure your Entrypoint to start CloudWatch Agent

Execution role requirements

Verifying it works

Viewing the metrics

Conclusion

Designing for failure: Building resilient systems on AWS

The 2 AM wake-up call nobody wants

What actually happened

How the team responded and what they changed

1. Use Partition Placement Groups to isolate failure domains

2. Monitor system status checks and use composite alarms

3. Automate instance recovery and replacement

4. Build automated incident response with AWS Systems Manager

5. Invest in observability before you need it

Placement groups: Quick reference

The bigger picture: Resilience is a practice

Related posts

Key takeaways

Burst to Region: Overflow AWS Outposts workloads to Amazon EC2

When to use this pattern