AWS Compute Blog

AWS Outposts monitoring and reporting: A comprehensive Amazon EventBridge solution

Matt Price — Tue, 14 Apr 2026 16:18:12 +0000

Organizations using AWS Outposts racks commonly manage capacity from a single AWS account and share resources through AWS Resource Access Manager (AWS RAM) with other AWS accounts (consumer accounts) within AWS Organizations. In this post, we demonstrate one approach to create a multi-account serverless solution to surface costs in shared AWS Outposts environments using Amazon EventBridge, AWS Lambda, and Amazon DynamoDB. This solution reports on instance runtime and allocated storage for Amazon Elastic Compute Cloud (Amazon EC2), Amazon Relational Database Services (Amazon RDS), and Amazon Elastic Block Store (Amazon EBS) services running on Outposts racks. In turn, teams can track the cost of infrastructure associated with their workloads across AWS accounts. This solution is a framework that can be customized to meet your organization’s specific business objectives.

Solution overview

The following is the Terraform-based reference architecture used to represent the solution, including EventBridge, DynamoDB, and Lambda across a multi-account environment. Relevant launch events are tracked in EventBridge that invoke Lambda functions, which are logged in DynamoDB tables (see sample code). This allows reporting on captured event data through the AWS SDK for Python (Boto3).
Figure 1: Reference architecture for reporting solution on AWS Outposts

Prerequisites

The following prerequisites are necessary to implement this solution:

At least two active AWS accounts in the same AWS Organization as the Outposts owner account.
- One AWS account, which is the data collection account to store the event data (this doesn’t have to be the account that owns the Outposts).
- Workload accounts where resources are deployed on Outposts.
AWS Command Line Interface (AWS CLI) installed and configured on an administrative instance. For more information, see Installing, updating, and uninstalling the AWS CLI in the AWS CLI documentation.
Terraform installed on the same administrative instance. For more information, see the Terraform documentation.
Make sure that you have the necessary AWS Identity and Access Management (IAM) permissions necessary to create the AWS resources using Terraform in all accounts.
Prior Experience with Terraform deployments on AWS Cloud. To increase your familiarity, you can explore Get Started – AWS on the HashiCorp website.
Access to clone the AWS Outposts Monitoring and Reporting git repository.
SDK for Python installed and configured on a local machine.

Walkthrough

The following sections walk you through how to deploy this solution.

Deploying in data collection account

Step 1: Create a bucket in-Region to hold the Terraform state file in the data collection account.

aws s3 mb s3://state-bucket-name

Step 2: Clone the repository.On your local machine, clone the repository that contains the sample by running the following command:

git clone https://github.com/aws-samples/sample-outposts-monitoring-and-reports.git

Navigate to the cloned repository by running the following command:cd sample-outposts-monitoring-and-reports/data_collection

Step 3: Edit the providers.tf to configure the AWS provider.



provider "aws" {
  region = ""
}

Step 4: Edit the backend.tf to provide the Terraform state bucket and Outposts anchored AWS Region.

terraform {
  backend "s3" {
    bucket = ""
    key    = "terraform.tfstate"
    region = ""
  }
}

Step 5: Modify the variables.tf.From the root directory of the cloned repository, modify the variables.tf file with the target Region and workload accounts as shown in the following example. The target Region is the collection destination.

variable "aws_region" {
  description = "AWS region for resources"
  type        = string
  default     = ""
}

variable "allowed_account_id" {
  description = "AWS account ID allowed to put events to the event bus"
  

}

Initialize the configuration directory of the data collection account to download and install the providers defined in the configuration by running the following command:

terraform init

All resources are deployed with minimal permissions to serve as an example. We recommend viewing all configurations to make sure that they meet your organizational security policies. Step 6: Deploy infrastructure in the data collection account.Run terraform plan on the configuration to and review which resources are created:

terraform plan

When you have reviewed the plan, run the following command and enter “yes” to accept the changes and deploy:

terraform apply

Deployment should take less than 5 minutes. If you receive any errors, review the previously mentioned steps to ensure that you followed them in their entirety. If the errors persist, reach out to AWS Support for additional guidance.

Deploying in workload account

The data collection account receives events from EventBridge and performs intelligent analysis and storage from the AWS Outposts resource data.Step 1: Navigate to the workload account directory by running the following command:

cd ../workload_account

Step 2: Edit variables.tf to set up the Region and event bus Amazon Resource Name (ARN).

variable "aws_region" {
  description = "AWS region for resources"
  type        = string
  default     = ""
}

variable "event_bus_arn" {
  description = "target event bus arn"
  type        = string
  default     = ""
}

Edit the code to update the event bus name.

Step 3: Run the following command to create the backend.tf and create the Terraform state bucket for each workload account.

./init-backend.sh

This is an idempotent operation that creates a file from the template and a bucket with a fixed name including the account ID if it doesn’t exist.

Step 4: Initialize the configuration directory of the Data Collection Account to download and install the providers defined in the configuration by running the following command:

terraform init

Step 5: Deploy the infrastructure in the Data Collection Account.Run a terraform plan on the configuration and review which resources are created:

terraform plan

After you have reviewed the plan, run the following command and enter “yes” to accept the changes and deploy:

terraform apply

Deployment should take less than 5 minutes. If you receive any errors, follow the troubleshooting steps in the previous section.

At this point, any Amazon EC2 or Amazon RDS instances and Amazon EBS volumes are logged to the DynamoDB tables in the data collection account. Repeat Steps 3–5 for each workload account running resources on AWS Outposts with appropriate account credentials. If you’re deploying at scale and using AWS Control Tower consider using AWS Control Tower Account Factory for Terraform (AFT).

Running monthly reports

With this solution in place, reports can be generated on demand. These reports can be customized by modifying the Python example scripts shown to support your needs. Reports can be created from a local machine with credentials that have access to the DynamoDB tables in the data collection account. The examples were created from the source directory of the data collection account git repository. Run the following command to view the report for Amazon RDS usage in September 2025:

./rds_runtime_calculator.py --year 2025 --month 9 --output rds_report.csv

Figure 2: Example of RDS runtime report

Run the following command to view the report for Amazon EBS usage in September 2025:

./ebs_volume_reporter.py --year 2025 --month 9 --output ebs_report.csv

Figure 3: Example of EBS usage report

Run the following command to view the report for Amazon EC2 usage in September 2025:

./ec2_runtime_calculator.py --month 9 --year 2025 --output ec2_report.csv

Figure 4: Example of EC2 runtime report

Cleaning up

Complete the following steps to clean up the resources that were deployed by this solution. For each workload account, complete the following:

cd sample-outposts-monitoring-and-reports/workload_account

terraform destroy

Enter “yes” to proceed. You can then manually empty and remove the terraform state S3 bucket for that account.

For the data collection, complete the following:

cd ../data_collection

terraform destroy

Enter “yes” to proceed. You can then manually empty and remove the terraform state S3 bucket for that account.

Conclusion

Customers who have shared multi-account Outposts deployments can use this solution to create account level reporting for Outposts resources using real-time event capture and processing, state analysis and categorization, historical usage metrics, and serverless architecture. Teams can use this to visualize and report on the costs of running their workloads on Outposts. The event-driven design supports accurate tracking while maintaining low operational overhead. The solution scales effectively across multiple Outposts and accounts, providing a unified view of hybrid infrastructure. Keep in mind that you can extend the functionality described here to meet your business objectives.

Deploy this solution today using the GitHub repository to gain financial insights to share with the tenants of your Outposts workload accounts. Reach out to your AWS account team, or fill out this form to learn more about Outposts.

Building Memory-Intensive Apps with AWS Lambda Managed Instances

Guy Haddad — Fri, 10 Apr 2026 19:54:44 +0000

Building memory-intensive applications with AWS Lambda just got easier. AWS Lambda Managed Instances gives you up to 32 GB of memory—3x more than standard AWS Lambda—while maintaining the serverless experience you know. Modern applications increasingly require substantial memory resources to process large datasets, perform complex analytics, and deliver real-time insights for use cases such as in-memory analytics, Machine Learning (ML) model inference, and real-time semantic search. AWS Lambda Managed Instances gives you a familiar serverless programming model and experience combined with the flexibility of being able to choose the underlying Amazon EC2 instance types and providing developers with access to large memory configurations.

In this post, you will see how AWS Lambda Managed Instances enables memory-intensive workloads that were previously challenging to run in serverless environments, using an AI-powered customer analytics application as a practical example. You’ll see cost savings of up to 33% compared to standard Lambda for predictable workloads, while eliminating the operational overhead of managing EC2 instances.

Understanding AWS Lambda Managed Instances

AWS Lambda Managed Instances runs your AWS Lambda functions on the Amazon EC2 instance types of your choice in your account, including Graviton4 and memory-optimized instance types. AWS handles underlying infrastructure lifecycle including provisioning, scaling, patching, and routing, while you benefit from Amazon EC2 pricing advantages like Savings Plans and Reserved Instances.

Key benefits include:

Flexible instance selection: Choose from compute-optimized (C), general-purpose (M), and memory-optimized (R) instance families
Configurable memory-CPU ratios: Optimize resource allocation for your workload
Multi-concurrent invocations: One execution environment handles multiple invocations simultaneously, improving utilization for I/O-heavy applications
Dynamic scaling: Instances scale based on CPU utilization without cold starts

AWS Lambda Managed Instances is best suited for high-volume, predictable workloads that benefit from sustained compute capacity and larger memory configurations.

Memory-Intensive Workloads Work Best with AWS Lambda Managed Instances

This blog focuses on one of AWS Lambda Managed Instances’ most powerful capabilities: running memory-intensive workloads that require more than the standard AWS Lambda’s 10 GB memory and 250MB ZIP limits. Here are the use cases where AWS Lambda Managed Instances helps:

In-Memory Analytics — Load gigabytes of structured data into memory at initialization and serve sub-millisecond analytical queries across thousands of invocations
ML Model Inference — Keep large model weights resident in memory across invocations for consistent, low-latency inference without a dedicated endpoint.
Real-Time Semantic Search — Build vector similarity search over large embedding indexes held entirely in memory, enabling natural language queries over millions of records without an external vector database.
Graph Processing — Hold large graph structures in memory for traversal algorithms that require the full graph to be accessible at once.
Scientific & Numerical Computing — Run simulations, Monte Carlo methods, and large matrix operations that require substantial working memory and benefit from memory-optimized Amazon EC2 instance families.
Large-Scale Report Generation — Aggregate and transform multi-gigabyte datasets in memory to generate complex reports or dashboards on demand, without staging data through intermediate storage.

Use Case: AI-Powered Customer Analytics with AWS Lambda Managed Instances

To demonstrate the power of AWS Lambda Managed Instances for memory-intensive applications, we built an AI-Powered Customer Analytics application that combines in-memory data processing with ML-based semantic search. The application loads in memory 1 million customer behavioral records (sessions, purchases, browsing patterns) from a Parquet file in S3 into a Pandas DataFrame and an embeddings cache consuming 200MB, then responds for analytics queries:

Customer Analysis — Deep-dive into individual customer behavior: engagement scores, conversion rates, purchase patterns, and AI-generated customer segments
Semantic Search — Natural language queries powered by FastEmbed (sentence-transformers/all-MiniLM-L6-v2) that find similar customers using vector similarity
Cohort Analysis — Real-time segmentation by device, country, age group with aggregated metrics

Architecture Overview

Our AI-powered customer analytics application demonstrates this in practice: 1 million records in memory (200MB), a compact sentence transformer model for semantic search, sub-second query performance, and zero infrastructure to manage. The solution uses a simple, serverless architecture:

Customer transaction data (Parquet format) is stored in Amazon S3
Amazon Cognito User Pool authenticates users and issues JWT tokens for API access
Amazon API Gateway routes requests with Cognito authorizer validation, rate limiting (5 requests/second, burst 10), X-Ray tracing, and access logging
AWS Lambda function with AWS Lambda Managed Instances loads the entire dataset (200MB) and all-MiniLM-L6-v2 model (900MB) into memory during initialization while also performing a threaded embeddings cache generation. This step can consume about 14GB of the allocated memory, exceeding standard AWS Lambda’s 10 GB limit
Analytics queries execute against the in-memory data using the model
Results are returned in milliseconds for interactive analysis

Deploy the Application

The below steps walk you through deploying the application to AWS using the AWS Serverless Application Model (SAM). The deployment process packages your Lambda function code, uploads artifacts to Amazon S3, and provisions all required AWS resources including Lambda functions, IAM roles, and any configured VPC networking via AWS CloudFormation.

Prerequisites

Make sure you have the following tools installed locally:

AWS CLI configured with credentials
SAM CLI installed
Python 3.13+ installed locally
Docker or Finch (required for container builds)
AWS account with appropriate permissions
A VPC with at least 2 subnets (across different Availability Zones) and a security group — required for the Lambda Managed Instances capacity provider
Supported regions: Check AWS Capabilities by Region for supported regions

Getting Started

The complete source code for this application is available in our GitHub repository. To deploy it yourself follow the below steps and refer to the full deployment instructions hosted on GitHub.

1. Clone the repository

git clone https://github.com/aws-samples/sample-lambda-managed-instances-analytics.git

2. Navigate to the project folder

cd sample-lambda-managed-instances-analytics

chmod +x setup-data.sh deploy-lambda.sh

3. Generate sample data and upload to S3

./setup-data.sh

This script will create an S3 bucket (if needed), generate 1M rows of sample data, and upload the data to S3.

4. Build and deploy the Lambda function

./deploy-lambda.sh

This script will build the container image with FastEmbed, push it to ECR, and deploy the Lambda function along with Capacity Provider, API Gateway, and Cognito User Pool. After deployment, it automatically generates the UI authentication configuration and prompts you to create a test user.

Run the Application

1. Start the UI

The application includes a simple HTML-based UI through which you can test the AWS Lambda function using Amazon API Gateway:

cd ui && python3 -m http.server 8000

2. Open your browser at http://localhost:8000 and click ‘Sign In’ to authenticate via Cognito using the username/password that you created during deployment

3. Enter your API endpoint URL. Test connection and click system Info.

Test the Application

a. Customer Analysis — Enter one or more User IDs to get more information on the customer behavior: engagement scores, conversion rates, purchase patterns, and AI-generated customer segments

b. Semantic Search – Enter natural language queries like “list high value customers from USA” in the Semantic Search and verify the results. Note that the response is very fast as the analytics data and FastEmbed models are loaded into memory during init stage

c. Cohort Analysis — Enter the query data to get Real-time segmentation by device, country, age group with aggregated metrics

Observability

AWS Lambda Managed Instances automatically publishes metrics to Amazon CloudWatch, giving you visibility into function performance and capacity utilization. Monitor InitDuration to track dataset and model load time at startup, MaxMemoryUsed to confirm your data fits within configured memory, and ProvisionedConcurrencySpilloverInvocations to detect when AWS Lambda Managed Instances capacity is exhausted.

Enable AWS Lambda Insights for enhanced per-invocation metrics including CPU time and memory utilization over time. Use Amazon CloudWatch Log Insights to query INIT_START, INIT_END, and REPORT log entries for initialization and memory details per invocation.

What Makes This Better with AWS Lambda Managed Instances

Without AWS Lambda Managed Instances, building this same application would require one of these alternatives:

Option A: EC2 with auto-scaling — Full control, full responsibility: patching, scaling policies, load balancing, and deployment pipelines — all on you.
Option B: Redesign for standard Lambda — Swap in-memory data for an external database and replace the ML model with Amazon SageMaker endpoint. More latency, more cost, more complexity.

With AWS Lambda Managed Instances, you write a single AWS Lambda function, define a Capacity Provider, and deploy with SAM. AWS Lambda handles the Amazon EC2 instances, scaling, and lifecycle, giving you the memory you need with the operational simplicity you want. The in-memory approach eliminates network latency and disk I/O, delivering consistent sub-200ms response times for complex analytics.

Cost Considerations

AWS Lambda Managed Instances uses Amazon EC2-based pricing with a management fee. For predictable workloads, you can leverage Amazon EC2 Savings Plans or Reserved Instances to reduce costs significantly.

Example cost comparison (us-east-1, 32 GB memory, 1M invocations/month):

AWS Lambda (standard): ~$267/month (on-demand pricing)
AWS Lambda Managed Instances: ~$180/month (with 1-year Compute Savings Plan)
Savings: 33% reduction

The cost benefits increase with higher memory configurations and sustained workloads that can take advantage of Amazon EC2 pricing discounts.

Best Practices

Based on experience building this solution, here are key recommendations:

Memory sizing: Start with your dataset size plus 50% overhead for processing. Monitor Amazon CloudWatch metrics to optimize.
Initialization strategy: Load large datasets during the init phase to amortize the cost across multiple invocations.
Concurrency configuration: Set PerExecutionEnvironmentMaxConcurrency based on your workload’s I/O characteristics. Higher values work well for I/O-bound analytics.
Data format: Use columnar formats like Parquet for efficient memory usage and fast loading.
Monitoring: Track initialization duration, memory utilization, and invocation latency in Amazon CloudWatch to identify optimization opportunities.

Cleanup

When you’re done exploring the solution, it’s good practice to remove all provisioned resources to avoid ongoing charges. For the full cleanup commands and exact steps, refer to the project’s README.md in GitHub repository.

Conclusion

AWS Lambda Managed Instances opens up a new class of serverless applications that support larger AWS Lambda layer packages and more memory. Memory-intensive workloads — in-memory analytics, ML inference, graph processing, scientific computing — can now run with the simplicity of AWS Lambda and the resources of Amazon EC2. The customer analytics example demonstrates how in-memory processing with AWS Lambda Managed Instances delivers performance improvements over traditional database queries while maintaining serverless benefits like automatic scaling and pay-per-use pricing.

Ready to get started? Explore the AWS Lambda Managed Instances documentation and try building your own memory-intensive serverless application. You can find the complete code for this example on GitHub.

Accelerate CPU-based AI inference workloads using Intel AMX on Amazon EC2

Santosh Kumar — Mon, 30 Mar 2026 16:43:10 +0000

This post shows you how to accelerate your AI inference workloads by up to 76% using Intel Advanced Matrix Extensions (AMX) – an accelerator that uses specialized hardware and instructions to perform matrix operations directly on processor cores – on Amazon Elastic Compute Cloud (Amazon EC2) 8th generation instances. You’ll learn when CPU-based inference is cost-effective, how to enable AMX with minimal code changes, and which configurations deliver optimal performance for your models.

Many organizations find that CPU-based inference is more suitable for their production Artificial Intelligence/Machine Learning (AI/ML) workloads after evaluating factors like cost, operational complexity, and infrastructure compatibility. As more organizations deploy AI solutions, improving how models run on standard CPUs has become a critical cost control strategy for workloads where CPU inference provides the right balance of performance and economics.

IDC, a global market intelligence and advisory firm, projects that worldwide AI spending will reach $632 billion by 2028, growing at a 29% compound annual growth rate from 2024, with inference costs representing a significant portion of operational expenses. Deloitte, a leading professional services firm specializing in technology consulting and research, forecasts that inference – the running of AI models – will make up two-thirds of all AI compute by 2026, far exceeding initial training costs. This makes optimizing AI/ML inference on CPU crucial for controlling long-term AI/ML operational expenses.

At the core of AI inference workloads are matrix multiplication operations – the mathematical foundation of neural networks that drives computational demand. These matrix-heavy operations create a performance bottleneck for CPU-based inference, resulting in suboptimal performance for AI/ML workloads. This creates three key challenges for organizations: balancing cost optimization with performance requirements, meeting real-time latency demands, and scaling efficiently with variable workload demands. Intel’s Advanced Matrix Extensions (AMX) technology addresses these challenges by accelerating matrix operations directly on CPU cores, making CPU-based inference competitive and cost-effective.

AMX capabilities and architecture

AMX supports multiple data formats including BF16 which preserves the range of 32-bit floating point operations in half the space, INT8 maximizes throughput when accuracy can be slightly compromised, and FP16 offers a balance between the two. This flexibility lets you match precision to your specific needs.

Introduced in 2023 with 4th Generation Intel Xeon Scalable processors, AMX consists of eight 1KB tile registers (specialized on-chip memory for matrix data) and a Tile Matrix Multiply Unit (TMUL – dedicated hardware for matrix calculations) that enables processors to perform 2048 INT8 operations or 1024 BF16 operations per cycle. These tile registers provide efficient matrix storage, reducing memory access overhead and improving computational efficiency for matrix operations central to neural networks. For real-world customer workloads, this translates to significantly faster inference times for transformer models, recommendation systems, and natural language processing tasks, while reducing the total cost of ownership through improved resource utilization and lower infrastructure requirements.

Figure 1: AMX Architecture showing AMX tile registers, processing units, and data flow within CPU core

Note: AMX operations, including tile setup and memory-to-tile data movement (which are handled automatically by the system), introduce small overhead that may outweigh benefits for smaller models or single-batch processing where insufficient matrix operations cannot amortize these costs, making batch size optimization critical for performance gains.

When to choose CPU inference with AMX

CPU inference with AMX acceleration benefits workloads including:

Batch processing and traditional ML: Content summarization, recommendation systems, and analytical workloads benefit from CPU’s cost efficiency and ability to handle sparse data structures and branching logic.

Small to medium-sized models: Models under 7B parameters and batch sizes of 8-16 samples achieve excellent performance through optimized threading, making CPUs ideal for applications like fraud detection and chatbots.

Variable demand workloads: E-commerce systems and applications with unpredictable traffic patterns can quickly scale CPU instances up or down based on demand, avoiding the fixed costs of dedicated accelerator hardware that sits idle during low-traffic periods.

Complex business logic: Applications like financial risk assessment and content moderation that need to combine ML predictions with business rules and conditional logic work well on CPUs, which handle mixed workloads better than specialized accelerators.

Implementation: AMX optimization with PyTorch

PyTorch, a popular open-source machine learning framework, includes built-in Intel optimizations through oneDNN (Intel’s Deep Neural Network library) that automatically use AMX when available. Setup requires installing dependencies and configuring environment variables for optimal performance.

Install dependencies

# Install transformers and torch
pip install torch transformers

Configure environment variables

These environment variables tell oneDNN library how to optimize your inference workload for AMX.

Enable AMX instruction set (tells oneDNN to use AMX tiles for matrix operations):
```
export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
```
Optimize thread affinity (binds threads to CPU cores for better cache performance):
```
export KMP_AFFINITY=granularity=fine,compact,1,0
```
Use all available CPU cores for parallel processing:
```
export OMP_NUM_THREADS=$(nproc)
```
Cache compiled kernels (avoids recompilation overhead on subsequent runs):
```
export ONEDNN_PRIMITIVE_CACHE_CAPACITY=4096
```
Set default precision to BF16 (enables automatic AMX acceleration):
```
export ONEDNN_DEFAULT_FPMATH_MODE=bf16
```
(Optional) Enable verbose logging to verify AMX activation:
```
export ONEDNN_VERBOSE=1
```

BF16 optimization example

With environment variables configured, implementing BF16 optimization requires minimal to no code changes. The following example demonstrates how PyTorch automatically leverages AMX tile registers for matrix operations when BF16 precision is used.

Note: This is a simplified example for demonstration purposes; adapt the code to your specific use case and requirements.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
import time

# Load model and tokenizer from HuggingFace
model_name = "google/gemma-3-1b-it"

model_revision = "dcc83ea841ab6100d6b47a070329e1ba4cf78752"
tokenizer = AutoTokenizer.from_pretrained(
    model_name,
    revision=model_revision
)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    revision=model_revision
)
# Fix tokenizer padding issue for batch processing
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# Enable BF16 precision for automatic AMX acceleration
model = model.to(dtype=torch.bfloat16)
model.eval()  # Set to inference mode

# Inference function with BF16 autocast
def run_optimized_inference(prompts):
    inputs = tokenizer(prompts, padding=True, 
                      return_tensors="pt")  # Tokenize input
    
    with torch.no_grad():  # Disable gradients for inference
        with torch.amp.autocast('cpu',
                               dtype=torch.bfloat16):  # BF16 autocast
            outputs = model.generate(
                **inputs,
                max_length=100,     # Set maximum sequence length 
                do_sample=False     # Use greedy decoding
            )
    return outputs

# Example usage with performance measurement
prompts = ["What are the benefits of cloud computing?"]
start_time = time.time()
results = run_optimized_inference(prompts)  # Run BF16-optimized inference
elapsed_time = time.time() - start_time
tokens_generated = len(results[0]) - len(tokenizer.encode(
    prompts[0]))  # Count new tokens

# Display results and performance metrics
print(tokenizer.decode(results[0], skip_special_tokens=True))
print(f"Latency: {elapsed_time*1000:.1f}ms, "
      f"Throughput: {tokens_generated/elapsed_time:.1f} "
      f"tokens/sec")

Performance benchmarks

To validate AMX performance benefits, we conducted benchmarks across multiple popular language models representing different use cases and model sizes.

Benchmarking methodology and environment

We tested two improvements: hardware generation advances (m8i vs m7i) and AMX optimization impact (FP32 vs BF16). This shows you both upgrade paths for your workloads.

Models tested: BigBird-RoBERTa-large (355M), Microsoft DialoGPT-large (762M), Google Gemma-3-1b-it (1B), DeepSeek-R1-Distill-Qwen-1.5B (1.5B), Llama-3.2-3B-Instruct (3B), YOLOv5 (tested with 30 images at ~1200×800 resolution with 5 iterations for each image)
Amazon EC2 instance types: m8i.4xlarge, m7i.4xlarge (8^th & 7^th gen general-purpose Amazon EC2 instances with 16 vCPUs and 64 GiB memory, both AMX-capable)
Batch sizes: 1, 8, 32 (number of input samples processed simultaneously in a single inference call)
Iterations: 5 runs per configuration
Comparison types:
- Instance generation comparison (m8i vs m7i performance)
- AMX optimization impact (32-bit floating-point (FP32) vs Brain Floating Point 16 (BF16) on same instance)
Optimizations: FP32 baseline vs BF16 AMX
Framework: PyTorch 2.8.0 (which has built-in Intel optimizations)
Region: AWS us-west-2
Measurement methodology: In our benchmarks, ‘inference latency’ represents the complete model inference execution time including input tokenization and full sequence generation (for generative models) or complete forward pass (for non-generative models). Each measurement is the average of 5 iterations after warm-up iterations, excluding model loading time. We use this metric because AMX’s matrix multiplication acceleration improves performance throughout the complete forward pass.

Note: Throughout this blog, FP32 refers to the default 32-bit floating-point precision, while BF16 refers to Brain Floating Point 16-bit precision with AMX acceleration enabled.

Disclaimer: Performance results are based on internal testing and may vary depending on specific workloads, configurations, and environments.

Detailed result: BigBird-RoBERTa-large

This benchmark represents document classification, content summarization, and text analysis workloads typical in batch processing where high throughput is desirable and offline inference scenarios where strict latency requirements are not critical.

Figure 2: m7i.4xlarge vs m8i.4xlarge inference latency comparison for model BigBird-RoBERTa-large (355M parameters)

Figure 3: m7i.4xlarge vs m8i.4xlarge throughput comparison for BigBird-RoBERTa-large model across batch sizes 1, 8, and 32

Figure 4: FP32 vs BF16 inference latency comparison for model BigBird-RoBERTa-large (355M parameters) on m7i.4xlarge and m8i.4xlarge instances across batch sizes

BigBird-RoBERTa-large model benchmarking demonstrates three key performance improvements. Figure 2 shows m8i hardware delivers 4-20% latency reduction across batch sizes compared to m7i for both FP32 and BF16 with AMX, providing immediate benefits without application changes. With AMX and BF16, performance gains decrease at higher batch sizes as AMX overhead exceeds benefits for smaller models like BigBird-RoBERTa-large. Figure 3 validates these improvements with corresponding 4-25% throughput gains, enabling better resource utilization for production applications. Figure 4 demonstrates that enabling AMX with BF16 optimization provides the most significant impact, reducing m8i latency by 55-67% compared to non-AMX FP32 baseline, enabling 2-3x higher processing capacity and reduced compute costs.

The analysis above demonstrates the methodology for interpreting benchmark results using BigBird-RoBERTa-large as a representative example. The remaining models (DialoGPT-large, Gemma-3-1b-it, DeepSeek-R1-Distill-Qwen-1.5B, and Llama-3.2-3B-Instruct) follow identical testing procedures and exhibit similar performance patterns, with variations primarily in the magnitude of improvements based on model size and architecture. The comprehensive analysis of five models and their performance implications are synthesized in the following section.

Benchmarking result for additional models

To validate AMX’s effectiveness across diverse AI workloads, we benchmarked five additional models representing different use cases and model sizes. Each model follows the same testing methodology described above, with performance patterns showing how AMX benefits vary based on model architecture, parameter count, and batch size.

DialoGPT-large (762M) – Conversational AI

This benchmark represents conversational AI, chatbots, and real-time dialogue systems where low latency and consistent response times are critical for user experience.

Figure 5: m7i.4xlarge vs m8i.4xlarge inference latency comparison for model DialoGPT-large (762M parameters)

Figure 6: m7i.4xlarge vs m8i.4xlarge throughput comparison for DialoGPT-large model across batch sizes 1, 8, and 32

Figure 7: FP32 vs BF16 inference latency comparison for model DialoGPT-large (762M parameters) on m7i.4xlarge and m8i.4xlarge instances across batch sizes

Gemma-3-1b-it (1B) – General Purpose

This benchmark represents general-purpose language understanding tasks, content generation, and smaller model deployments suitable for cost-sensitive applications and variable demand workloads.

Figure 8: M7i.4xlarge vs M8i.4xlarge inference latency comparison for model Gemma-3-1b-it (1B parameters)

Figure 9: m7i.4xlarge vs m8i.4xlarge latency and throughput comparison for Gemma-3-1b-it across model batch sizes 1, 8, and 32

Figure 10: FP32 vs BF16 inference latency comparison for model Gemma-3-1b-it (1B parameters) on m7i.4xlarge and m8i.4xlarge instances across batch sizes

DeepSeek-R1-Distill-Qwen-1.5B (1.5B) – Reasoning

This benchmark represents reasoning and analytical workloads, including complex decision-making systems, financial analysis, and applications requiring sophisticated logic processing.

Figure 11: m7i.4xlarge vs m8i.4xlarge inference latency comparison for model DeepSeek-R1-Distill-Qwen-1.5B (1.5B parameters)

Figure 12: m7i.4xlarge vs m8i.4xlarge latency and throughput comparison for DeepSeek-R1-Distill-Qwen-1.5B model across batch sizes 1, 8, and 32

Figure 13: FP32 vs BF16 inference latency comparison for model DeepSeek-R1-Distill-Qwen-1.5B (1.5B parameters) on m7i.4xlarge and m8i.4xlarge instances across batch sizes

Llama-3.2-3B-Instruct (3B) – Large model

This benchmark represents larger model deployments for complex instruction-following tasks, advanced content generation, and applications requiring higher model capacity while maintaining cost efficiency.

Figure 14: m7i.4xlarge vs m8i.4xlarge inference latency comparison for model Llama-3.2-3B-Instruct (3B parameters)

Figure 15: m7i.4xlarge vs m8i.4xlarge latency and throughput comparison for Llama-3.2-3B-Instruct model across batch sizes 1, 8, and 32

Figure 16: FP32 vs BF16 inference latency comparison for model Llama-3.2-3B-Instruct (3B parameters) on m7i.4xlarge and m8i.4xlarge instances across batch sizes

Yolov5 – Computer vision model

This benchmark represents computer vision workloads including object detection, image classification, and real-time video processing applications where consistent throughput is important for production deployments.

Instance type	Inference latency in Sec (Processing time per image)		Throughput (Image processed per sec)
Instance type	FP32	BF16	FP32	BF16
m8i.4xlarge	0.034	0.029	29.23	34.63
m7i.4xlarge	0.038	0.031	26.39	32.28
m8i improvement	10.5%	6.5%	10.8%	7.3%

Key insights: m8i instances deliver 7-11% better performance than m7i across both precision formats. Combining hardware upgrade with AMX optimization, m8i with BF16 delivers up to 24% lower latency and 31% higher throughput compared to m7i with FP32.

Benchmark result summary

The detailed graphs above demonstrate consistent performance patterns across tested models. Key findings:

M8i vs M7i instance performance

m8i instances deliver 9-14% average and up to 20% better performance than m7i across the tested models through hardware advances: up to 4.6x larger L3 cache, higher base frequencies, up to 2.5x higher DDR5 bandwidth, and enhanced AMX execution with FP16 support.

Model	Use Case	m8i average latency improvement*
BigBird-RoBERTa-large (355M)	Document analysis	10%
DialoGPT-large (762M)	Conversational AI	14%
Gemma-3-1b-it (1B)	General purpose	10%
DeepSeek-R1 (1.5B)	Reasoning tasks	11%
Llama-3.2-3B (3B)	Large model deployment	12%
YOLOv5	Computer vision	9%

* Average across all tested configurations (FP32 and BF16 at batch sizes 1, 8, and 32)

AMX acceleration impact (FP32 vs BF16)

BF16 precision with AMX delivers 21-72% performance improvements at batch sizes of 8 and above compared to FP32 baseline on the same instance type. These results compare FP32 vs BF16 performance on m8i.4xlarge, with performance gains varying by model size and batch configuration. Larger batch sizes show greater AMX benefits.

Model	Latency improvement (%)
Model	Batch 1	Batch 8	Batch 32
BigBird-RoBERTa-large	55	67	63
DialoGPT-large	– 44*	21	30
Gemma-3-1b-it	6	42	24
DeepSeek-R1	24	68	59
Llama-3.2-3B	27	72	68

* At batch size 1, DialoGPT-large’s autoregressive decoding generates tokens sequentially, producing many small matrix operations where AMX tile setup overhead exceeds the acceleration benefit. At batch sizes 8 and above, multiple sequences are processed in parallel, creating larger matrix operations that amortize this overhead and deliver 21-30% improvement.

Performance patterns by batch size

Larger models (1B+ parameters) show consistently better AMX performance across the tested batch sizes:

Batch size 1: Mixed results – larger models show 6-27% improvement, smaller models may experience AMX overhead
Batch size 8: Strong performance gains of 21-72% across the tested models, with larger models showing greater benefits
Batch size 32: Significant improvements of 24-68% for most models, demonstrating AMX’s batch processing strength

Batch size optimization guidelines

AMX performance scales with batch size, with optimal range varies by model size. Performance saturates beyond batch 16 due to hardware limits including memory bandwidth and compute bottlenecks.

Model Size	Performance Gain	Recommended Batch Size	Notes
<1B parameters	21-67%	8-32	Batch 1 results vary by architecture*
1-2B parameters	42-68%	4-16	6-24% gains even at batch 1
3B+ parameters	27-72%	1-8	Benefits across batch sizes

* Encoder models (BigBird) show 55% gains at batch 1; autoregressive models (DialoGPT) may experience overhead.

Combined performance benefits

When we combine AMX optimization with 8th generation instances (m8i), the performance improvements compound significantly. For example, Llama-3.2-3B-Instruct running with BF16 AMX on m8i instances can achieve up to 76% better performance compared to FP32 inference on m7i instances at optimal batch sizes (batch 8: m7i FP32 45.51s vs m8i BF16 10.93s = 76% improvement; batch 32: m7i FP32 62.60s vs m8i BF16 17.47s = 72% improvement).

Throughput scaling

Across the tested models, throughput (tokens/sec) increases proportionally with latency reduction. This consistent relationship demonstrates that AMX optimizations translate directly to improved inference efficiency.

Price-Performance Analysis: Gemma-3-1b-it Model

While m8i.4xlarge instances are priced slightly higher than m7i.4xlarge ($0.847 vs $0.806 per hour in us-west-2), they deliver superior price-performance. To illustrate the economic benefits, we analyzed cost per 1 million tokens using Gemma-3-1b-it as a representative example. M8i delivers up to 13% better price-performance over m7i through hardware generation advances, with both instances running BF16 AMX.

Batch Size	Data Type	m7i.4xlarge		m8i.4xlarge		Price-Performance improvement
Batch Size	Data Type	Throughput (tokens/sec)	$ per 1M token	Throughput (tokens/sec)	$ per 1M token	Price-Performance improvement
1	BF16(AMX)	14.3	$15.66	17.2	$13.67	13%
8	BF16(AMX)	71	$3.16	82.3	$2.86	9%
32	BF16(AMX)	119.1	$1.88	127.8	$1.84	2%

Combining the hardware upgrade with BF16 AMX optimization delivers up to 44% better price-performance compared to FP32 on m7i.

Batch Size	m8i.4xlarge			m7i.4xlarge			Price-Performance improvement
Batch Size	Data Type	Throughput (tokens/sec)	$ per 1M token	Data Type	Throughput (tokens/sec)	$ per 1M token	Price-Performance improvement
1	BF16(AMX)	17.2	$13.67	FP32	14.9	$15.03	9%
8	BF16(AMX)	82.3	$2.86	FP32	44.1	$5.08	44%
32	BF16(AMX)	127.8	$1.84	FP32	89.2	$2.51	27%

Key findings from the price-performance analysis:

Combined optimization delivers up to 44% better price-performance: m8i with AMX and BF16 outperforms m7i with FP32 at batch size 8 – consistent with our batch size optimization guidelines where batch sizes of 4-16 deliver optimal results for 1B models like Gemma-3-1b-it, achieving $2.86 per 1M tokens for applications like chatbots and fraud detection.
Larger batches maximize cost efficiency: Batch size 32 reduces costs further to $1.84 per 1M tokens, a 27% improvement over m7i FP32 – ideal for throughput-oriented workloads like content summarization and recommendation systems where latency requirements are flexible.

Production deployment recommendation

BF16 AMX: Delivers 21-72% performance improvements at recommended batch sizes while maintaining model accuracy, making it suitable for production workloads including fraud detection systems, content moderation, and real-time recommendation engines
Batch processing: Target batch sizes of 4-16 based on your use case – smaller batches (1-4) for latency-sensitive applications like chatbots, larger batches (8-16) for throughput-focused scenarios like document analysis and offline processing
Instance selection: m8i instances provide consistent 9-14% performance improvements over m7i, delivering immediate ROI for existing CPU inference workloads without requiring application changes
Model size consideration: Larger models (1B+ parameters) show better AMX utilization across batch sizes, making them ideal candidates for m8i deployment in complex reasoning and content generation applications

Conclusion and next steps

By using Intel AMX on Amazon EC2 8th generation instances, you can achieve substantial performance improvements for AI inference workloads. Our benchmarks demonstrate up to 72% performance improvements across popular language models, making CPU inference more competitive for batch processing, real-time applications, recommender systems, and variable demand workloads while delivering substantial cost savings through improved resource utilization.

Key takeaways:

BF16 AMX optimization delivers up to 72% performance improvements across model sizes, with batch 8 showing 21-72% gains and batch 32 showing 24-68% gains
Batch sizes of 4-8 provide optimal performance for most models—DialoGPT achieves 21% improvement in latency at batch 8, while Llama-3.2-3B achieves 72% improvement
8th generation instances deliver up to 14% performance improvements over m7i across the tested workloads
Combined optimizations (m8i + BF16 AMX) can achieve compound performance improvements up to 76% in optimal configurations (vs m7i FP32), making CPU inference highly competitive for cost-sensitive applications
M8i instances deliver up to 13% better price-performance vs m7i (lower cost per 1M tokens), based on our analysis of the Gemma-3-1b-it model
Proper environment configuration is critical for AMX activation

You can implement these optimizations immediately. AMX hardware acceleration combined with PyTorch’s Intel-specific enhancements requires configuring environment variables while delivering substantial speed gains. Begin with BF16 optimization on your existing models, then explore INT8 quantization for additional gains.

Next steps:

Launch an Intel based Amazon EC2 8th generation instance (m8i.4xlarge)
Install PyTorch (includes built-in Intel optimizations)
Configure AMX environment variables
Measure performance improvements
Scale your optimized inference workloads

Additional resources

Build high-performance apps with AWS Lambda Managed Instances

Debasis Rath — Mon, 30 Mar 2026 14:53:01 +0000

High-performance applications such as CPU-intensive processing, memory-heavy analytics, and steady-state data pipelines often require more predictable compute resources than standard AWS Lambda configurations provide. AWS Lambda Managed Instances (LMI) addresses this by letting you run Lambda functions on selected Amazon EC2 instance types while preserving the Lambda programming model. You can choose over 400 Amazon Elastic Compute Cloud (Amazon EC2) instance types from general purpose, compute optimized, or memory optimized instance families to match workload requirements. AWS Lambda continues to manage infrastructure operations such as instance lifecycle management, operating system patching, runtime updates, request routing, and automatic scaling. This approach gives your teams greater control over compute characteristics, EC2 pricing model and reduces operational overhead of managing servers or clusters.

In this post, you will learn how to configure AWS Lambda Managed Instances by creating a Capacity Provider that defines your compute infrastructure, associating your Lambda function with that provider, and publishing a function version to provision the execution environments. We will conclude with production best practices including scaling strategies, thread safety, and observability for reliable performance.

Figure 1. Creating Function on LMI

Creating Capacity Providers

A Capacity Provider defines the infrastructure blueprint for running LMI functions on Amazon EC2. It specifies instance types, network placement, and scaling behavior. To create a Capacity Provider, you need two parameters: an IAM role (Capacity Provider Operator Role) granting Lambda permissions to launch and manage instances and your VPC configuration with subnets and security groups. Create this role in your account with the AWSLambdaManagedEC2ResourceOperator managed policy following the Principle of Least Privilege (granting only the minimum permissions necessary).

This command creates a Capacity Provider with instance types and scaling configuration:

aws lambda create-capacity-provider \
  --capacity-provider-name my-lmi-capacity \
  --vpc-config SubnetIds=subnet-abc123,subnet-def456,SecurityGroupIds=sg-xyz789 \
  --permissions-config CapacityProviderOperatorRoleArn=arn:aws:iam::123456789012:role/LMIOperatorRole \
  --instance-requirements Architectures=x86_64,AllowedInstanceTypes=c5.2xlarge,r5.4xlarge \
  --capacity-provider-scaling-config MaxVCpuCount=50,ScalingMode=Auto \
  --region us-east-1

This command returns a Capacity Provider ARN that you’ll use to create your LMI function. Your functions behavior depends on four main configurations in the capacity provider:

Instance selection

Lambda currently supports three Amazon EC2 instance families (.large and up): C (compute optimized) for CPU-heavy work, M (general purpose) for balanced workloads, and R (memory optimized) for large datasets. Choose x86 (Intel/AMD) or ARM (Graviton) architectures. If you don’t specify instance types, Lambda defaults to appropriate instances based on your function’s memory and CPU configuration. This is the recommended starting point unless you have specific performance requirements. When you need more control, use AllowedInstanceTypes to specify only the instance types that Lambda can use or use ExcludedInstanceTypes to exclude specific types while allowing all other instance types. You can’t use both parameters together.

VPC and networking

Configure multiple subnets across Availability Zones. Lambda creates a minimum Amazon EC2 fleet of three instances distributed across your configured Availability Zones to maintain availability and resiliency. Egress traffic from functions, including Amazon CloudWatch Logs, transits through the Amazon EC2 instance’s network interface in your Amazon Virtual Private Cloud (Amazon VPC). As functions send logs and metrics to CloudWatch, you will need internet access through a NAT Gateway or VPC endpoints with AWS PrivateLink for Amazon CloudWatch. This only affects egress traffic; function invoke requests don’t flow through your VPC. Security groups attached to your instances should allow only the traffic your function code needs. With LMI, configure VPC once at the Capacity Provider level instead of per function, simplifying management for multiple LMI functions. Standard Lambda functions continue to use their own VPC configurations. This Capacity Provider VPC configuration applies only to LMI functions.

Figure 2. LMI Networking

Scaling configuration

Set MaxVCpuCount to cap compute capacity and control costs. New invocations throttle when you reach this limit until capacity frees up. Lambda monitors CPU utilization and scales instances automatically. Choose automatic scaling mode where Lambda tunes thresholds based on load patterns, or manual mode where you set a target CPU utilization percentage. Multiple functions can share the same Capacity Provider to reduce costs through better resource utilization, though you might want separate providers for functions with different performance or isolation requirements.

Security

Lambda encrypts Amazon Elastic Block Store (Amazon EBS) volumes attached to EC2 instances with a service-managed key by default. You can provide your own AWS Key Management Service (AWS KMS) key for encryption. Place instances in private subnets with restrictive security groups for enhanced security.

Creating Lambda Managed Instance Functions

You create an LMI function similarly to creating a standard Lambda function. You package your code, set your runtime, assign an execution role, and configure memory. The difference is specifying a CapacityProviderConfig to tell Lambda which Capacity Provider to use and how to size each execution environment. Specify CapacityProviderConfig during function creation with the Capacity Provider ARN and configure two execution environment settings. ExecutionEnvironmentMemoryGiBPerVCpu sets the memory-to-vCPU ratio (2:1, 4:1, or 8:1) based on your workload type and PerExecutionEnvironmentMaxConcurrency defines how many concurrent requests share each execution environment. This table shows how memory and vCPU allocation maps across supported execution environment ratio.

2:1 Ratio(Compute optimized)		4:1 Ratio(General purpose)		8:1 Ratio(Memory optimized)
Memory (GB)	vCPU(s)	Memory (GB)	vCPU(s)	Memory (GB)	vCPU(s)
2	1	4	1	8	1
4	2	8	2	16	2
6	3	12	3	24	3
8	4	16	4	32	4
10	5	20	5
12	6	24	6
14	7	28	7
16	8	32	8
…	…
32	16

Function Memory-to-CPU configuration

Set the function’s memory size (up to 32 GB for LMI) and ExecutionEnvironmentMemoryGiBPerVCpu ratio. The default ratio is 2:1. A 2:1 ratio map to compute optimized instances for CPU-intensive tasks like video encoding, 4:1 map to a general purpose for balanced workloads, and 8:1 maps to a memory optimized instances for large in-memory datasets or caching. You must set memory in multiples of the ratio. LMI requires a 2 GB minimum as execution environments need sufficient memory to handle multiple concurrent requests. LMI supports up to 32 GB memory per execution environment.

Multi-Concurrency settings

LMI supports multiple concurrent invocations sharing the same execution environment, reducing cost per invocation by maximizing vCPU utilization. This is particularly effective for I/O-bound workloads, where invocations waiting on database queries or API calls yield vCPU usage to other invocations during idle periods. Lambda defaults to max concurrency per execution environment based on your runtime: Node.js (64 per vCPU), Java, and .NET (32 per vCPU), Python (16 per vCPU). Use PerExecutionEnvironmentMaxConcurrency to set a lower limit based on your workload’s resource needs. Decrease it if you’re experiencing memory pressure or CPU contention. When environments reach their configured max concurrency, new invocations throttle until capacity frees up at the execution environment level. This table captures the maximum concurrency per vCPU for each supported programming language.

Language	Default Max Concurrency
Node.js	64 per vCPU
Java	32 per vCPU
.NET	32 per vCPU
Python	16 per vCPU

This command creates a Lambda function and associates it with your Capacity Provider:

aws lambda create-function \
  --function-name my-lmi-function \
  --runtime python3.13 \
  --role arn:aws:iam::123456789012:role/LambdaExecutionRole \
  --handler app.lambda_handler \
  --zip-file fileb://function.zip \
  --memory-size 4096 \
  --capacity-provider-config '{
    "LambdaManagedInstancesCapacityProviderConfig": {
      "CapacityProviderArn": "arn:aws:lambda:us-east-1:123456789012:capacity-provider:my-lmi-capacity",
      "ExecutionEnvironmentMemoryGiBPerVCpu": 4.0,
      "PerExecutionEnvironmentMaxConcurrency": 10
    }
  }' \
  --region us-east-1

Publishing Lambda Managed Instance Functions

Important: publish a function version before invoking an LMI function. Publishing triggers Lambda to provision Amazon EC2 instances and initialize execution environments, so that the configured baseline capacity is ready before you start invoking. Expect a brief delay before your code goes live as Lambda provisions and launches Amazon EC2 instances. With LMI, execution environments pre-warm after publishing and remain invoke-ready, without cold starts for published versions. Standard Lambda environments initialize on first invoke (cold starts).

This command publishes a Lambda function version and provisions capacity:

aws lambda publish-version --function-name my-lmi-function \
--region us-east-1

After publishing, the function works with standard invocation methods including direct invokes, event source mappings, and service integrations with Amazon API Gateway, Amazon Simple Storage Service (Amazon S3), Amazon DynamoDB Streams, and Amazon EventBridge.

Figure 3. LMI Invocation from event sources

Scaling LMI Functions

Lambda monitors CPU utilization at Capacity Provider level. When CPU utilization reaches the target threshold, Lambda automatically provisions additional EC2 instances, and creates more execution environments on those instances, up to the MaxVCpuCount limit you configured for your capacity provider. As demand decreases, Lambda consolidates workloads onto fewer EC2 instances. You can choose automatic scaling mode (Lambda adjusts thresholds based on your patterns) or manual mode (you set a target CPU percentage). Automatic mode works for variable traffic patterns or when getting started. Manual mode fits when you have predictable patterns and want precise control over scaling thresholds for cost optimization.

Min and max execution environments

Control scaling at the function level with min and max execution environments. The default minimum is 3 execution environments to maintain high availability across Availability Zones. Your total function concurrency equals the number of execution environments multiplied by PerExecutionEnvironmentMaxConcurrency. For example, with min set to 3 and PerExecutionEnvironmentMaxConcurrency of 10, you have provided capacity for 30 concurrent invocations. With max set to 20, you can scale up to 200 concurrent invocations with incoming traffic, based on CPU utilization or concurrency saturation per execution environment. Set max to cap total concurrency and prevent noisy neighbor issues when multiple functions share a Capacity Provider. LMI maintains a minimum number of execution environments with a minimum Amazon EC2 fleet, while standard Lambda scales to zero when idle. Set both min and max to 0 to deactivate a function without deleting it.

Figure 4. LMI Scaling

This command updates the minimum and maximum execution environments for your function:

aws lambda put-function-scaling-config \
  --function-name my-lmi-function \
  --qualifier $LATEST \
  --function-scaling-config MinExecutionEnvironments=5,MaxExecutionEnvironments=20 \
  --region us-east-1

We’ll cover scaling patterns and throughput optimization strategies in depth in a separate blog post.

Best Practices and Production Considerations

Thread Safety

Since LMI supports multiple invocations sharing execution environments, your code must be thread-safe. Code that isn’t thread-safe causes data corruption, security issues, or unpredictable behavior under concurrent load.

Thread safety essentials

Avoid mutating shared objects or global variables. Use thread-local storage for request-specific data. Initialize shared clients (AWS SDK, database connections) outside the function handler and verify that configurations remain immutable during invocations. Write to /tmp using request-specific file names to prevent concurrent writes.

Runtime-specific guidance

Java applications should use immutable objects, thread-safe collections, and proper synchronization. Node.js applications should use async context for request isolation. Python applications run separate processes per execution environment. So, focus on interprocess coordination and file locking for /tmp access.

Workload Optimization

I/O-bound workloads perform better with higher concurrency per environment. Use asynchronous patterns and non-blocking I/O to maximize efficiency. CPU-bound workloads get no benefit from concurrency greater than one per vCPU. Instead, configure more vCPUs per function for true parallelism for compute-heavy tasks like data transformation or image processing.

Testing

Validate your code under concurrent execution. Test with multiple simultaneous invocations to detect race conditions and shared state issues before production deployment. You can use LocalStack for local emulation of LMI. Learn more about LocalStack’s LMI support in their announcement blog.

Compatibility

Tools like Powertools for AWS work with LMI without code changes. However, if you’re reusing existing Lambda function code, layers, or packaged dependencies on LMI, test for thread safety and compatibility with the multi-concurrent execution model before production deployment.

Observability

LMI automatically publishes CloudWatch metrics at two levels: capacity provider (CPU, memory, network, and disk utilization across your Amazon EC2 fleet) and execution environment (concurrency, CPU, and memory per function). Monitor CPUUtilization to understand scaling headroom and right-size your MaxVCpuCount. Track ExecutionEnvironmentConcurrency against ExecutionEnvironmentConcurrencyLimit to catch throttling before it impacts users. Lambda publishes metrics at 5-minute intervals. Use CloudWatch alarms to stay ahead of capacity limits in production.

Conclusion

AWS Lambda Managed Instances combines serverless simplicity with compute flexibility, helping you run high-performance workloads with reduced operational complexity. You maintain the familiar programming model of Lambda while accessing the diverse instance types of Amazon EC2 and predictable pricing, making it well-suited for data processing pipelines, compute intensive operations and cost-sensitive steady-state applications.

Ready to get started with LMI? Deploy our Monte Carlo risk simulation example from GitHub to see LMI in action with a real compute-intensive workload. The sample includes complete infrastructure code and walks you through capacity provider configuration, function setup, and performance optimization.

We want to hear from you. Share your feedback, questions, and use cases on re:Post.

Enhancing auto scaling resilience by tracking worker utilization metrics

Brian Moore — Tue, 24 Mar 2026 16:17:58 +0000

A resilient auto scaling policy requires metrics that correlate with application utilization, which may not be tied to system resources. Traditionally, auto scaling policies track system resource such as CPU utilization. These metrics are easily available, but they only work when resource consumption correlates with worker capacity. Factors such as high variance in request processing time, mixed instance types, or natural changes in application behavior over time can break this assumption.

Worker utilization tracking offers an alternative approach. Using a combination of total worker slots, work in flight, and work waiting in the backlog, a utilization value can be calculated for use in an auto scaling policy. This approach remains accurate across fleets with mixed instance types, applications with variable latencies, and requires no changes as your application evolves.

The limitations of resource-based scaling

Traditional auto scaling policies track system resource metrics like CPU utilization, assuming a direct correlation between resource consumption and available application capacity. Consider an application that reads messages from Amazon Simple Queue Service (SQS), processes them, and writes results to Amazon DynamoDB. If this application uses a fixed-size thread pool to process messages, such as 10 worker threads, the application reaches maximum capacity when all threads are busy, regardless of CPU utilization.

In our example, each worker spends most of its time waiting for DynamoDB responses rather than consuming CPU. All 10 threads become occupied handling requests, but CPU utilization stays low. From the perspective of the auto scaling policy, the fleet looks like it has enough capacity because plenty of CPU headroom remains. Meanwhile, new messages accumulate in the SQS queue because no workers are available to process them.

For queue-based workloads, AWS provides guidance to scale based on an acceptable backlog per worker. This is a calculated target based on your application’s average processing latency (queue delay). This works well when processing times are consistent, but breaks down if an application has variable latency characteristics.

Consider an image processing application that initially handles thumbnails taking 500 ms each. Using the traditional guidance with a target latency of 5 seconds you calculate an acceptable backlog of 10 messages per worker and deploy your scaling policy. Over time, the application evolves to also process 4K photos which take 2 seconds each. Eventually 4K photos are 50% of your traffic and total latency for queued messages has increased to 12.5 seconds, 2.5x more than your initial target.

The scaling policy is no longer fit for its intended purpose because your original latency assumptions no longer reflect reality. To keep this type of scaling effective you must also remember to update your scaling policies as your application behavior evolves.

A shift to using mixed instance types in your application can lead to additional complexity when using traditional resource-based scaling policies. Different instance types may handle the same workload at different CPU levels leading to an unbalanced average that misrepresents your actual application health. By changing your mental model to consider how much work your application can accept instead of how much of a system resource is available you can improve your scaling rules and better model your application’s capacity.

Understanding worker utilization

Worker utilization measures the ratio of active work to available processing capacity. To calculate it, divide total work by total workers.

We use an SQS-based processing application as an example to demonstrate how worker utilization operates, but this approach can also be applied to other applications where work units and worker capacity are measurable. In our example application total work consists of messages waiting to be processed plus messages currently being processed. Amazon CloudWatch provides these values through the ApproximateNumberOfMessagesVisible metric (messages waiting in the queue) and the ApproximateNumberOfMessagesNotVisible metric (messages currently being processed or in flight). Each host in your application should publish the number of available workers as a custom CloudWatch metric with at least a 1-minute period. For Java thread pools or Python multiprocessing pools, this represents the pool or process count. The formula works regardless of the metric period. Using the shortest period possible allows more responsive target tracking and enables Fast Target Tracking if your application has sub-minute data points.

To derive the formula, we can use the following CloudWatch Metric Math expressions:

totalWork = FILL(backlog, REPEAT) + FILL(inFlight, REPEAT)
utilizationRatio = totalWork / workers

Where:

backlog = ApproximateNumberOfMessagesVisible with the Maximum statistic.
inFlight = ApproximateNumberOfMessagesNotVisible with the Maximum statistic.
workers = Your custom TotalWorkers metric with the Sum statistic.

Putting the components together the final expression for your target tracking scaling policy uses the following formula:

IF(FILL(workers, 0) > 0, utilizationRatio, IF(totalWork > 0, 1, 0))

The FILL function uses last known values if SQS metrics are delayed, and the IF statement handles the case where you have no traffic and your fleet scales to zero instances. When there are no available workers, the formula metric reports 1 to indicate that the workers are fully saturated. This prevents the application from getting stuck at zero capacity and not being able to respond to any requests.

In this formula, a value of 1 or higher represents full or over saturation, where all workers are busy with no spare capacity, like running at 100% CPU. Values below 1 indicate available capacity for your application to process more work.

For applications without a measurable backlog metric, you can track worker utilization using only the in-flight work. This approach works for APIs or other synchronous workloads where work arrives and is immediately assigned to workers rather than queuing. In these cases, the formula becomes:

IF(FILL (workers, 0) > 0, utilizationRatio, IF(FILL(inFlight, 0) > 0, 1, 0))

In this scenario the utilization ratio is calculated as follows:

utilizationRatio = FILL(inFlight, REPEAT) / workers

The definitions of workers and inFlight remain the same for this formula. The primary difference is that the ratio directly tracks workers available and does not consider the backlog as an option.

How worker utilization prevents outages

Worker utilization-based scaling works for any application that can define available workers and total work. When the ratio of total work to available workers exceeds your threshold, the system scales out. This approach measures whether workers are available to handle the workload and treats application bottlenecks consistently. Whether workers are waiting on network I/O, performing CPU-intensive calculations, or experiencing another bottleneck doesn’t matter; the only question is whether total work exceeds available worker capacity. Any situation causing messages to accumulate on the queue increases the utilization ratio and triggers scale-out.

Implementing worker utilization scaling

To set up worker utilization-based auto scaling, identify metrics to use in the formula discussed earlier. First, identify a metric to track the amount of work being worked on. For SQS-based processing, AWS provides this metric. Second, implement a custom metric from your application representing the total workers. Optionally you can also identify a metric to track the available backlog of work.

Using CloudWatch metric math, you calculate the utilization metric and use it in a target tracking scaling policy. Here is an example AWS CloudFormation snippet showing the metric math configuration for a Amazon EC2 Auto Scaling group. This snippet shows only the scaling policy configuration and is only an example, before using in production fully test with your application. Your complete template also needs IAM roles with appropriate permissions for SQS, DynamoDB, and CloudWatch access.

ScalingPolicy: 
  Type: AWS::AutoScaling::ScalingPolicy 
  Properties: 
    AutoScalingGroupName: !Ref AutoScalingGroup 
    PolicyType: TargetTrackingScaling 
    TargetTrackingConfiguration: 
      TargetValue: 0.7 
      CustomizedMetricSpecification: 
        Metrics: 
          - Id: backlog 
            MetricStat: 
            Metric: 
              Namespace: AWS/SQS 
              MetricName: ApproximateNumberOfMessagesVisible 
              Dimensions: 
                - Name: QueueName 
                  Value: !GetAtt ProcessingQueue.QueueName 
              Stat: Maximum 
          - Id: inFlight 
            MetricStat: 
            Metric: 
              Namespace: AWS/SQS 
              MetricName: ApproximateNumberOfMessagesNotVisible 
              Dimensions: 
                - Name: QueueName 
                  Value: !GetAtt ProcessingQueue.QueueName 
              Stat: Maximum 
          - Id: workers 
            MetricStat: 
            Metric: 
              Namespace: YourApp 
              MetricName: TotalWorkers 
            Stat: Sum 
          - Id: totalWork 
            Expression: FILL(backlog, REPEAT) + FILL(inFlight, REPEAT) 
          - Id: utilizationRatio 
            Expression: totalWork / workers 
          - Id: utilization 
            Expression: IF(FILL(workers, 0) > 0, utilizationRatio, IF(totalWork > 0, 1, 0)) 
            ReturnData: true

This approach also works for Amazon ECS services using AWS Application Auto Scaling. The metric math configuration remains the same, but you create an AWS::ApplicationAutoScaling::ScalingPolicy resource instead, adapting the parameters accordingly.

Choosing a target utilization

Since the worker utilization metric directly tracks the available capacity of your application, the target utilization value you choose reflects your organization’s balance between cost efficiency and availability. Lower target values provide more headroom for traffic spikes and faster response to load changes but result in higher infrastructure costs due to lower utilization. Higher target values maximize cost efficiency by keeping workers busy but leave less headroom for sudden traffic increases.

When choosing a target consider traffic patterns, acceptable latency during scale-out events, and cost sensitivity. Applications with unpredictable traffic spikes may benefit from lower targets, while an application with predictable load can safely use higher targets. Start with a moderate value like 0.7 and adjust based on observed behavior and your business requirements. If you previously tracked a resource utilization metric such as CPU, consider starting with the same target.

Monitoring resource utilization for cost optimization

While worker utilization drives scaling decisions, CPU and latency should be regularly evaluated to ensure cost-effective operations. Resource-based metrics can identify host resizing opportunities to better match your application requirements. If no scale-in happens when CPU utilization is consistently low, you are likely running instances that are too large for your workload. By using worker utilization in an auto scaling policy, you can switch to a different instance type without adjusting the auto scaling policy. The formula automatically adapts as you add different instance types or update the capacity per worker.

Conversely, if CPU utilization is consistently high while worker utilization remains at your target, your instances might be undersized. Upgrading to larger instance types can improve per-worker throughput, allowing each worker to process tasks faster. Changes to your auto scaling policy are not needed in this situation either. As messages are processed faster, they spend less time in the in-flight state, and the utilization ratio naturally adjusts.

This approach manages application availability independent of instance size, while resource utilization guides cost optimization. Each can be optimized independently without complex coordination.

Conclusion

Worker utilization-based auto scaling reduces the operational burden of continuously validating your scaling rules as application requirements and infrastructure change. By tracking the ratio of work to workers, your auto scaling policies automatically respond to capacity constraints based on available work. The approach works across workloads with discrete processing units and remains effective when you modify instance configurations or application worker pool sizes.

Implementation requires identifying a metric for available work, publishing a custom metric representing total workers, and using CloudWatch metric math in a target tracking scaling policy. This setup provides resilience that scaling based solely on resource metrics cannot achieve, while maintaining the flexibility to optimize costs and change your instance size without impacting system availability.

To get started:

Identify an application in your environment that uses a worker pool.
Instrument the application to publish worker count metrics.
Configure a scaling policy tracking worker utilization.
Monitor how the system responds to traffic changes and capacity events.

Learn more

To learn more about auto scaling and monitoring, see the following resources:

Best practices for Lambda durable functions using a fraud detection example

Debasis Rath — Mon, 23 Mar 2026 22:04:39 +0000

AWS Lambda durable functions extend the Lambda programming model to build fault-tolerant multi-step applications and AI workflows using familiar programming languages. They preserve progress despite interruptions and execution can suspend for up to one year, for human approvals, scheduled delays, or other external events, without incurring compute charges for on-demand functions.

This post walks through a fraud detection system built with durable functions. It also highlights the best practices that you can apply to your own production workflows, from approval processes to data pipelines to AI agent orchestration. You will learn how to handle concurrent notifications, wait for customer responses, and recover from failures without losing progress. If you are new to durable functions, check out the Introduction to Durable Functions blog post first.

Fraud detection with human-in-the-loop

Consider a credit card fraud detection system, which uses an AI agent to analyze incoming transactions and assign risk scores. For ambiguous cases (medium-risk scores), the system needs human approval before authorizing a transaction. The workflow branches based on risk:

Low risk (score < 3): Authorize immediately
High risk (score ≥ 5): Send to the fraud department immediately
Medium risk (score 3–4): Suspend transaction, send SMS and email to cardholder, wait up to 24 hours for confirmation (wait time is customizable)

Figure 1. Agentic Fraud Detection with durable Lambda functions

With human-in-the-loop workflows, response times can vary from minutes to hours. These delays introduce the need to durably preserve the state without consuming compute resources while waiting. With financial systems, we must also implement idempotency to guard against duplicate messages (invocations) and recover from failures without reprocessing completed work. To address these requirements, developers implement polling patterns with external state stores like Amazon DynamoDB or Amazon Simple Storage Service (Amazon S3) to manage idempotency, pay for idle compute while waiting for callbacks, introduce external orchestration components, or build asynchronous message-driven systems to handle long-processing tasks.

Lambda durable functions provide a new alternative to address these challenges through durable execution, a pattern that uses checkpoints (saved state snapshots) to preserve progress and replays from saved state to recover from failures or resume after waiting. With checkpointing capabilities, you no longer need to pay Lambda compute charges while waiting, whether for callbacks, scheduled delays, or external events. Learn how to implement durable functions using the complete fraud detection implementation at this GitHub repository. You can deploy it to your AWS account and experiment with the code as you read. The repository includes deployment instructions, sample data, and helper functions for testing.

As we walk through the code, we’ll focus on best practices for designing workflows with durable execution and how to apply these patterns correctly in production workflows.

Design steps to be idempotent

Durable execution is designed to preserve progress through checkpoints and replay, but that reliability model means step logic can execute more than once. When steps retry, how do you prevent duplicate actions like charges to the credit card or repeated customer SMS or email notifications?

Durable functions use at-least-once execution by default, executing each step at least one time, potentially more if failures occur. When a step fails, it retries. There are two strategies to design idempotent steps that prevent duplicate side effects: using external API idempotency keys and using the at-most-once step semantics built into durable functions.

Strategy A: External API Idempotency Keys

// Strategy A: Use external API idempotency keys
await context.step(`authorize-${tx.id}`, async () => {
  return payment.charges.create({
    amount: tx.amount,
    currency: 'usd',
    idempotency_key: `tx-${tx.id}`, // Prevents duplicate charges
    description: `Transaction ${tx.id}`
  });
});

Notice the configuration:

idempotency_key in API call: If the step retries, the payment processor recognizes it’s a duplicate request and returns the original result
Defense in depth: Two layers of protection: Lambda checkpointing and external API idempotency

Each layer provides independent protection. If Lambda’s checkpoint fails, the external API prevents duplicate charges. For legacy systems without idempotency support, where it’s critical that an operation is not executed more than once, use at-most-once semantics:

Strategy B: Use At-Most-Once Semantics

For legacy systems without idempotency support, use at-most-once execution, a delivery feature that executes each step zero or one time, never more:

// Strategy B: At-most-once step semantics
await context.step("charge-legacy-system", async () => {
  return await legacyPaymentSystem.charge(tx.amount);
}, {
  semantics: StepSemantics.AtMostOncePerRetry,
  retryStrategy: createRetryStrategy({ maxAttempts: 0 })
});

This checkpoints before step execution, preventing the step from re-execution on retries. The tradeoff? If the step fails, you must decide whether to retry (risking duplicates) or fail the entire workflow.

Use idempotency for critical side effects like payment processing, database writes, external API calls, state transitions, and resource provisioning. Read more about idempotency here.

Prevent duplicate executions with DurableExecutionName

Idempotent steps prevent duplicate side effects within a single execution, but what about duplicate workflow executions running concurrently? For example, duplicate messages in the queue, users clicking “Submit” multiple times in the UI, or the same event arriving via multiple channels like webhook and API. Without protection, each invocation creates a separate durable execution, potentially running the fraud check multiple times, sending duplicate notifications, and creating confusion about which execution is authoritative. Durable functions provide DurableExecutionName to help ensure only one concurrent execution per unique name.

// Invoke fraud detection function with execution name
await lambda.invoke({
  FunctionName: 'fraud-detection',
  InvocationType: 'Event',
  DurableExecutionName: `tx-${transactionId}`,
  Payload: JSON.stringify({
    id: transactionId,
    amount: 6500,
    location: 'New York, NY',
    vendor: 'Amazon.com'
  })
});

Notice the configuration:

DurableExecutionName: tx-${transactionId}: Uses the transaction ID as a unique execution identifier
InvocationType: ‘Event’: Asynchronous invocation supports long-running workflows beyond 15 minutes
One execution per transaction: If three invocations arrive with the same transaction ID, only the first creates an execution. Subsequent requests with the same execution name and payload receive an idempotent response returning the existing execution’s ARN, rather than creating a new execution.

Lambda durable functions work with Lambda event sources, including event source mappings (ESM) such as Amazon Simple Queue Service (Amazon SQS), Amazon Kinesis, and DynamoDB Streams. ESMs invoke durable functions synchronously and inherit Lambda’s 15-minute invocation limit. Therefore, like direct Request/Response invocations, durable functions executions using event source mappings cannot exceed 15 minutes.

For workflows exceeding 15 minutes, use an intermediary Lambda function between the event source mapping and durable function:

// Intermediary function for SQS -> Durable function
export const handler = async (event) => {
  for (const record of event.Records) {
    const transaction = JSON.parse(record.body);
    await lambda.invoke({
      FunctionName: process.env.FRAUD_DETECTION_FUNCTION,
      InvocationType: 'Event',
      DurableExecutionName: `tx-${transaction.id}`,
      Payload: JSON.stringify(transaction)
    });
  }
};

This removes the 15-minute limit, allows executions up to one year, and enables custom execution name parameters for idempotency. Use Powertools for AWS Lambda to prevent duplicate invocations of the durable function when the event source mapping retries the intermediary function. Additionally, configure failure handling for your event source to capture failed invocations for future redrive or replay. For example, dead letter queues for SQS, or on-failure destinations for other event sources.

Match timeouts to invocation type

One important configuration detail ties these patterns together: matching your timeout settings to your invocation type. Lambda synchronous invocations (RequestResponse) have a hard 15-minute timeout limit. If you configure a durable execution to run for 24 hours but invoke it synchronously, the synchronous invocation fails immediately with an exception. Durable functions support workflows up to one year when invoked asynchronously.

// Lambda function configuration
{
  FunctionName: 'fraud-detection',
  Timeout: 300,
  MemorySize: 512,
  DurableConfig: {
    ExecutionTimeout: 90000
  }
}

And invoke asynchronously:

// Async invocation for long-running workflow
await lambda.invoke({
  FunctionName: 'fraud-detection',
  InvocationType: 'Event',
  DurableExecutionName: `tx-${transactionId}`,
  Payload: JSON.stringify(transaction)
});

Notice the configuration:

Timeout: 300: Lambda function timeout (5 minutes in this example, up to a maximum of 15 minutes). This defines the maximum duration for each active execution phase, including the initial invocation and any subsequent replays. Set this to cover the longest expected active processing time in your workflow.
ExecutionTimeout: { hours: 25 }: Durable execution timeout covers the workflow’s expected total duration including suspension periods. Set this slightly above the longest wait timeout to avoid edge cases.
InvocationType: ‘Event’: Asynchronous invocation removes the 15-minute limit and enables executions up to one year.

The Lambda function timeout applies to active execution phases (AI calls, notification sending). During suspension (waiting for callbacks), the function isn’t running, so this timeout doesn’t apply. Setting the durable execution timeout to a meaningful boundary prevents workflows from running longer than expected. Without an explicit timeout, executions can run up to the maximum lifetime of one year.

	Synchronous (RequestResponse)	Asynchronous (Event)
Total duration	Under 15 minutes	Up to 1 year
Caller needs result	Yes	No
Idempotency support	Yes	Yes
Waits with suspension	Yes	Yes

Execute Concurrent Operations with context.parallel()

In the fraud detection workflow, the system notifies the cardholder through multiple channels such as SMS and email. Preserving business logic when executing parallel workflows introduces code complexities such as managing execution state across branches, handling synchronization, and coordinating branch completion. Durable functions simplify parallel workflow implementation using context.parallel(), which executes branches concurrently while maintaining durable checkpoints for each branch and provides configurable options to handle partial completions. By checkpointing and managing the state internally, durable functions help make sure that the state is preserved even if there are retries or failures. Note that context.parallel() manages the internal execution state for each branch. If your branches interact with a shared external state (such as a database), you’re responsible for managing concurrent access to that external state.

// Human-in-the-loop: verify via email AND SMS (first response wins)
let verified = await context.parallel("human-verification", [
  (ctx) => ctx.waitForCallback("SendVerificationEmail",
    async (callbackId) => sendCustomerNotification(callbackId, 'email', tx)
  ),
  (ctx) => ctx.waitForCallback("SendVerificationSMS",
    async (callbackId) => sendCustomerNotification(callbackId, 'sms', tx)
  )
], {
  maxConcurrency: 2,
  completionConfig: {
    minSuccessful: 1 // Continue after 1 success
  }
});

Notice the configuration:

maxConcurrency: 2: Both notifications sent at the same time
minSuccessful: 1: We only need one channel to succeed, whichever responds first wins

Each parallel branch waits for its callback independently, and the durable execution checkpoints each branch as part of the execution state. Using the minSuccessful parameter, you control the minimum number of successful branch executions required for the parallel operation to complete. In this example, only one of the two branches needs to succeed. Verifications through SMS or email are both valid, and the workflow resumes as soon as either channel completes successfully. We call this the first-response-wins pattern. This pattern works well when you only need a single successful result from any parallel branch and want the remaining branches to stop blocking progress.

But what happens if neither channel responds? Without timeouts, this workflow could remain suspended for up to the configured execution lifetime.

Always configure callback timeouts

Let’s add timeout protection to the parallel verification from the previous section. context.waitForCallback() accepts a timeout option that bounds how long each branch waits before throwing an exception. By wrapping the parallel call in a try/catch, you can implement fallback logic when users don’t respond in time.

// Enhanced: parallel verification with timeout and error handling
let verified;
try {
  verified = await context.parallel("human-verification", [
    (ctx) => ctx.waitForCallback("SendVerificationEmail",
      async (callbackId) => sendCustomerNotification(callbackId, 'email', tx),
      { timeout: { days: 1 } }  // Wait up to 1 day for email response
    ),
    (ctx) => ctx.waitForCallback("SendVerificationSMS",
      async (callbackId) => sendCustomerNotification(callbackId, 'sms', tx),
      { timeout: { days: 1 } }  // Wait up to 1 day for SMS response
    )
  ], {
    maxConcurrency: 2,
    completionConfig: {
      minSuccessful: 1
    }
  });
} catch (error) {
  const isTimeout = error.message?.includes("timeout");
  if (isTimeout) {
    context.logger.warn("Customer verification timeout", { error, txId: tx.id });
    // Fallback: escalate to fraud department
    return await context.step("sendToFraudDepartment", async () =>
      sendToFraudDepartment(tx, true)
    );
  }
  throw error; // Re-throw non-timeout errors
}

Notice what changed from the previous section:

timeout: { days: 1 }: Each callback branch now has a maximum wait time of 1 day. If neither the email nor SMS callback arrives within that window, a timeout exception is thrown.
try/catch with timeout detection: The catch block distinguishes between timeout errors and other exceptions. When a timeout occurs, the workflow implements fallback logic by escalating the transaction to the fraud department, while non-timeout errors are re-thrown to be handled by the durable execution retry mechanism.

Without this error handling, the entire execution fails unhandled. The timeout also works with the minSuccessful configuration: if one branch times out but the other succeeds, the parallel operation still completes successfully since only one successful result is required.

For advanced use cases where the callback handler performs long-running work, you can also configure a heartbeatTimeout to detect stalled callbacks before the main timeout expires. See the Lambda Developer Guide for details.

Use callback timeouts for human approvals, external API callbacks, asynchronous processing, and third-party integrations.

Putting it all together: complete fraud detection implementation

Now let’s see how all the best practices work together in the complete fraud detection workflow:

import { withDurableExecution } from "@aws/durable-execution-sdk-js";
import { BedrockAgentCoreClient, InvokeAgentRuntimeCommand } from "@aws-sdk/client-bedrock-agentcore";

const agentRuntimeArn = process.env.AGENT_RUNTIME_ARN;
const agentRegion = process.env.AGENT_REGION || 'us-east-1';
const client = new BedrockAgentCoreClient({ region: agentRegion });

export const handler = withDurableExecution(async (event, context) => {
  const tx = {
    id: event.id,
    amount: event.amount,
    location: event.location,
    vendor: event.vendor
  };

  // AI fraud assessment with error handling
  tx.score = await context.step("fraudCheck", async () => {
    try {
      const payloadJson = JSON.stringify({ input: { amount: tx.amount } });
      const command = new InvokeAgentRuntimeCommand({
        agentRuntimeArn: agentRuntimeArn,
        qualifier: 'DEFAULT',
        payload: Buffer.from(payloadJson, 'utf-8'),
        contentType: 'application/json',
        accept: 'application/json'
      });
      const response = await client.send(command);
      const responseText = await response.response.transformToString();
      const result = JSON.parse(responseText);
      return result?.output?.risk_score ?? 5;  // Default to high-risk if score unavailable
    } catch (error) {
      context.logger.error("Fraud check failed", { error, txId: tx.id });
      return 5;
    }
  });

  // Route based on AI decision
  if (tx.score < 3) {
    // Best Practice: Idempotent authorization
    return await context.step(`authorize-${tx.id}`, async () =>
    authorizeTransaction(tx, { idempotency_key: `tx-${tx.id}` })
    );
  }

  if (tx.score >= 5) {
    return await context.step(`sendToFraudDepartment-${tx.id}`, async () =>
      sendToFraudDepartment(tx)
    );
  }

  // Medium risk: need human verification
  await context.step(`suspend-${tx.id}`, async () => suspendTransaction(tx));

  // Best Practice: Concurrent operations with timeout configuration
  let verified;
  try {
    verified = await context.parallel("human-verification", [
      (ctx) => ctx.waitForCallback("SendVerificationEmail",
        async (callbackId) => sendCustomerNotification(callbackId, 'email', tx),
        { timeout: { days: 1 } }
      ),
      (ctx) => ctx.waitForCallback("SendVerificationSMS",
        async (callbackId) => sendCustomerNotification(callbackId, 'sms', tx),
        { timeout: { days: 1 } }
      )
    ], {
      maxConcurrency: 2,
      completionConfig: {
        minSuccessful: 1
      }
    });
  } catch (error) {
    const isTimeout = error.message?.includes("timeout");
    context.logger.warn(
      isTimeout ? "Customer verification timeout" : "Customer verification failed",
      { error, txId: tx.id }
    );
    return await context.step(`timeout-escalate-${tx.id}`, async () =>
      sendToFraudDepartment(tx, true)
    );
  }

  // Idempotent final step with idempotency key
  return await context.step(`finalize-${tx.id}`, async () => {
    const action = !verified.hasFailure && verified.successCount > 0
      ? "authorize"
      : "escalate";
    if (action === "authorize") {
      return authorizeTransaction(tx, true, { idempotency_key: `finalize-${tx.id}` });
    }
    return sendToFraudDepartment(tx, true);
  });
});

Notice how the best practices work together: context.parallel() sends SMS and email concurrently, resuming when either channel responds. Both callbacks configure 1-day timeouts with try/catch handling that escalates on timeout. The DurableExecutionName: tx-${transactionId} parameter (specified at invocation time, shown in the following CLI example) provides execution-level deduplication, while idempotency keys in the authorization steps prevent duplicate charges at the application layer. Asynchronous invocation (InvocationType: 'Event') enables the 24-hour wait period.

Once deployed, invoke the function asynchronously with a sample transaction to see it in action:

transactionId="123456789"
aws lambda invoke \
  --function-name "fraud-detection:$LATEST" \
  --invocation-type Event \
  --durable-execution-name "tx-${transactionId}" \
  --cli-binary-format raw-in-base64-out \
  --payload "{\"id\": \"${transactionId} \", \"amount\": 6500, \"location\": \"New York, NY\", \"vendor\": \"Amazon.com\"}" \
  --region us-east-2 \
  response.json

Upon successful invocation, you can view the execution state in the Lambda console’s durable operations view. The execution shows a suspended state, waiting for customer response:

Figure 2: Suspended execution state

Notice the fraudCheck and suspendTransaction steps show as succeeded with checkpointed results. The human-verification parallel operation shows that both SMS and email branches started. The timeline shows the function in a suspended state. Simulate a customer response by sending a callback success through the console, AWS Command Line Interface (AWS CLI) or Lambda API:

aws lambda send-durable-execution-callback-success \
	--callback-id <CALLBACK_ID_FROM_EMAIL_OR_SMS> \
	--result '{"status":"approved","channel":"email"}' \
	--cli-binary-format raw-in-base64-out

Figure 3: Completed execution with customer approval

After receiving the customer’s approval, the durable execution resumes from its checkpoint, authorizes the transaction, and completes. The execution spanned hours but consumed only seconds of compute time.

Conclusion

With durable functions, Lambda extends beyond single-event processing to power core business processes and long-running workflows, while retaining the operational simplicity, reliability, and scale that define Lambda. You can build applications that run for days or months, survive failures, and resume where they left off, all within the familiar event-driven programming model.

Deploy the fraud detection workflow from our GitHub repository and experiment with human-in-the-loop patterns in your own account. For core concepts, see Introduction to AWS Lambda Durable Functions. For comprehensive documentation, see the Lambda Developer Guide. Browse Serverless Land for reference architectures and discover where durable execution fits in your designs.

Share your feedback, questions, and use cases in the SDK repositories or on re:Post.

Testing Step Functions workflows: a guide to the enhanced TestState API

D Surya Sai — Sun, 22 Mar 2026 17:06:38 +0000

AWS Step Functions recently announced new enhancements to local testing capabilities for Step Functions, introducing API-based testing that developers can use to validate workflows before deploying to AWS. As detailed in our Announcement blog post, the TestState API transforms Step Functions development by enabling individual state testing in isolation or as complete workflows. This supports mocked responses and actual AWS service integrations, and provides advanced capabilities. These capabilities include Map/Parallel states, error simulation with retry mechanisms, context object validation, and detailed inspection metadata for comprehensive local testing of your serverless application.

The TestState API can be accessed through multiple interfaces such as AWS Command Line Interface (AWS CLI), AWS SDK, LocalStack. By default, TestState API in AWS CLI and SDK runs against the remote AWS endpoint, providing validation against the actual Step Functions service infrastructure. We’ve partnered with LocalStack to offer an additional testing endpoint for the TestState API. Developers can use LocalStack for unit testing their workflows by changing the AWS SDK client endpoint configuration to point to LocalStack: http://localhost.localstack.cloud:4566/ instead of AWS endpoint. This approach provides complete network isolation when needed. For a streamlined development experience, you can also use the LocalStack VSCode extension to automatically configure your environment to point to the LocalStack endpoint. This approach is detailed in the AWS blog post.

This blog post demonstrates building test suites to unit test your Step Functions workflows using the AWS SDK for Python using the pytest framework. The complete implementation is available in the GitHub repository.

Building test cases using the TestState API

This example workflow implements a real-world ecommerce order processing system using JSONata for advanced data transformations. It incorporates complex Step Functions patterns including distributed Map states, Parallel execution, and waitForTaskToken callback mechanisms. The process validates orders through AWS Lambda functions, distributes order item processing with configurable failure tolerance, runs parallel payment and inventory updates, handles human approval workflows using task tokens, then persists orders in Amazon DynamoDB with notification delivery. This workflow demonstrates advanced error handling with multiple Catchers and Retriers, exponential backoff for Lambda throttling and DynamoDB limits, and sophisticated state transitions that were previously challenging to test locally. This makes it the recommended choice for demonstrating the use of enhanced TestState API’s local testing features.

The complete workflow is available in the GitHub repository, where you can examine the full state machine definition and see how JSONata expressions handle data transformation throughout the execution flow.

Figure 1: State machine workflow that demonstrates a real-world ecommerce order processing system.

Effective Step Functions testing requires a systematic approach to TestState API integration that provides state validation, error simulation, and assertion capabilities. The testing framework is built using Python’s pytest framework, using fixtures to automatically provide pre-configured runner instances that handle TestState API client initialization and state machine definition loading. This eliminates repetitive setup code and provides consistent test environments. The enhanced TestState API supports both mock integrations and actual integrations with AWS services, providing flexibility in testing strategies. For this demonstration, you use mock integrations to showcase how a complete local testing can be achieved without having any resources deployed to AWS accounts.

This framework is built for demonstration purposes, and you can similarly build your own testing frameworks using other programming languages like Java, Node.js. The testing framework uses method chaining patterns to create readable test cases with comprehensive assertion methods, automatic output chaining between state executions, and error simulation for testing retry mechanisms, backoff intervals, and catch blocks across AWS service error conditions.

The following test implementations demonstrate the testing capabilities that are achievable with the enhanced TestState API in local development environments. The test cases are run against the preceding Statemachine.

Test Case 1: Lambda throttling and retry mechanism testing

Service integrations with Statemachines like AWS Lambda, Amazon DynamoDB may face throttling depending on their usage. A key capability of the enhanced TestState API is its ability to simulate retry mechanisms with control over retry counts and backoff intervals. This test demonstrates the enhanced TestState API’s retry testing capabilities through the stateConfiguration.retrierRetryCount parameter and inspectionData.errorDetails response fields. This response field provides retryBackoffIntervalSeconds for validating exponential backoff calculations, retryIndex for tracking retry attempt sequences, and catchIndex for identifying which error handler processed the exception. These enhanced inspection capabilities enable validation of retry logic, backoff strategies, and error propagation patterns across complex state machine workflows.

def test_lambda_throttling_retry_mechanism(self, runner):
"""Test retry mechanism for Lambda.TooManyRequestsException"""
throttling_error = {
"Error": "Lambda.TooManyRequestsException",
"Cause": "Request rate exceeded"
}

# Test first retry attempt
(runner
.with_input({"orderId": "order-retry-test"})
.with_mock_error(throttling_error)
.with_retrier_retry_count(0)
.execute("ValidateOrder")
.assert_retriable()
.assert_error("Lambda.TooManyRequestsException"))

# Verify exponential backoff calculation
response = runner.get_response()
error_details = response['inspectionData']['errorDetails']
assert error_details['retryBackoffIntervalSeconds'] == 2

# Test retry exhaustion
(runner
.with_retrier_retry_count(3)
.execute("ValidateOrder")
.assert_caught_error()
.assert_next_state("ValidationFailed"))

Test Case 2: Map state testing with tolerance thresholds

Distributed Map states present unique testing challenges due to their parallel processing nature and failure tolerance capabilities. The enhanced TestState API provides specialized configuration options for testing these complex scenarios.

def test_map_state_tolerated_failure_threshold(self, runner):
"""Test Map state with tolerated failure threshold"""
test_input = {
"orderId": "order-map-test",
"orderItems": [
{"itemId": "item-1"}, {"itemId": "item-2"}, 
{"itemId": "item-3"}, {"itemId": "item-4"}
]
}

# Test normal Map state execution
map_success_result = [
{"itemId": "item-1", "processed": True},
{"itemId": "item-2", "processed": True}
]

(runner
.with_input(test_input)
.with_mock_result(map_success_result)
.execute("ProcessOrderItems")
.assert_succeeded()
.assert_next_state("ParallelProcessing"))

# Test tolerance threshold exceeded scenario
tolerance_error = {
"Error": "States.ExceedToleratedFailureThreshold",
"Cause": "Map state exceeded tolerated failure threshold"
}

(runner
.with_input(test_input)
.with_mock_error(tolerance_error)
.execute("ProcessOrderItems")
.assert_caught_error()
.assert_next_state("ValidationFailed"))

This test demonstrates the enhanced TestState API’s Map state testing capabilities through the stateConfiguration.mapIterationFailureCount parameter for simulating iteration failures. The API provides comprehensive inspection data including inspectionData.afterItemSelector for validating ItemSelector transformations, inspectionData.afterItemBatcher for batch processing validation, inspectionData.toleratedFailureCount and inspectionData.toleratedFailurePercentage for threshold verification. When the specified failure count exceeds the configured tolerance, the API correctly returns States.ExceedToleratedFailureThreshold, enabling testing of Map state resilience patterns.

Test Case 3: WaitForCallback pattern testing

The waitForCallback integration requires context object construction to simulate realistic execution environments, particularly for human approval workflows.

def test_context_object_usage_in_jsonata_expressions(self, runner):
"""Test Context object usage in waitForTaskToken scenarios"""
test_input = {
"orderId": "order-context-test",
"amount": 125.0
}

context_data = {
"Task": {"Token": "ahbdgftgehbdcndsjnwjkhas327yr4hendc73yehdb723y"},
"Execution": {
"Id": "arn:aws:states:us-east-1:123456789012:execution:test:exec-123"
},
"State": {
"Name": "WaitForApproval",
"EnteredTime": "2025-01-15T10:45:00Z"
}
}

mock_result = {
"approved": True,
"taskToken": "ahbdgftgehbdcndsjnwjkhas327yr4hendc73yehdb723y"
}

(runner
.with_input(test_input)
.with_context(context_data)
.with_mock_result(mock_result)
.execute("WaitForApproval")
.assert_succeeded()
.assert_next_state("CheckApproval"))

# Verify JSONata expressions processed context correctly
response = runner.get_response()
after_args = json.loads(response['inspectionData']['afterArguments'])
assert after_args['Payload']['taskToken'] == context_data['Task']['Token']

This test demonstrates the enhanced TestState API’s support for waitForCallback integrations through the `context` parameter for realistic Context object simulation. The API enables comprehensive testing of JSONata expressions that reference $states.context.Task.Token, $states.context.Execution.Id, and other context fields. The inspectionData.afterArguments response field validates that JSONata expressions correctly processed the context data, while the API automatically handles the complexity of task token embedding in service integration payloads for waitForCallback testing scenarios.

Test Case 4: Happy path testing – complete workflow validation

Happy path testing validates that workflows execute correctly under normal operating conditions. The enhanced TestState API allows you to chain state executions together, automatically passing outputs between states to simulate a complete workflow execution.

def test_complete_order_processing_workflow(self, runner):
"""Integration test: Complete happy path workflow using method chaining"""
test_input = {
"orderId": "order-12345",
"amount": 150.75,
"customerEmail": "customer@example.com",
"orderItems": [
{"itemId": "item-1", "quantity": 2, "price": 50.25}
]
}

# Test ValidateOrder state
(runner
.with_input(test_input)
.with_mock_result({"statusCode": 200, "isValid": True})
.execute("ValidateOrder")
.assert_succeeded()
.assert_next_state("CheckValidation"))

# Test CheckValidation choice state (no mock needed)
validation_output = runner.get_output()
(runner
.with_input(validation_output)
.clear_mocks()
.execute("CheckValidation")
.assert_succeeded()
.assert_next_state("ProcessOrderItems"))

This test demonstrates how the TestState API maintains state context between executions, enabling realistic workflow simulation. The get_output() method retrieves the processed output from one state to use as input for the next, mimicking actual Step Functions execution behavior.

Note: The code snippet above shows only the first two states of the complete workflow test for brevity. The full test code with all states (ProcessOrderItems, ParallelProcessing, WaitForApproval, CheckApproval, SaveOrderDetails, and SendNotification) can be viewed in the complete GitHub repository, demonstrating end-to-end workflow validation using the same method chaining pattern.

Integration with modern CI/CD pipelines

In this section, we will explore how to integrate the previous unit tests in a CI CD pipeline to enable local testing.

The sample repository includes a GitHub Actions workflow that demonstrates how TestState API testing integrates into continuous integration and continuous delivery (CI/CD) pipelines. The workflow (.github/workflows/test-and-deploy.yml) provides a two-step process that validates before any AWS resources are deployed using AWS Serverless Application Model (AWS SAM).

The CI/CD pipeline follows the following pattern:

Unit Tests: Executes the complete TestState API test suite using pytest tests/unit_test.py -v
SAM Deploy: Deploys AWS resources using sam build and sam deploy

To enable the GitHub Actions workflow to deploy resources to your AWS account, configure these AWS credentials in your GitHub repository settings. For detailed setup instructions, see the AWS blog post.

Following are the required secrets to be configured in GitHub repository settings:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_REGION

In production environments, you can typically extend this basic pipeline to include additional stages. The enhanced pipeline often begins with deploying to a development account first, followed by integration testing against deployed resources. The final stage involves moving to production with proper approval gates and security scanning compliance checks.

Conclusion

The enhanced TestState API enables testing Step Functions workflows locally without requiring AWS deployments that accelerated development cycles, and reduce testing times. This post demonstrates how to implement testing for state types including Map states with tolerance thresholds, retry mechanisms with exponential backoff, and waitForTaskToken patterns with context object simulation using mock integrations for isolated testing.

By integrating TestState API testing into CI/CD pipelines, you can validate workflow logic before deployment, reducing the risk of production issues. The GitHub Actions workflow example demonstrates an implementation that runs tests and deploys resources in a controlled sequence. The complete code examples and testing framework are available in the GitHub repository to implement similar testing practices for Step Functions workflows.

Enabling high availability of Amazon EC2 instances on AWS Outposts servers (Part 3)

Brianna Rosentrater — Fri, 06 Mar 2026 23:11:22 +0000

This post is part 3 of the three-part series ‘Enabling high availability of Amazon EC2 instances on AWS Outposts servers’. We provide you with code samples and considerations for implementing custom logic to automate Amazon Elastic Compute Cloud (EC2) relaunch on Outposts servers. This post focuses on guidance for using Outposts servers with third party storage for boot and data volumes, whereas part 1 and part 2 focus on automating EC2 relaunch between standalone servers. Outposts servers support integration with Dell PowerStore, HPE Alletra Storage MP B10000 systems, NetApp on-premises enterprise storage arrays, and Pure Storage FlashArray.

Outposts servers provide compute and networking services that are designed for low-latency, local data processing needs for on-premises locations such as retail stores, branch offices, healthcare provider locations, or environments that are space-constrained. Outposts servers use EC2 instance store storage to provide non-durable block-level storage to the instances running stateless workloads. For applications that require persistent storage, you can create a three-tier architecture by connecting your Outposts servers to a third-party storage appliance. In this post, you will learn how to implement custom logic to provide high availability (HA) for your applications running on Outposts servers using two or more servers for N+1 fault tolerance. The code provided is meant to help you get started, and can be modified further for your unique workload needs.

Overview

In the following sections we will show how custom logic can be used to automate EC2 instance relaunch between two or more Outposts servers using boot and data volumes on third party storage. If your EC2 instance fails while using this solution, an Amazon CloudWatch alarm monitoring the EC2 StatusCheckFailed_Instance metric of your source EC2 instance will be triggered, and you will receive an Amazon Simple Notification Service (Amazon SNS) notification. An AWS Lambda function will then relaunch your EC2 instance onto the destination Outposts server that you’ve set up for resiliency. This is done using a launch template created during setup, and the script will connect your relaunched instance to the existing boot and data volumes on your third party storage appliance. This storage device provides shared storage for your Outposts servers. If a single server fails, new instances can connect to existing volumes on the array. This allows for a zero data loss Recovery Point Objective (RPO) and a Recovery Time Objective (RTO) equaling the time it takes to launch your EC2 instance. Take advantage of the features on your storage appliance for configuring data durability and resiliency to hardware failures, and make sure that you are regularly backing up your SAN volumes.

Figure 1 – Solution Architecture for automated EC2 Relaunch

Prerequisites

The following prerequisites are required to complete the walkthrough:

Two Outposts servers that can be set up as an active-active or active-passive resilient pair.
For workloads with a low threshold for downtime, ensure that your secondary Outpost server that’s used for recovery has a unique service link connection.
Outposts servers must be colocated within the same Layer 2 (L2) network.
Network latency between the Outposts servers must not exceed 5ms round trip time (RTT).
A storage appliance that supports the iSCSI protocol. Credentials to manage the storage appliance initiator/target mappings. See Simplifying the use of third-party block storage with AWS Outposts for more information.
If you’re setting this up from an Outposts consumer account, you must configure Amazon CloudWatch cross-account observability between the consumer account and the Outposts owning account to view Outposts metrics in your consumer account.
Create launch templates for the EC2 instances that you want to protect, the launch wizard will help you create these.
Credentials with permissions for AWS CloudFormation, Amazon EC2, and (optional) AWS Secrets Manager if authentication is required. IAM Permission Examples.md is provided in the repository.
A Windows or Linux host that can access the storage appliance and your AWS account (management computer).
AWS Outposts iPXE Amazon Machine Image (AMI) from the AWS Marketplace.
Python 3.8 or later (recommended) is used to run the init.py script that dynamically creates a CloudFormation stack in the account specified as an input parameter.
AWS SDK for Python (Boto3) version 1.26.0 or later recommended.
Operating system with iSCSI boot support (Windows Server 2022 and Red Hat Enterprise Linux 9 AMIs are provided).
Internet access to AWS service endpoints for the private subnet hosting the recovery Lambda function.
Download the repository ec2-outposts-autorestart_3Pstorage.

Walkthrough

The first step is to deploy an EC2 instance configured to boot from a volume on the third-party storage that is prepared with an OS boot image. This step uses the launch wizard portion of the solution.

Download and extract the OutpostServer_Recovery_3Pstorage repository to the management computer that has the AWS SDK for Python (Boto3) and Python installed.
Run launch_wizard from the sample-outposts-third-party-storage-integration directory. You can run interactively or provide arguments for region, subnet, iPXE AMI, storage vendor, storage management ip, and credentials.

Figure 2 – Running launch wizard

When prompted for a feature name, enter sanboot.
For Guest OS type, enter in Linux or Windows.
When prompted “Do you want to continue with this unverified AMI?”, select Y.
The launch wizard will provide a list of instance types available on the Outpost server associated with the subnet you specified. Enter the instance type that you want to use.
The launch wizard will now prompt you for optional EC2 Key Pair, Security Group, and Instance Profile settings for the EC2 instance that you are launching.
Next, the launch wizard prompts you to specify an instance name. Note that specifying an instance name is required to set up automated instance recovery because the instance name is used as part of the recovery process.

Figure 3 – Taking user input for variable values

The launch wizard prompts for root volume size. This is the root volume that the iPXE AMI boots from. The default is a 1GB volume on the Outpost server instance storage.
Next, the launch wizard prompts you to select which third party storage controller you want to use based on the management ip that you specified. In this example, we are using NetApp, so I select a NetApp Storage Virtual Machine (SVM) named outpost_iscsi.
If the connection to the storage array is successful and the protocol is available (iSCSI or NVMe over TCP) you are provided additional storage options for initiator group and logical unit number (LUN).
In this example, we are using NetApp with iSCSI, so I can select an existing initiator group or create a new one.
You can specify an existing initiator qualified name (IQN), or the launch wizard can generate a new one. IMPORTANT: Make sure that IQNs are unique to each instance because duplicates can cause data corruption.
Next the launch wizard prompts which LUN’s you want to connect to this instance. For this example, I am going to use a Windows Server 2022 boot volume that I already created on the NetApp storage array.
You are now asked which storage array target interface you want to use for connecting to these LUNs.
The launch wizard provides the capability to specify guest OS scripts to customize the OS after sanboot. Combining this capability with storage array cloning provides a streamlined process for deploying new instances.
The launch wizard now displays the EC2 user data template that it generated for use with the iPXE AMI and asks if you want to proceed with launching the instance.
After the EC2 instance is launched, select yes to proceed with automated instance recovery setup.

Figure 4 – Running launch template creation script

Generating EC2 launch templates for recovery and failback

In the second step, we are generating EC2 launch templates for the EC2 instance launched in step 1. Launch templates can be generated for the primary and secondary Outpost servers. The launch template for the secondary Outpost server can be used for automated or manual recovery of the EC2 instance. Failback to the primary Outpost server is manual using the primary launch template.

Select the instance that you want automated recovery for and select the subnet that you launched the instance in. This subnet represents the primary Outpost server that the instance is running on.

Figure 5 – Selecting subnets for EC2 instance relaunch

When prompted to create a second launch template for Outpost server recovery, select yes, and then select to use the same instance (for recovery on different Outpost server).
When you get a list of available subnets, select the subnet that’s associated with your secondary Outpost server. This is the server that the EC2 instance will be launched on in the event of the EC2 StatusCheckFailed_Instance metric triggers the CloudWatch alarm.
You will see both launch templates created successfully.

Deploying automated EC2 instance recovery

The third step creates a CloudFormation template for monitoring, notifications, and automated recovery of the EC2 instance deployed in step 1. The CloudFormation template automatically captures the instance and secondary launch template information necessary for automatic recovery.

Select Y to set up automated recovery. This will create a CloudFormation stack.
Provide a name and description for the CloudFormation stack.
Select whether you want automated recovery or notification only. This provides flexibility to choose manual or automatic recovery based on whether you want to verify the primary Outpost server is down before initiating recovery.
In the AWS CloudFormation console, monitor the CloudFormation stack creation process.

Figure 6 – CloudFormation stack creation in progress

After the CloudFormation Stack is complete, you have successfully deployed an EC2 instance using third party storage for boot and data volumes on a primary Outpost server. You also created instance recovery capabilities by using the Amazon Outpost server automated recovery solution for third party storage.
You can verify whether the EC2 StatusCheckFailed_Instance is healthy under the Alarms section in the Amazon CloudWatch console.

Considerations

The logic discussed in this post relies on the secondary destination Outposts server having a connected service link. For more information about how to create a highly available service link connection for your Outpost servers, see the Networking section of AWS Outposts High Availability Design and Architecture Considerations whitepaper.

Clean up

Confirm whether it is safe to terminate the Amazon EC2 instance that you launched with this walkthrough. The operating system and data volumes are on the third party storage, so EC2 instance termination only removes the iPXE AMI from the Outposts server instance storage. To clean up, complete the following steps.

Terminate the Amazon EC2 instance. Then, verify that the Instance state is Terminated to ensure that the instance is not using Outposts server resources.
Delete the Amazon EC2 Launch Templates associated with the Amazon EC2 instance that you terminated. The names of the launch templates that were automatically generated will start with ‘lt-‘, followed by the instance name and the instance id. If you generated a recovery launch template, it will have a ‘-recovery’ suffix in the name.
Delete the AWS CloudFormation Stack. The Stack name will start with ‘autorestart-‘ followed by the Amazon EC2 instance name.
Clean up your initiators, initiator group, and LUNs on the third party storage array.

Conclusion

With the use of custom logic through AWS tools such as CloudFormation, CloudWatch, Amazon SNS, and AWS Lambda, you can architect for HA for stateful workloads on Outposts server. By implementing the custom logic in this post, you can automatically relaunch EC2 instances running on a source Outposts server to a secondary destination Outposts server if an instance fails, and connect to existing volumes on a shared storage appliance for recovery. This also reduces the downtime of your applications in the event of a hardware or service link failure. The code provided in this post can be further expanded upon to meet the unique needs of your workload.

While the use of infrastructure-as-code (IaC) can improve your application’s availability and be used to standardize deployments across multiple Outposts servers, it’s crucial to do regular failure drills to test the custom logic in place. This is to make sure that you understand your application’s expected behavior on relaunch in the event of a failure. To learn more about Outposts servers, visit the Outposts servers User Guide. Reach out to your AWS account team, or fill out this form to learn more about Outposts servers.

Optimizing Compute-Intensive Serverless Workloads with Multi-threaded Rust on AWS Lambda

Daniel Abib — Wed, 25 Feb 2026 12:49:44 +0000

Customers use AWS Lambda to build Serverless applications for a wide variety of use cases, from simple API backends to complex data processing pipelines. Lambda’s flexibility makes it an excellent choice for many workloads, and with support for up to 10,240 MB of memory, you can now tackle compute-intensive tasks that were previously challenging in a Serverless environment. When you configure a Lambda function’s memory size, you allocate RAM and Lambda automatically provides proportional CPU power. When you configure 10,240 MB, your Lambda function has access to up to 6 vCPUs.

However, there’s an important consideration that many developers discover: simply allocating more memory may not automatically make your function faster. If your code runs sequentially, it will only use one vCPU regardless of how many are available. The remaining vCPUs sit idle while you’re still paying for the full memory allocation.

To help benefit from Lambda’s multi-core capabilities, your code should explicitly implement concurrent processing through multi-threading or parallel execution. Without this, you’re paying for compute power you’re not using.

Rust provides excellent support for this pattern. The AWS Lambda Rust Runtime provides developers with a language that combines exceptional performance with built-in concurrency primitives. In this post, we show you how to implement multi-threading in Rust to achieve 4-6x performance improvements for CPU-intensive workloads.

Our Test Workload: Why Bcrypt Password Hashing?

For this analysis, we use bcrypt password hashing as our CPU-intensive workload to evaluate multi-core scaling behavior. This choice is deliberate for several reasons:

Real-world relevance: Bcrypt is commonly used in authentication systems, making our benchmarks practically relevant rather than synthetic.
Predictable CPU work: Bcrypt with cost factor 10 provides approximately 100ms of pure CPU work per operation on typical hardware, creating a consistent and measurable baseline.
Embarrassingly parallel: Each hash operation is completely independent, making it an ideal candidate for parallel processing without shared state or lock contention.
CPU-bound: Bcrypt is deterministic and CPU-bound (not memory or I/O bound), isolating the performance characteristics we want to measure.

Throughout this post, we process batches of passwords and measure how multi-threading improves throughput as we scale from 1 to 6 vCPUs.

Understanding Lambda’s vCPU Allocation

AWS Lambda allocates CPU resources proportionally to the configured memory. According to AWS Lambda function memory documentation, at 1,769 MB a function has the equivalent of one vCPU.

vCPU Allocation by Memory:

Memory (MB)	Approximate vCPUs
128 – 1,769	~1
1,770 – 3,538	~2
3,539 – 5,307	~3
5,308 – 7,076	~4
7,077 – 8,845	~5
8,846 – 10,240	~6

Note: The num_cpus crate returns the number of logical CPUs visible to the Lambda environment, which may differ from the allocated vCPU share. At lower memory configurations, you may see 2 CPUs reported even though only 1 vCPU worth of compute time is allocated.

Solution Overview

The solution consists of a Rust Lambda function that:

Receives a request specifying the number of items to process
Detects available vCPUs and configures a thread pool accordingly
Processes items in parallel using the Rayon library (a data parallelism library that allows you to convert sequential iterators into parallel ones with a .par_iter() call)
Returns performance metrics including duration and throughput

Architecture Diagram: Lambda receives request, initializes Rayon thread pool based on WORKER_COUNT environment variable, processes bcrypt hashes in parallel across multiple vCPUs, and returns results.

Creating a Multi-threaded Rust Lambda Function

Create a new Lambda project using Cargo Lambda:

cargo lambda new rust-multithread-demo
cd rust-multithread-demo

Dependencies

Update Cargo.toml with the necessary dependencies:

[package]
name = "rust-multithread-lambda"
version = "0.1.0"
edition = "2021"

[dependencies]
lambda_runtime = "1.0.0"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
bcrypt = "0.15"
rayon = "1.7"
num_cpus = "1.16"

[profile.release]
opt-level = 3
lto = true
codegen-units = 1
strip = true

The optimization flags in [profile.release] reduce binary size and improve performance:

opt-level = 3: Maximum optimization
lto = true: Link-time optimization for smaller binaries
strip = true: Remove debug symbols

Implementing the Lambda Entry Point

First, let’s look at how we initialize the thread pool during cold start:

src/main.rs:

use lambda_runtime::{run, service_fn, Error, LambdaEvent};
mod handler;
use handler::{function_handler, get_worker_count, init_thread_pool, ProcessRequest};

#[tokio::main]
async fn main() -> Result<(), Error> {
    // Initialize Rayon thread pool at cold start (once per container lifecycle)
    init_thread_pool(get_worker_count());

    run(service_fn(|event: LambdaEvent<ProcessRequest>| async move {
        function_handler(event.payload).await
    }))
    .await
}

Why initialize in main() and not in the handler?

Deterministic Configuration: The thread pool is configured once per container, before any requests arrive. This prevents race conditions if multiple requests try to initialize concurrently.
Container Reuse: Lambda containers can serve multiple requests. Initializing in main() ensures the configuration is set during the cold start and persists for all subsequent warm invocations.
Performance: Thread pool setup happens during cold start (already counted as initialization time), not during request processing.

Implementing the Request Handler

src/handler.rs:

use serde::{Deserialize, Serialize};
use std::env;
use std::sync::Once;
use std::time::Instant;
use std::collections::HashSet;
use std::sync::Mutex;
use rayon::prelude::*;

static INIT: Once = Once::new();

#[derive(Deserialize)]
pub struct ProcessRequest {
    count: usize,
    mode: String,
}

#[derive(Serialize)]
pub struct ProcessResponse {
    processed: usize,
    duration_ms: u128,
    mode: String,
    workers: usize,
    detected_cpus: usize,
    avg_ms_per_item: f64,
    memory_used_kb: u64,
    threads_used: usize, // Actual threads that processed items (proves multi-threading)
}

// CPU-intensive bcrypt hashing with cost factor 10
fn hash_password(password: &str) -> Result<String, bcrypt::BcryptError> {
    bcrypt::hash(password, 10)
}

// Process items one at a time (baseline for comparison)
fn process_sequential(items: Vec<String>) -> Result<(Vec<String>, usize), Box<dyn std::error::Error + Send + Sync>> {
    let results: Result<Vec<String>, _> = items
        .iter()
        .map(|item| hash_password(item))
        .collect();
    results
        .map(|r| (r, 1))
        .map_err(|e| Box::new(e) as Box<dyn std::error::Error + Send + Sync>)
}

// Process items in parallel using Rayon's work-stealing scheduler
// Thread pool size is configured once at cold start via init_thread_pool()
fn process_parallel(items: Vec<String>) -> Result<(Vec<String>, usize), Box<dyn std::error::Error + Send + Sync>> {
    let thread_ids: Mutex<HashSet<std::thread::ThreadId>> = Mutex::new(HashSet::new());

    let results: Result<Vec<String>, _> = items
        .par_iter()
        .map(|item| {
            thread_ids.lock().unwrap().insert(std::thread::current().id());
            hash_password(item)
        })
        .collect();

    let threads_used = thread_ids.lock().unwrap().len();
    results
        .map(|r| (r, threads_used))
        .map_err(|e| Box::new(e) as Box<dyn std::error::Error + Send + Sync>)
}

// Get worker count from env var or detect CPUs, clamped to 1-6
pub fn get_worker_count() -> usize {
    if let Ok(count_str) = env::var("WORKER_COUNT") {
        if let Ok(count) = count_str.parse::<usize>() {
            return count.clamp(1, 6);
        }
    }
    num_cpus::get().clamp(1, 6)
}

// Initialize Rayon global thread pool (only once per Lambda container)
pub fn init_thread_pool(workers: usize) {
    INIT.call_once(|| {
        let _ = rayon::ThreadPoolBuilder::new()
            .num_threads(workers)
            .build_global();
    });
}

// Read RSS memory from /proc/self/statm (Linux only)
fn get_memory_usage_kb() -> u64 {
    std::fs::read_to_string("/proc/self/statm")
        .ok()
        .and_then(|s| s.split_whitespace().nth(1)?.parse::<u64>().ok())
        .map(|pages| pages * 4)
        .unwrap_or(0)
}

// Main Lambda handler - processes items sequentially or in parallel
pub async fn function_handler(request: ProcessRequest) -> Result<ProcessResponse, Box<dyn std::error::Error + Send + Sync>> {
    if request.count == 0 { return Err("count must be greater than 0".into()); }
    if request.count > 1000 { return Err("count exceeds maximum of 1000 items".into()); }

    let items: Vec<String> = (0..request.count)
        .map(|i| format!("password_{:06}", i))
        .collect();

    let workers = get_worker_count();
    let mode = match request.mode.as_str() {
        "sequential" => "sequential",
        "parallel"   => "parallel",
        _            => if workers > 1 { "parallel" } else { "sequential" },
    };

    let start = Instant::now();
    let (results, threads_used) = match mode {
        "sequential" => process_sequential(items)?,
        _            => process_parallel(items)?,
    };
    let duration_ms = start.elapsed().as_millis();

    Ok(ProcessResponse {
        processed: results.len(),
        duration_ms,
        mode: mode.to_string(),
        workers: if mode == "parallel" { workers } else { 1 },
        detected_cpus: num_cpus::get(),
        avg_ms_per_item: duration_ms as f64 / request.count as f64,
        memory_used_kb: get_memory_usage_kb(),
        threads_used,
    })
}

Key Implementation Details

Thread Pool Initialization at Cold Start: The code initializes the thread pool in main() before the Lambda runtime starts, not during request processing. This approach is designed to eliminate race conditions and provide deterministic behavior across all invocations.

Important Note: Lambda initializes the thread pool once per container. The thread pool configuration retains its original value even if you change the WORKER_COUNT environment variable between invocations within the same container. For production deployments, keep WORKER_COUNT consistent for the function’s lifecycle.

Input Validation: The handler validates that count is between 1 and 1000 to prevent resource exhaustion.

Thread Tracking: The threads_used field proves multi-threading is working by tracking unique thread IDs during parallel processing. This provides empirical validation that work is distributed across multiple threads.

Memory Tracking: The memory_used_kb field reports RSS memory usage by reading /proc/self/statm, providing visibility into actual memory consumption.

Mode Selection: The function supports three modes:

sequential: Single-threaded processing
parallel: Multi-threaded processing using Rayon
auto: Automatically selects based on available workers

Building and Deploying

With the implementation complete, let’s compile the function for Lambda’s environment and deploy it to AWS.

# Build for ARM64 (Graviton2) - recommended for cost efficiency
cargo lambda build --release --arm64

# Or build for x86_64
cargo lambda build --release --x86-64

The build process produces a binary of approximately 1.7 MB (uncompressed) or 0.8 MB (zipped).

Deploy to AWS

Use Cargo Lambda to deploy the function with your desired memory configuration and worker count.

# Deploy with 6144 MB memory (4 vCPUs) and 4 workers
cargo lambda deploy rust-multithread-lambda \
    --memory 6144 \
    --timeout 30 \
    --env-var WORKER_COUNT=4

Note: To test different configurations, repeat the build and deploy commands with different --memory values and WORKER_COUNT settings for each configuration you want to benchmark. For comprehensive testing across architectures, build with --arm64, deploy all memory configurations, then rebuild with --x86-64 and deploy again.

Required IAM Permissions

The Lambda execution role needs the following permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        }
    ]
}

Test the Function

After deployment, verify the function works correctly by invoking it with a test payload.

aws lambda invoke \
    --function-name rust-multithread-lambda \
    --payload '{"count":20,"mode":"parallel"}' \
    --cli-binary-format raw-in-base64-out \
    response.json

Performance Benchmarks

We tested multiple configurations on ARM64 (Graviton2) to measure the impact of multi-threading.

Test workload: Processing 20 bcrypt password hashes (cost factor 10)

Note: Benchmark results may vary between runs due to factors such as Lambda placement, underlying hardware differences, and AWS infrastructure conditions. The numbers presented here are representative of typical performance observed across multiple test runs.

Performance Results: ARM64 (Graviton2)

Memory	vCPUs	Workers	Avg (ms)	P50 (ms)	P95 (ms)	P99 (ms)	Min	Max	Speedup
1536 MB	~1	1	1,885	1,882	1,898	1,898	1,877	1,907	1.00x
2048 MB	~2	2	1,334	1,331	1,341	1,341	1,324	1,356	1.41x
4096 MB	~3	3	685	683	699	699	669	704	2.75x
6144 MB	~4	4	463	464	467	467	453	469	4.07x
8192 MB	~5	5	338	343	345	345	325	346	5.57x
10240 MB	~6	6	280	278	292	292	271	293	6.73x

Performance Results: x86_64

Memory	vCPUs	Workers	Avg (ms)	P50 (ms)	P95 (ms)	P99 (ms)	Min	Max	Speedup
1536 MB	~1	1	1,671	1,675	1,681	1,681	1,659	1,684	1.00x
2048 MB	~2	2	1,253	1,249	1,265	1,265	1,241	1,294	1.33x
4096 MB	~3	3	892	891	899	899	888	900	1.87x
6144 MB	~4	4	429	425	443	443	417	449	3.89x
8192 MB	~5	5	330	323	349	349	317	358	5.06x
10240 MB	~6	6	292	292	298	298	291	298	5.72x

Architecture Comparison

Memory	Workers	ARM64 Avg	x86_64 Avg	Diff %	Faster Arch
1536 MB	1	1,885 ms	1,671 ms	-12.8%	x86_64
2048 MB	2	1,334 ms	1,253 ms	-6.4%	x86_64
4096 MB	3	685 ms	892 ms	+23.2%	ARM64
6144 MB	4	463 ms	429 ms	-7.9%	x86_64
8192 MB	5	338 ms	330 ms	-2.4%	x86_64
10240 MB	6	280 ms	292 ms	+4.1%	ARM64

Key Observations

Cold Start Performance: Rust’s cold start initialization times are consistently between 19-28 ms across all memory configurations and architectures. ARM64 (Graviton2) shows slightly faster cold starts (19-23 ms) compared to x86_64 (26-29 ms). Both are significantly faster than interpreted runtimes because the binary is pre-compiled.

Near-Linear Scaling: Both architectures achieve impressive speedups:

ARM64: 6.73x speedup with 6 workers (exceeds theoretical 6x!)
x86_64: 5.72x speedup with 6 workers

Latency Consistency: The P95 and P99 metrics show excellent consistency:

ARM64 at 6 vCPUs: P50=278ms, P95=292ms, P99=292ms (low variance)
x86_64 at 6 vCPUs: P50=292ms, P95=298ms, P99=298ms

Both architectures show consistent latency at maximum parallelization.

Cost Analysis

Let’s analyze the cost implications of different configurations for processing 20 bcrypt hashes.

Cost Comparison: ARM64 vs x86_64 (us-east-1, as of January 2026):

Config	Memory	Workers	ARM64 Duration	ARM64 Cost/1M	x86_64 Duration	x86_64 Cost/1M	Cheaper Arch
1 vCPU	1536 MB	1	1,885 ms	$38.60	1,671 ms	$42.78	ARM64
2 vCPU	2048 MB	2	1,334 ms	$36.46	1,253 ms	$42.77	ARM64 *
3 vCPU	4096 MB	3	685 ms	$37.47	892 ms	$60.80	ARM64
4 vCPU	6144 MB	4	463 ms	$37.97	429 ms	$44.00	ARM64
5 vCPU	8192 MB	5	338 ms	$36.94	330 ms	$45.10	ARM64
6 vCPU	10240 MB	6	280 ms	$38.27	292 ms	$49.87	ARM64

*Cheaper Arch

Cost Formulas:

ARM64: (Memory in GB) × (Duration in seconds) × $0.0000133334
x86_64: (Memory in GB) × (Duration in seconds) × $0.0000166667 (25% higher rate)

Key Insight: The 2 vCPU ARM64 configuration provides the lowest cost at $36.46 per million invocations while achieving 1.41x speedup. All ARM64 configurations remain cost-competitive ($36-$39 range) despite significant performance differences, demonstrating how increased throughput can offset higher memory costs.

Choosing the Right Configuration:

Priority	Recommended Config	Rationale
Lowest Cost	ARM64, 2048 MB, 2 workers	$36.46/1M invocations, 1.41x speedup
Balanced	ARM64, 4096 MB, 3 workers	$37.47/1M invocations, 2.75x speedup
Low Latency	ARM64, 10240 MB, 6 workers	280ms avg, 6.73x speedup

When to Use Multi-threaded Rust on Lambda

Recommended Use Cases

Batch data processing: Transform, validate, or enrich large datasets
Cryptographic operations: Hashing, encryption, digital signatures
Image/video processing: Resize, transcode, analyze media files
Scientific computing: Simulations, data analysis, machine learning inference
High-volume workloads: Functions invoked >100,000 times per day benefit from optimization

When to Consider Alternatives

I/O-bound operations: Use async Rust instead of multi-threading for database queries or API calls
Simple transformations: Functions completing in <100ms rarely benefit from parallelization
Low-volume workloads: Development overhead may not be justified for <10,000 invocations per day
Rapid prototyping: Python or Node.js may be more appropriate when iteration speed is critical

Cleanup

To delete the resources created in this post:

# Delete the Lambda function
aws lambda delete-function --function-name rust-multithread-lambda

# Delete the CloudWatch log group
aws logs delete-log-group --log-group-name /aws/lambda/rust-multithread-lambda

Note: If you deployed multiple configurations for testing, you’ll need to delete each function individually by repeating the delete command with each function name, or use the SAM template for bulk cleanup:

aws cloudformation delete-stack --stack-name rust-multithread-benchmark

Conclusion

When you allocate more memory to your Lambda function, AWS provides proportionally more vCPUs—up to 6 vCPUs at 10,240 MB. However, sequential code only uses one vCPU, leaving the additional compute power idle while you pay for the full allocation. Multi-threaded Rust with Rayon enables you to harness all available vCPUs for CPU-intensive workloads, transforming unused capacity into real performance gains.

Our benchmarks demonstrate this clearly:

Near-linear scaling: ARM64 achieved 6.73x speedup with 6 workers—you get proportional returns on your vCPU investment
Fast cold starts: 19-28 ms initialization across all configurations, eliminating the cold start concerns often associated with compiled languages
Consistent latency: ARM64 at 6 vCPUs shows only 1ms variance between P50 and P99, critical for predictable response times
Cost efficiency: ARM64 is 15-20% cheaper than x86_64 with better scaling at maximum parallelization

The key takeaway: If your Lambda function performs CPU-intensive work and you’re allocating more than 1,769 MB of memory, you likely have multiple vCPUs available. Without multi-threading, those vCPUs sit idle. Rayon’s parallel iterators allow you to switch from sequential to parallel processing by changing .iter() to .par_iter() in your code.

Recommended starting point: ARM64 with 4096 MB (3 workers) offers an excellent balance of cost and performance for most workloads. Scale up to 6 vCPUs for latency-critical applications, or down to 2 vCPUs for maximum cost savings.

Additional Resources

The complete sample code, SAM template, and test scripts from this post are available at Github Repository.

Amazon SageMaker AI now hosts NVIDIA Evo-2 NIM microservices

Malvika Viswanathan — Tue, 24 Feb 2026 18:48:08 +0000

This post is co-written with Neel Patel, Abdullahi Olaoye, Kristopher Kersten, Aniket Deshpande from NVIDIA.

Today, we’re excited to announce that the NVIDIA Evo-2 NVIDIA NIM microservice are now listed in Amazon SageMaker JumpStart. You can use this launch to deploy accelerated and specialized NIM microservices to build, experiment, and responsibly scale your drug discovery workflows on Amazon Web Services (AWS).

In this post, we demonstrate how to get started with these models using Amazon SageMaker Studio.

NVIDIA NIM microservices on AWS

NVIDIA NIM integrates closely with AWS managed services, such as Amazon Elastic Compute Cloud (Amazon EC2), Amazon Elastic Kubernetes Service (Amazon EKS), and Amazon SageMaker AI, to support deployment of generative AI models at scale. As part of NVIDIA AI Enterprise, which is available in the AWS Marketplace, NVIDIA NIM is a set of microservices designed to accelerate the deployment of generative AI. These prebuilt containers support a broad spectrum of generative AI models, from open source community models, to NVIDIA Nemotron and custom models. NIM microservices are deployed with just a few lines of code, or with a few actions in the SageMaker Studio console. Engineered to facilitate seamless generative AI inferencing at scale, NIM ensures that generative AI applications can be deployed on various AWS services.

NVIDIA BioNeMo Evo 2 overview

NVIDIA BioNeMo is a platform of NIM microservices, developer tools, and AI models that accelerate building, adapting, and deploying biomolecular AI models for drug discovery. It packages curated training recipes, data loaders, and domain-optimized pretrained models for DNA, RNA, and proteins, alongside NVIDIA CUDA-X libraries such as NVIDIA cuEquivariance. These components power tasks such as 3D structure prediction, de novo design, virtual screening, docking, and property prediction with GPU-accelerated performance.

NVIDIA NIM microservices provide optimized, API-first inference that integrates directly into enterprise pipelines across on-premises and the cloud, providing scalable and secure deployment with faster time-to-market and lower Total Cost of Ownership (TCO). The Evo 2 NIM delivers a 40-billion parameter foundation model (FM) trained on a vast dataset of genomes that can be used to predict protein function, identify mutations, and accelerate bioengineering research. Furthermore, the Evo 2 NIM can be chained with other NIM microservices such as ESMFold to create end-to-end, containerized workflows that cut time-to-insight while streamlining deployment through consistent APIs.

SageMaker Studio overview

SageMaker Studio is a web-based integrated development environment (IDE) for machine learning (ML) that provides a unified visual interface for all of the tools that you need to complete each step of the ML development lifecycle. SageMaker Studio provides complete access, control, and visibility into each step of the ML workflow, from data preparation to model building, training, and deployment.

The key features of SageMaker Studio include:

Unified interface: Access all SageMaker capabilities through a single, web-based visual interface
Jupyter notebooks: Fully managed Jupyter notebooks with pre-configured kernels for popular ML frameworks
Model management: Browse, deploy, and manage models from AWS Marketplace and other sources through an intuitive interface
Collaboration: Share notebooks, experiments, and models with your team members
Built-in security: Integrated with AWS Identity and Access Management (IAM) for secure access control
Cost management: Monitor and control costs with built-in usage tracking and resource management tools

Amazon SageMaker JumpStart overview

SageMaker JumpStart is a fully managed service that offers state-of-the-art foundation models for various use cases such as content writing, code generation, question answering, copywriting, summarization, classification, and information retrieval. It provides a collection of pre-trained models that you can deploy quickly, accelerating the development and deployment of ML applications. One of the key components of SageMaker JumpStart is model hubs, which offer a vast catalog of pre-trained models, such as Mistral, for a variety of tasks. You can now discover and deploy Evo 2 NIM in Amazon SageMaker Studio or programmatically through the SageMaker Python SDK, so you can derive model performance and MLOps controls with Amazon SageMaker AI features such as Amazon SageMaker Pipelines, Amazon SageMaker Debugger, or container logs. The model is deployed in a secure AWS environment and in your VPC, helping to support data security for enterprise security needs.

Prerequisites

Before getting started with deployment, make sure that your IAM service role for SageMaker AI has the SageMakerFullAccess permission policy attached. To deploy the NVIDIA NIM microservices successfully, confirm one of the following:

Make sure that your IAM role has the following permissions, and that you have the authority to make AWS Marketplace subscriptions in the AWS account used:

aws-marketplace:ViewSubscriptions
aws-marketplace:Unsubscribe
aws-marketplace:Subscribe

If your account is already subscribed to the model, then you can skip to the following Deploy section. Otherwise, start by subscribing to the model package and move to the Deploy section after.

Subscribe to the model package

To subscribe to the model package, complete the following steps:

Open the SageMaker Jumpstart portal from the SageMaker AI page.
Search for Evo 2 NIM.
Choose View model, and on the Model details page choose Subscribe. This will take you to the AWS Marketplace listing for the Evo 2 NIM.
On the AWS Marketplace listing page, choose View purchase options, review the purchase terms and choose the Subscribe button if you and your organization agree with EULA, pricing, and support terms.
Choose Continue to with the configuration and choose an AWS Region where you have the service quota for the desired instance type.

A product Amazon Resource Name (ARN) is displayed. This is the model package ARN that you need to specify while creating a deployable model using the SageMaker SDK.

Option 1: Deploy the Evo 2 NIM using SageMaker Studio

The following section outlines how to deploy the EVO 2 NIM using SageMaker Studio.

Getting started with SageMaker Studio

Begin by accessing the AWS Management Console and navigating to the SageMaker AI service. When you’re in the SageMaker AI console, locate Studio in the left navigation panel and choose Open Studio next to your user profile. If you haven’t set up a SageMaker Studio domain yet, then you must create a new domain and user profile first. This launches the web-based SageMaker Studio interface where you can manage all aspects of your ML workflow.

Navigating to model packages

Within SageMaker Studio, look for Models in the left sidebar and choose JumpStart base models tab within the Models interface. This section contains all available model packages in SageMaker JumpStart, including those from the AWS Marketplace

Locating the Evo-2 NIM model

Use the search functionality to find the NVIDIA Evo-2 NIM model by searching for terms such as “Evo-2” or “NVIDIA”. When you locate the model package in the filtered results, choose it to view the Model overview page. This page provides an overview of the model and can have a Notebooks tab that will show a sample notebook that contains an example showing how to use the NIM. You can choose Open in JupyterLab to open the notebook in JupyterLab and use it as a starting point for using the NIM.

Configuring the model deployment

On the model package overview page, choose the Deploy button on the top right to begin the deployment process. You must configure several important settings: provide a unique endpoint name (such as “Evo-2-nim-endpoint”), choose an appropriate instance type (ml.g6e.12xlarge is recommended for optimal performance), set the initial instance count (typically 1 for initial testing), and specify an endpoint configuration name. Review all of these settings carefully before proceeding.

Initiating and monitoring the deployment

After verifying your configuration settings, choose Deploy to start the deployment process for creating a Real-time inferance endpoint. Navigate to the Deployments section and then the Endpoints section in the left sidebar to monitor the deployment progress. The endpoint status initially shows Creating and typically takes 5–10 minutes to complete. You can track the progress and should see the status change to InService once the deployment is successful.

Testing and validation

When your endpoint is deployed and shows the In Service status, you can optionally test it directly through the SageMaker Studio interface. Choose your deployed endpoint from the endpoints list to access the Endpoint summary page. Scroll down and select the Playground tab. If available, you will see two options: Test the sample request and Use Python SDK example code. You can use either option to validate the deployment by using a sample protein sequence. This validates the endpoint is working correctly before integrating it into your applications.

Option 2: Deploy Evo 2 using the SageMaker SDK

In this section we walk through deploying the Evo-2 NIM through the SageMaker SDK. Make sure that you have the account-level service limit for using ml.g6e.12xlarge for endpoint usage as one or more instances. Furthermore, NVIDIA provides a list of supported instance types that support deployment. Refer to the AWS Marketplace listing for the model to see the supported instance types. To request a service quota increase, go to the AWS service quotas.

import sagemaker
import boto3
from sagemaker import ModelPackage, get_execution_role
import json
# Initialize SageMaker session and role
role = get_execution_role()
sagemaker_session = sagemaker.Session()
# Model Package ARN from your AWS Marketplace subscription
# Replace this with your actual Model Package ARN after subscription
model_package_arn = "arn:aws:sagemaker:<region>:<account-id>:model-package/Evo-2-nim-model"
# Create model from AWS Marketplace Model Package
model = ModelPackage(
    role=role, 
    model_package_arn=model_package_arn,
    sagemaker_session=sagemaker_session
)
# Deploy the model to an endpoint
predictor = model.deploy(
    initial_instance_count=1,
    instance_type="ml.g6e.12xlarge",  # Using recommended NVIDIA GPU instance
    endpoint_name="Evo-2-endpoint",
    wait=True
)

Run Inference with Evo 2 SageMaker endpoint

When you have the model, you can use a sample text to do an inference request. NIM on SageMaker supports the OpenAI API inference protocol inference request format. For an explanation of the supported parameters, go to the Evo-2 API documentation.

Real-time inference example

sm_runtime = boto3.client("sagemaker-runtime", region_name=region)

generate_payload = {

 "sequence": "ACGTACGTACGT",

 "num_tokens": 100,

 "temperature": 0.7,

 "top_k": 3,

}

response = sm_runtime.invoke_endpoint(

EndpointName='Evo2-40b-2-1-0',

ContentType="application/json",

Body=json.dumps(generate_payload),

)

result = json.loads(response["Body"].read())

print("Generated DNA:", result["sequence"])
print("Elapsed (ms):", result.get("elapsed_ms"))

Example output:

Generated DNA: ACGTACATATGTTCGTACATTCGCACAGACGCCATTTTGAAAAATGCTTTAAATGGATTCAGAATTGGTCAAAATGCATAAATCCATCAAAATTTTTTTC
Elapsed (ms): 10770

Cleaning up

To avoid unwanted charges, complete the steps in this section to clean up your resources.

Deleting the endpoint from SageMaker Studio

In SageMaker Studio, navigate to the Endpoints section in the left sidebar under Inference to view all your active endpoints. Locate your Evo-2 NIM endpoint in the list and select it to open the endpoint details page. On this page, there is a Delete button. Choose Delete and confirm the deletion when prompted. The endpoint status changes to Deleting and disappears from your endpoints list when the deletion is complete. This process typically takes a few minutes, and when it’s deleted the endpoint stops incurring charges immediately.

Delete the SageMaker endpoint

The SageMaker endpoint that you deployed incurs costs if you leave it running. Use the following code to delete the endpoint if you want to stop incurring charges. For more details, go to Delete endpoints and resources.

# Delete endpoint when done (important for cost management)
predictor.delete_endpoint()

Conclusion

The availability of NVIDIA Evo-2 NIM microservices on Amazon SageMaker Jumpstart represents a significant advancement for researchers and organizations working in drug discovery. This solution provides GPU-accelerated multiple sequence alignments and dramatically speeds up structure prediction pipelines that are critical for protein design and antibody research. Users can implement the flexible deployment options—through SageMaker Studio, or SageMaker SDK—to choose the approach that best fits their workflow and technical expertise. The optimized performance of these NIM microservices, combined with the scalability and security of SageMaker, enables faster time-to-insight while streamlining the deployment of complex biomolecular AI models. We encourage you to try the Evo-2 NIM today and look out for future release of MSA-search and Boltz-2 NIMs to accelerate your drug discovery workflows and use the power of NVIDIA’s specialized microservices on AWS infrastructure.

Building fault-tolerant applications with AWS Lambda durable functions

Rahul Pisal — Fri, 06 Feb 2026 16:54:39 +0000

Business applications often coordinate multiple steps that need to run reliably or wait for extended periods, such as customer onboarding, payment processing, or orchestrating large language model inference. These critical processes require completion despite temporary disruptions or system failures. Developers currently spend significant time implementing mechanisms to track progress, handle failures, and manage resources when waiting for external events, shifting focus from business logic to undifferentiated tasks.

At re:Invent 2025, Amazon Web Services (AWS) launched AWS Lambda durable functions, a new capability extending Lambda’s event-driven programming model with built-in capabilities to build fault-tolerant multi-step applications and AI workflows using familiar programming languages. At its core, durable functions are regular Lambda functions, so your development and operational processes for Lambda continue to apply. However, when you create a Lambda function you can now enable durable execution, so that you can checkpoint progress, automatically recover from failures, and suspend execution for up to one year when waiting on long-running tasks, such as human-in-the-loop processes.

How Lambda durable functions work

When working with standard Lambda functions, your code runs from start to finish in a single invocation. If a failure occurs at any point during the execution, the entire function must be retried by the invoking event source. Any state that needs to be preserved between executions must be explicitly saved and retrieved. This is typically done by using external storage services such as Amazon DynamoDB or Amazon Simple Storage Service (Amazon S3). Furthermore, you must typically guard against duplicate (concurrent) invocations of the same event and have a strategy to safely deploy updates while continuing to process events.

In contrast, with Lambda durable functions, developers use durable operations such as “Steps” and “Waits” in the event handler to checkpoint progress, handle failures, and suspend execution during wait periods without incurring compute charges for on-demand functions. These durable operations and any optional state returned from them are automatically persisted by Lambda in a fully-managed durable execution backend. If failures occur during the execution, or if your function resumes its execution after being paused, Lambda invokes your function again, restoring (replaying) the previous state by executing the event handler from the start, but skipping over completed durable operations. To streamline this checkpoint/replay mechanism for developers, you can use the Lambda durable execution SDK to wrap or annotate your event handler, which enhances the existing Lambda context with several new methods like context.step() and context.wait(). Furthermore, you can use methods such as context.waitForCallback() to wait on external jobs or asynchronous processes, such as “human-in-the-loop” scenarios. The execution is paused until a SendDurableExecutionCallbackSuccess or SendDurableExecutionCallbackFailure response is sent to the Lambda API.

Getting started

Use the AWS Serverless Application Model (AWS SAM) to create a new durable function with sam init with an AWS Quick Start Template. Lambda durable functions are also supported by the AWS Cloud Development Kit (AWS CDK), AWS Command Line Interface (AWS CLI), AWS CloudFormation and other infrastructure as code (IaC) frameworks such as Terraform.

Consider the following function, which performs user onboarding. First, it creates a user profile based on some data, then it sends out an email for verification and waits until the user either confirms the email address, or a 24-hour timeout is reached. Finally, it sends out a confirmation.

import {
  DurableContext,
  withDurableExecution,
} from '@aws/durable-execution-sdk-js';
export const handler = withDurableExecution(
  async (event: OnboardingEvent, context: DurableContext) => {
    try {    
      // Create user profile
      const profile = await context.step("create-profile", async () =>
        createUserProfile(event.email, event.name)
      );
      // Wait for email verification via callback
      const verification = await context.waitForCallback(
        "wait-for-email-verification",
        async (callbackId) => {
          // Send email to user and pass callbackId
          await sendVerificationEmail(profile, callbackId);
        },
        {
          timeout: { hours: 24 } 
        }
      );
      // Send confirmation and welcome email
      const result = await context.step("complete-onboarding", async () => {
        if (!verification || !verification.verified) 
     return { ...profile, status: 'failed' };
        await sendWelcomeEmail(profile.email, profile.name);
        return { ...profile, status: 'active' };
      });
      return result;
    } catch (error) {
      // omitted 
    }
  }
);

Durable functions have built-in and fully customizable error handling for steps. For example, if the profile was successfully created and verified, but a temporary error occurred when sending out the confirmation, then the step is retried. The retry skips over any previously completed checkpoints, such as the profile creation and callback. Only the code within the send confirmation step is run again.

Next, you update the AWS SAM template to include your durable function. You create a Lambda durable function by including the DurableConfig setting for your function. Note that you currently cannot add a durable configuration to a function that was originally created without it. The ExecutionTimeout defines after which time the durable execution times out to protect against runaway or deadlock application bugs. This setting is separate from the invocation timeout, which defines for how long a single invocation can run. The maximum invocation timeout for a single function invocations remains unchanged at 15 minutes. With Lambda durable functions, you will typically see multiple invocations per durable execution, such as when using the wait capabilities in the SDK or automatic retries. You can set the ExecutionTimeout for up to one year when using asynchronous invocations.

The RetentionPeriodInDays defines how long the execution data of a durable execution is available to you after executions complete.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
 
Resources:
  UserOnboardingFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: UserOnboardingFunction
      CodeUri: ./src
      Handler: index.handler
      Runtime: nodejs24.x
      Architectures:
        - x86_64
      MemorySize: 256
      Timeout: 60		   // Timeout for an individual invocation
      DurableConfig:		   // This makes the function a durable function
        ExecutionTimeout: 90000 // 25h timeout for the durable execution overall
        RetentionPeriodInDays: 7 
UserOnboardingFunctionRole:
    Type: AWS::IAM::Role
    // omitted for brevity

You must include the necessary permissions for your function. For example, the AWSLambdaBasicDurableExecutionRole managed policy only allows the minimal AWS Identity and Access Management (IAM) actions to create/retrieve checkpoints and logs to increase security. Therefore, it does not include permissions to invoke other (durable) functions or manage callbacks. Refer to the documentation for more details.

Testing locally

Before deploying your function, you can test it locally using AWS SAM local invoke.

AWS SAM locally invokes your function and runs the event handler until it reaches the context.waitForCallback(). To complete callbacks, AWS SAM offers new commands to interact with your durable functions. In this example, you send a Success response to complete the callback. You can also include relevant data in the response. You can send the response directly using the on-screen guide or using another AWS SAM CLI command from another process.

sam local callback succeed <your-callback-id> --result '<your data>'

To inspect an execution, you can use AWS SAM to retrieve the durable execution history of your function, which includes details about steps, callbacks, and wait durations, as shown in the following example code.

sam local execution history <execution-arn>

Depending on your use case, you can instead send a Failure response to a callback and handle those errors in your code. For example, by performing compensation logic in a subsequent step:

sam local callback fail <your-callback-id> --error-data '<your data>'

Now that you have verified that your function works as intended, deploy it to AWS using sam deploy command.

Best practices and considerations

Invoking a Lambda durable function requires a qualified Amazon Resource Name (ARN), such as an alias or version. We recommend that you don’t use the $LATEST qualifier except for rapid prototyping or local testing. Using explicit versions ensures that replays always happen with the same code with which the execution was started. This is to ensure deterministic execution and prevent inconsistencies when updating your function code during executions.

We recommend bundling the durable execution SDK with your function code using your preferred package manager. The SDKs are fast-moving, so you can update dependencies as new features become available.

There are other durable operations in the Lambda durable functions SDK that you can use to build your application:

waitForCondition(): Pauses the execution of your function until a condition is met. For example, the status of a job polled with an API. For this to work, you provide the waitStrategy and a check function to poll the status.
parallel(): Runs multiple durable operations in parallel within the same function, with configurable options such as the maximum number of concurrent branches and desired failure behavior. This streamlines managing durability and checkpointing for simultaneous asynchronous actions.
map(): Creates a durable operation and checkpoint for each item of an array, based on the provided mapping function. The items are processed concurrently.
invoke(): Invokes another Lambda function and waits for its result. The SDK creates a checkpoint, invokes the target function, and resumes your function when the invocation completes. This enables function composition and workflow decomposition.

Refer to the developer guide for more details.

Lambda compute charges apply to all invocations, including any replays. When using wait operations, the function suspends execution and, for on-demand functions, doesn’t incur duration charges until execution resumes. You’re also charged for durable operations, data written, and data retention. To learn more about Lambda durable functions pricing, refer to the Lambda pricing page.

For the latest Region availability, visit the AWS Capabilities by Region page.

Conclusion

AWS Lambda durable functions extends the Lambda programming model to streamline building fault-tolerant, long-running applications using familiar programming patterns. You can use Lambda durable functions to write multi-step workflows in your preferred programming language, using built-in methods that automatically handle progress checkpointing and error recovery. This streamlines your architectures so that you can focus on your business logic, and optimize cost by charging only for active compute time.

You can build durable functions for Python or Node.js based Lambda functions using the Lambda API, AWS Management Console, AWS CLI, AWS CloudFormation, AWS SAM, AWS SDK, and AWS CDK.

To get started, visit the Lambda Developer Guide or watch the re:Invent breakout session.

Serverless ICYMI Q4 2025

Julian Wood — Fri, 30 Jan 2026 15:23:57 +0000

Stay current with the latest serverless innovations that can transform your applications. In this 31st quarterly recap, discover the most impactful AWS serverless launches, features, and resources from Q4 2025 that you might have missed.

In case you missed our last ICYMI, check out what happened in Q3 2025.

2025 Q4 calendar

Serverless at re:Invent 2025

This post covers the biggest serverless announcements from re:Invent 2025, highlighting key feature updates that can improve your applications, and shares valuable resources to keep you informed.

AWS re:Invent 2025 had more than 60,000 in-person attendees and more than 2 million online viewers for the keynotes. The event featured 3,500 sessions from 3,000 speakers, which included information on 530 AWS service and feature announcements.

Keynote Igniting the serverless movement

The serverless content consisted of two tracks: Containers and Serverless (CNS) and Application Integration (API). These tracks included 150 unique sessions watched in-person by more than 16,000 attendees. There were developer-focused experiences including a Road to re:Invent Hackathon, AWS Builder Loft, and Builders Arena. Serverlesspresso, the coffee shop powered by serverless technology, operated in two locations during the event: the Expo Hall and the certification lounge.

Serverless and developer community photo

Find a curated list of serverless videos on Serverless Land YouTube.

AWS Lambda durable functions

Managing state across multi-step serverless workflows has traditionally required complex external orchestration tools. AWS Lambda durable functions expand how developers can use Lambda. You can now build reliable multi-step applications and AI workflows directly within Lambda.

AWS Lambda durable functions code

Durable functions automatically checkpoint progress by saving the current state and completed steps at key points during execution. This allows them to suspend execution for up to one year during long-running tasks and recover from failures by resuming from the last checkpoint rather than restarting from the beginning, all without requiring additional infrastructure management.

Developers can now build in Python or TypeScript, wrap calls in steps with automatic retries and checkpointing. You can use waits to suspend execution for minutes, hours, or even up to a year without paying for idle compute. Durable functions use a replay mechanism to maintain state and handle failures gracefully. The replay mechanism works by re-executing your function code from checkpoints when recovering from failures, ensuring state consistency without data loss. This also means you don’t need complex external orchestration tools for many use cases. This can be helpful for AI workflows and multi-step applications where you need reliable state management without managing external infrastructure.

For more information, read the launch blog post and watch the re:Invent Breakout Session video: Deep Dive on AWS Lambda durable functions (CNS380)

AWS Lambda Managed Instances

Lambda now offers Lambda Managed Instances, a new compute option that combines Amazon EC2 flexibility with fully managed infrastructure. AWS automatically handles instance provisioning, scaling, and maintenance while allowing access to the full range of EC2 capabilities, including Graviton4, network-optimized instances, and other specialized compute options.

AWS Lambda Managed Instances configuration

Your functions run on dedicated EC2 capacity from your account, in your own Amazon Virtual Private Cloud (Amazon VPC). AWS still manages the operational overhead, including OS patching, load balancing, and auto-scaling. This gives you access to specialized hardware options while maintaining the serverless operational model. You can further improve costs by using EC2 pricing models, including Compute Savings Plans and Reserved Instances for Lambda workloads. Each instance can handle multiple concurrent requests, making this particularly valuable for high-volume, steady-state workloads where predictable pricing and specific hardware requirements matter.

For more information, read the launch blog post and watch the re:Invent Breakout Session video: Lambda Managed Instances: EC2 Power with Serverless Simplicity (CNS382).

Other Lambda announcements

Multi-tenant SaaS applications face challenges like data leakage between tenants and noisy neighbor effects where one tenant’s workload impacts others. They also struggle with implementing custom isolation mechanisms. Tenant isolation mode addresses these by processing function invocations in separate execution environments for each tenant. This manages tenant-level compute environment isolation automatically.

AWS Lambda tenant isolation

Lambda adds Provisioned Mode for Amazon SQS event-source mappings, providing predictable performance and reduced cold starts for high-throughput SQS processing workloads.

You can now send up to 1 MB of data in asynchronous Lambda invocations, increased from 256 KB, helping you build more complex data processing scenarios.

Lambda functions now support IPv6 networking, so you don’t need NAT Gateways when accessing the internet or other AWS services from VPC-connected functions.

Lambda internet connectivity through a NAT Gateway (IPv4) and Lambda internet connectivity through an egress-only internet gateway (IPv6).

Lambda Rust support is now generally available, moving from experimental status. This is backed by AWS Support and the Lambda availability SLA.

Lambda has expanded its runtime support by adding Python 3.14, Node.js 24, and Java 25 as both managed runtimes and container base images, providing access to the latest language features and ensuring long-term support.

Amazon ECS

Amazon Elastic Container Service (Amazon ECS) Express Mode streamlines the deployment and management of containerized applications by automating the infrastructure setup that traditionally slows down developers.

Amazon ECS Express Mode deployment

This means you can focus on building applications while deploying with confidence using AWS best practices. Express Mode lets you deploy production-ready containerized web applications and APIs with a single command. This automatically handles domains, networking, load balancing, AWS Identity and Access Management (IAM) roles, and auto-scaling through simplified APIs. When your applications evolve and require advanced features, you can seamlessly configure and access the full capabilities of the resources, including Amazon ECS. Learn more from the launch blog post.

Amazon ECS announced a public preview of a fully managed MCP server, enabling AI-powered experiences for development and operations. The Model Context Protocol (MCP) server provides enterprise-grade capabilities like automatic updates and patching, centralized security through AWS IAM integration, comprehensive audit logging via AWS CloudTrail, and the proven scalability, reliability, and support of AWS.

Amazon Elastic Container Registry (ECR) managed container image signing enhances your security posture and eliminates the operational overhead of setting up signing. Container image signing allows you to verify that images are from trusted sources. ECR automatically signs images as they are pushed using the identity of the entity pushing the image. Signing operations are logged through CloudTrail for full auditability.

Amazon API Gateway

Amazon API Gateway allows you to improve the responsiveness of your REST APIs by progressively streaming response payloads back to the client. With this new capability, you can use streamed responses to enhance user experience when building LLM-driven applications (such as AI agents and chatbots), improve time-to-first-byte (TTFB) performance for web and mobile applications, stream large files, and perform long-running operations while reporting incremental progress using protocols such as server-sent events (SSE).

Amazon API Gateway streaming

API Gateway introduces private integration with Application Load Balancers (ALBs). You can use this to expose your VPC-based applications securely through REST APIs without exposing your ALBs to the public internet.

You can also now configure enhanced TLS security policies on API endpoints and custom domain names, providing you with greater control over the security posture of your APIs.

Amazon EventBridge

Amazon EventBridge introduced an enhanced visual rule builder that helps developers discover and subscribe to events from custom applications and over 200 AWS services. The console-based interface integrates the EventBridge schema registry with a comprehensive event catalog and intuitive drag-and-drop canvas that simplifies building event-driven applications. Developers can browse and search through events with readily available sample payloads and schemas without having to hunt through individual service documentation. The schema-aware visual builder guides developers through creating event filter patterns and rules, reducing syntax errors and accelerating development time.

EventBridge also allows targeting SQS fair queues.

AWS Step Functions

AWS Step Functions allows for enhanced local testing through the TestState API, providing programmatic access to comprehensive testing capabilities without deploying to AWS. This helps you build automated test suites that validate your workflow definitions locally on your development machines. Test error handling patterns, data transformations, and mock service integrations using your preferred testing frameworks.

There is also a new metrics dashboard, giving you visibility into your workflow operations at both the account and state machine levels.

Other announcements

Savings Plans flexible pricing model extends to AWS managed database services with the launch of Database Savings Plans. This helps reduce database costs by up to 35% when committing to a consistent amount of usage ($/hour) over a 1-year term. Savings automatically apply each hour to eligible usage across supported database services, and additional usage beyond the commitment is billed at on-demand rates.

Amazon DynamoDB now supports multi-attribute composite keys in global secondary indexes. You no longer need to concatenate values into synthetic keys manually, which sometimes results in the need to backfill data before adding new indexes. Instead, you can create primary keys using up to eight existing attributes, making it easier to model diverse access patterns and adapt to new query requirements.

Amazon Bedrock introduced AgentCore with quality evaluations and policy controls for deploying trusted AI agents at scale.

Bedrock also added 18 fully managed open weight models, expanding AI model options for developers.

The Strands Agents SDK is an open source framework that takes a model-driven approach to building and running AI agents in just a few lines of code. TypeScript support is now available in preview so you can choose between Python and TypeScript for building Strands Agents.

Amazon S3 Vectors became generally available. S3 Vectors delivers purpose-built, cost-optimized vector storage for AI agents, inference, Retrieval Augmented Generation (RAG), and semantic search at billion-vector scale.

Serverless blog posts

October

November

Serverless Office Hours

Join our livestream every Tuesday at 11 AM PT for live discussions, Q&A sessions, and deep dives into serverless technologies. Episodes are available on-demand at serverlessland.com/office-hours.

October

Oct 7 – Amazon API Gateway Routing Rules
Oct 14 – Amazon DynamoDB Global Tables
Oct 21 – Building agents with Amazon Bedrock AgentCore
Oct 28 – “What’s new with Observability

November

December

Dec 9 – AWS Lambda Managed Instances
Dec 16 – AWS Lambda durable functions

Still looking for more?

The Serverless landing page has overall information about building serverless applications. The Lambda resources page contains case studies, webinars, whitepapers, customer stories, reference architectures, and even more Getting Started tutorials.

You can also follow the Serverless Developer Advocacy team to see the latest news, follow conversations, and interact with the team.

Julian Wood: @julian_wood, https://www.linkedin.com/in/julianrwood/
Eric Johnson: @edjgeek, https://www.linkedin.com/in/singledigit/
Gunnar Grosch: @GunnarGrosch, https://se.linkedin.com/in/gunnargrosch
Erik Hanchet: @ErikCH, https://www.linkedin.com/in/erikhanchett/
Salih Gueler: @salihgueler, https://www.linkedin.com/in/salihgueler/
Marcia Villalba: @mavi888uy, https://www.linkedin.com/in/marciavillalba

And finally, visit Serverless Land for all your serverless needs.

More room to build: serverless services now support payloads up to 1 MB

Anton Aleksandrov — Thu, 29 Jan 2026 22:16:14 +0000

To support cloud applications that increasingly depend on rich contextual data, AWS has raised the maximum payload size from 256 KB to 1 MB for asynchronous AWS Lambda function invocations, Amazon Simple Queue Service (Amazon SQS), and Amazon EventBridge. Developers can use this enhancement to build and maintain context-rich event-driven systems and reduce the need for complex workarounds such as data chunking or external large object storage.

Overview

Modern cloud applications rely on context-rich, structured data to drive intelligent behavior. Large language model (LLM) prompts, telemetry signals, personalization data, machine learning (ML) outputs, and user interaction logs are no longer simple strings. Instead, they’re typically complex, nested JSON or YAML objects carrying meaningful context. Previously, developers working with serverless services such as Amazon SQS, Lambda (asynchronous invocations and Amazon SQS event-source mapping), or EventBridge had to carefully manage their data to fit within the 256 KB payload size limit. This commonly meant chunking larger payloads, externalizing payloads to object stores such as Amazon S3, or using data compression. These workarounds added complexity and latency, creating edge cases that were difficult to monitor and debug.

With the recent launches, you can now transmit payloads up to 1 MB, significantly reducing the need for complex data chunking and architectural workarounds. This increased capacity streamlines design patterns, reduces operational overhead, and makes event-driven systems more intuitive to build and maintain. Developers can now include richer data in single payloads—from detailed LLM prompts and full system states to comprehensive context and complete transaction histories.

The new 1 MB payload size limit applies to asynchronous Lambda function invocations, whether you trigger them using either SQS event-source mapping, AWS Command Line Interface (AWS CLI), AWS SDKs, Lambda Invoke API, or AWS services such as EventBridge. The increased limit also extends to all messages and events flowing through Amazon SQS queues and EventBridge Event Buses.

Getting started

There’s nothing you need to do to get started. This enhancement is automatically applied to all new and existing Lambda functions, SQS queues, and EventBridge Event Buses.

If you were previously chunking data at 256KB (or lower) threshold, then you might need to make changes to your service configurations or business logic code to start using the new limit. For example, if you’ve explicitly set Amazon SQS MaximumMessageSize attribute, then you might need to adjust it to a new desired value. Larger payloads might also result in higher costs, as described in the following section.

Real-world example: rich event context in agentic event-driven architectures

Event-driven architectures allow services to operate independently without centralized coordination. In these systems, comprehensive event context is essential. With the increased 1 MB payload limit, events can now carry more comprehensive data—from user profiles and order details to historical interactions. This enables services such as inventory, shipping, and notifications to act autonomously.

Consider the following example. In hospitality and quick-service industries, customer satisfaction depends on timely, thoughtful service recovery. When a guest submits negative feedback through a survey, review, or complaint form, service teams must gather context, interpret the issue, and craft a response. Traditionally, this meant manually piecing together visit logs, loyalty data, and prior complaints. Now, this can be fully automated using an AI agent powered by AWS serverless services and Amazon Bedrock, as shown in the following figure.

Figure 1: Customer feedback processing pipeline

The workflow:

Receive: A new review is submitted through the Review application and emitted as an event to EventBridge Event Bus.
Detect: Event Bus delivers the event to downstream Feedback analysis agent. The agent running in a Lambda function recognizes the review as low-rating or complaint.
Enrich: The agent collects the guest’s visit metadata, booking details, loyalty activity, and complaint history using attached MCP tools into a single structured JSON payload (up to 1 MB).
Queue: The payload is sent to an SQS queue for further asynchronous processing by downstream components.
Generate: A separate Lambda function polls messages from Amazon SQS and invokes an Amazon Bedrock model to analyze the full complaint context, draft a personalized response, suggest a gesture (such as a refund or credit), and classify issue severity.
Deliver: The message is logged and sent to the customer, and to the service team for further analysis.

This use case demonstrates the importance of having a rich context: current and previous visits details, loyalty tier, prior interactions, and feedback history. Previously, teams had to offload pieces of context to Amazon S3 and reference them externally, adding latency and architectural complexity. The new 1 MB payload size means that all this information can be transported together, improving the serverless agentic workflow efficiency and streamlining maintenance.

Best practices when using large payloads

The following sections outline best practices that you should apply when using larger payloads.

Performance considerations

Monitor Lambda function memory usage carefully when working with larger payloads, because parsing and processing complex JSON objects can increase memory usage and execution duration. Test your systems thoroughly under load, especially for high-throughput applications, by benchmarking with realistic payload sizes and traffic patterns. Although the payload limit has increased to 1 MB, the Lambda 15-minute timeout and memory limits remain unchanged. When applicable, you can use compression to process even larger datasets efficiently, but remember to account for the added CPU overhead of compression and decompression in your performance calculations. Read the Monitoring best practices for event delivery with Amazon EventBridge post for more best practices to tune your event-driven architectures performances.

Operational guidelines

Configure dead-letter-queues (DLQ) to make sure that failed messages are retained for inspection and troubleshooting. This becomes especially important with larger payloads, because debugging complex data structures necessitates access to the complete message context. Implement robust error handling and retries to manage transient failures, particularly when processing rich payload content that may contain nested structures or complex relationships.

To further optimize throughput, you can batch similar smaller events together into a single payload. However, avoid mixing unrelated events and maintain clear boundaries between different business domains and processes.

Always make sure that your downstream dependencies are capable of handling larger payloads.

When to use external storage

Even with the increased 1 MB payload limit, there are scenarios where patterns such as claim check remain a sound architectural choice. These patterns involve storing a full payload in an external system, such as Amazon S3, and passing a lightweight reference through your event stream. This approach continues to provide value when payloads exceed the new limit, when data needs to be reused by multiple consumers, or when strict governance, traceability, and security requirements are involved. For example, audit logs, image metadata, or large ML inference inputs may still surpass the 1 MB boundary, even when compressed. Instead of risking truncation or fragmentation, a claim check enables consistent, scalable access to the complete data set.

You can use open source libraries such as the Kafka sink connector for EventBridge and Amazon SQS Extended Client Library (available for Python and Java) that abstract complexities of storing large objects in external storage.

Cost management

Although larger payloads enable richer context in your applications, logging full payloads can increase storage and processing costs. Services such as CloudWatch Logs charge based on data volume, thus implementing selective logging, payload truncation, or sampling becomes crucial for high-volume events. Consider logging only essential fields or implementing smart sampling strategies based on business importance.

For full payload archival and retention, evaluate cost-effective storage solutions such as Amazon S3 with appropriate lifecycle policies. This can include moving older logs to cheaper storage tiers or implementing automated cleanup procedures for non-critical data. Balance your retention needs with cost optimization by defining clear policies for what data needs to be kept and for how long.

Review the pricing pages for AWS Lambda, Amazon EventBridge, and Amazon SQS to learn about the costs of delivering and processing events and messages.

Conclusion

The increase in maximum payload size from 256 KB to 1 MB enables developers to build more efficient distributed architectures. You can use this enhancement to transport richer context in event and message payloads, reducing the need for complex workarounds that previously added architectural complexity and operational overhead. This added room to transmit rich context means that you can streamline your workflows, improve observability, and reduce architectural complexity whether using choreography or orchestration patterns.

Go to the developer guides for AWS Lambda, Amazon EventBridge, and Amazon SQS, to learn more about how to take advantage of this update.

To learn more about serverless architectures, visit Serverless Land.

Simplify network segmentation for AWS Outposts racks with multiple local gateway routing domains

Brianna Rosentrater — Fri, 16 Jan 2026 18:49:35 +0000

AWS now supports multiple local gateway (LGW) routing domains on AWS Outposts racks to simplify network segmentation. Network segmentation is the practice of splitting a computer network into isolated subnetworks, or network segments. This reduces the attack surface so that if a host on one network segment is compromised, the hosts on the other network segments are not affected. Many customers in regulated industries such as manufacturing, health care and life sciences, banking, and others implement network segmentation as part of their on-premises network security standards to reduce the impact of a breach and help address compliance requirements. Some AWS services also have network requirements that specify certain IP ranges to be used for endpoints, and may or may not support customers bringing their own IP pool (also called CoIP routing, see How to choose between CoIP and Direct VPC routing (DVR) modes on AWS Outposts rack for more information). Customers want the flexibility to use both routing modes (CoIP and DVR) on the same logical Outpost. With this new feature, AWS Outposts racks now support multiple LGW routing domains to meet subnetwork isolation and cloud service network requirements in an on-premises environment. For example, a leading automotive company deploys latency-sensitive manufacturing workloads on Outposts racks in a multi-AZ architecture for resiliency. This feature provides traffic separation between routing domains and enables both customer-owned IP (CoIP) and direct VPC routing (DVR) modes on the same logical Outpost.

In this post you will learn how to use multiple LGW routing domains on Outposts racks and considerations for implementation.

Overview

With the introduction of multiple LGW routing domains on Outposts, you can now create multiple routing domains and associate one or more VLANs with each routing domain. This allows you to integrate your Outposts rack into your existing on-premises network schema. Each LGW routing domain will have a unique LGW Virtual Interface (VIF) Group and an LGW Route Table, enabling logical network traffic isolation. You can have a mix of up to 10 active routing domains with route tables using either DVR or CoIP routing mode, and you can make changes to these routing domains as needed in a self-service fashion allowing for network flexibility as architectures are updated over time. These settings can be found in the AWS Outposts console under the Networking tab in the menu.

The following diagram shows an example of 3 VPCs, each with at least 1 subnet on the Outpost rack, and each VPC corresponds to its own routing domain. Each routing domain can then be associated with one or more VLANs, and one or more VPCs. You can only associate a VPC to one LGW routing domain per Outpost.

Figure 1 – Architecture diagram showing 3 routing domains

Walkthrough

Before creating a LGW routing domain, first you’ll need to create an LGW VIF group and an LGW route table. A local gateway routing domain is the association of a local gateway route table and local gateway VIF group. Each VIF group can be associated with one or more VLANs, but a route table can only be associated with one VIF group.

To create a LGW VIF Group, navigate to the AWS Outposts console, go to LGW virtual interfaces groups, and select Create VIF group. Enter your VIF details which include BGP and VLAN routing information, you must create 4 LGW VIFs per VIF group.

Figure 2 – Creating VIF group for RD1 routing domain

After creating your VIF group, create a LGW route table. You’ll have the option to use Direct VPC Routing (DVR) or Customer-owned IP address pool (CoIP) routing. If CoIP routing is selected, you’ll have the option to enter your CIDR before creating. A LGW route table’s routing mode cannot be changed after creating. However, you can disassociate a LGW route table from a VIF group and attach a new route table if you need to change the routing mode of a VIF group.

Figure 3 – Creating LGW route table for RD1 routing domain

After you’ve created your LGW route table and VIF group, you can proceed to the final step which is to create your LGW routing domain where you will associate the LGW route table and VIF group.

Figure 4 – Creating LGW routing domain for RD1

You can view and create up to 10 active routing domains through the AWS Outposts console under the Networking tab.

Figure 5 – Local Gateway (LGW) routing domains

Considerations

Multiple LGW routing domains feature is only available on second-generation Outposts racks.
Avoid overlapping IP addresses across subnetworks and local routing domains as those can create IP routing conflicts.
A VIF group can only be associated to one LGW route table/routing domain at a time. A routing domain is the association of a VIF group and LGW route table.
LGW routing domain will allow for logical local network traffic isolation, however all traffic will still travel across your local gateway Link Aggregation Control Protocol (LACP) Link Aggregation Group (LAG) to uplink into your on-premises network.
Additional network isolation can be achieved through Virtual Routing and Forwarding (VRF) on Cisco platforms or Routing Instances on Juniper equipment, providing logical separation of routing tables and enabling secure multi-tenancy within the same physical infrastructure.
You can only associate a VPC to one LGW routing domain per Outpost. You can self-serve to change VPC association as needed. Multiple on-premises VLANs can be connected to a single routing domain.

Conclusion

This post demonstrated how to configure multiple local routing domains on Outposts racks to integrate into your on-premises network. For more information see LGW routing domains section in the AWS Outposts user guide. Reach out to your AWS account team to learn more about Outposts racks network configuration options.

In addition to multiple LGW routing domains, we have also announced several updates to Outposts in the past week to help you meet digital sovereignty and local data processing needs. To learn more, read the following announcements:

To discuss Outposts with an expert on any of these topics, submit this form.

Optimizing storage performance for Amazon EKS on AWS Outposts

Arun Kumar — Tue, 13 Jan 2026 18:57:12 +0000

Amazon Elastic Kubernetes Service (Amazon EKS) on AWS Outposts brings the power of managed Kubernetes to your on-premises infrastructure. Use Amazon EKS on Outposts rack to create hybrid cloud deployments that maintain consistent AWS experiences across environments. As organizations increasingly adopt edge computing and hybrid architectures, storage optimization and performance tuning become critical for successful workload deployment.

Outposts extend AWS infrastructure, services, APIs, and tools to virtually any datacenter, co-location space, or on-premises facility. In this blog post you will learn about your storage options and their performance characteristics which is essential for building resilient, high-performing applications using Amazon EKS on Outposts.

Amazon EKS on Outposts deployment options

The following two sections outline the differences between Amazon EKS extended and local cluster deployment options available on Outposts.

Amazon EKS extended cluster architecture

Amazon EKS extended clusters on Outposts provide a powerful solution for organizations seeking to use the benefits of Kubernetes while maintaining certain workloads on-premises, as shown in the following figure. This hybrid architecture allows businesses to extend their EKS clusters from the AWS Cloud to their own data centers or edge locations using Outposts. The Kubernetes control plane remains in the AWS Region, providing centralized management and benefiting from the AWS infrastructure in the cloud and on the Outpost.

Outposts is designed to be a connected service, and needs reliable network connectivity to the AWS Region using the Outposts service link.

Figure 1 : Extended cluster

Amazon EKS local cluster architecture

Amazon EKS local clusters deploy the Kubernetes control plane on your Outpost, as shown in the following figure. This provides greater network resilience against outages as cluster operations run entirely on the Outposts and reduces the dependency on network connectivity to the AWS Region. Having the Kubernetes control plane hosted on your Outpost also reduces latency for cluster operations.

Figure 2: Local cluster

Storage options for Amazon EKS extended clusters on Outposts

Persistent Volumes (PV) and Persistent Volume Claims (PVC) serve as a critical abstraction layer in Kubernetes, separating the storage consumption details from storage provisioning, and allowing administrators to manage storage resources independently from how applications consume them. PVs and PVCs make sure of data persistence across pod restarts and rescheduling events, making them essential for applications that need to maintain state, such as databases, file storage systems, and other data-intensive workloads. The abstraction provided by PV and PVC enables platform-agnostic storage management, where applications can request storage through PVCs without needing to know the underlying storage implementation details. PVs and PVCs support dynamic provisioning through Storage Classes, allowing for automated storage allocation based on application demands, while also providing features such as access modes, capacity management, and reclaim policies to effectively manage the storage lifecycle in a Kubernetes cluster.

Integrating Amazon EBS with Amazon EKS

Amazon Elastic Block Store (Amazon EBS) provides high-performance block storage that’s ideal for low-latency applications providing consistent performance. When deployed on Outposts racks, EBS volumes are stored on the Outposts hardware, providing significant performance advantages over network-attached storage solutions, as shown in the following figure.

Figure 3 : Integrating Amazon EBS with Amazon EKS on Outposts

Benefits and use cases

Storage: EBS volumes on Outposts racks provide data access without dependency on external connectivity.
Performance: Local storage delivers consistent latency and high IOPS/throughput.
Cost: On-premises storage eliminates data transfer costs and reduces bandwidth needs, lowering the total cost of ownership.

Implementation considerations

Consider the following when using EBS on Outposts rack:

EBS volumes on Outposts are tied to a single rack and the availability zone the Outpost is homed to, needing applications to address single-point-of-failure risks.
Protect data using EBS snapshots in the parent Region and schedule regular backups.
Capacity on Outposts is finite, monitor Outposts storage usage and plan expansions proactively to avoid insufficient capacity errors.

Refer to Dynamic Volume Provisioning to learn more about deploying pod with the EBS volume attached.

Amazon EFS with Amazon EKS

Amazon Elastic File System (Amazon EFS) provides scalable, shared file storage that can be accessed across multiple AWS Availability Zones (AZs) and on-premises environments. Although Amazon EFS with Amazon EKS on Outposts maintains the same setup procedures as standard cloud deployments, there is a critical dependency on the service link connection between your Outposts and the AWS Region. Amazon EFS is not a locally supported service on Outposts, so connectivity to the AWS Region is required to use this service with your Outpost.

Amazon EFS allows multiple pods to concurrently access shared file systems. It is well-suited for applications that need collaborative data access, content management, and distributed processing workloads.

Amazon EFS as a persistent storage solution for Amazon EKS extended cluster instances

Amazon EFS as a PV for your Amazon EKS extended cluster operates through a hybrid architecture where the Amazon EFS file system resides in the Region, but mount points can be created on the worker nodes running on Outposts subnets through the service link as shown in the following figure.

Figure 4 : Amazon EFS as a persistent storage solution for extended clusters

Benefits and use cases

Shared storage capabilities: multiple pods can access a centralized file system, enabling shared data, code, and assets across instances.
Scalability: storage capacity and performance automatically scale with usage, eliminating manual provisioning and upfront planning.
Compliance: Amazon EFS provides full file system features and compatibility for traditional applications, such as locking, permissions, and directory structure.

Challenges and limitations

Consider the following when using Amazon EFS with Outposts:

Network latency: file access involves network traversal to Amazon EFS in the Region, adding more latency and making small or metadata operations potentially slow for latency-sensitive applications.
Throughput: aggregate throughput is restricted by the available bandwidth on the service link between the Outposts and AWS Region. This impacts concurrent access and large file transfers during peak usage.
Dependency on AWS Region connectivity: Amazon EFS needs continuous connectivity to the parent Region. Disruptions may affect file system availability, operations, and disaster recovery processes.
Data Transfer charges: Since EFS is in AWS Parent region and EKS worker nodes and pods are in Outpost additional charges are applicable.

You can refer to Amazon EFS Features and When to Choose Amazon EFS for more detailed insights into its capabilities and use cases.

Deploying pods on extended clusters using Amazon EFS as PV

Refer to Use Elastic File System Storage with Amazon EFS for deployment guidance. Note, Create Amazon EFS mount targets in subnets that are in the same Availability Zone (AZ) as the Outposts subnets.

Amazon S3 with Amazon EKS extended cluster

Amazon Simple Storage Service (Amazon S3) on Outposts delivers local object storage on your Outposts, allowing applications to use Amazon S3 APIs for storing and retrieving data while keeping it onsite. It is ideal for workloads that need Amazon S3 compatibility, low latency access to object data, and local data residency.

You should use Amazon S3 access point Amazon Resource Names (ARNs) and not bucket ARNs for proper integration with Amazon EKS workloads.

Learn more about Amazon S3 on Outposts.

Figure 5 : Amazon S3 with Amazon EKS extended cluster on Outposts

Benefits and use cases

Data archiving and compliance: Enables cost-effective, locally retained storage for logs, audit trails, regulatory compliance, backups, and sensitive healthcare data with strict residency requirements.
Content distribution and media: Provides ultra-low latency local storage for serving static content, media streaming, digital asset management, and gaming asset delivery.
Data lake and analytics: Supports local data processing for analytics, ETL, machine learning (ML), real-time Internet of Things (IoT) data handling, and business intelligence with reduced latency and transfer costs.
Application integration: Seamlessly integrates with Amazon S3 compatible apps for backup, synchronization, microservices storage, API-driven workflows, and container image management on-premises.

Refer to How is Amazon S3 on Outposts different from Amazon S3 and the Amazon S3 on Outposts documentation to learn more.

Deploying pods on extended clusters using Amazon S3 as PV

Step 1: Create Amazon S3 on Outposts bucket
Step 2: Create Amazon S3 Access Point (necessary for Amazon EKS integration)
Step 3: Configure IAM roles and policies
Step 4: Install Amazon S3 CSI driver
Step 5: Deploying your pod with Amazon S3 volume attached
Step 6: Complete Amazon S3 configuration with Kubernetes

Refer to the documentation Static Provisioning on Outposts bucket for more details on Step 5.

Best practices for optimizing performance

Optimizing performance starts with selecting the right storage type for your workload: Amazon EBS for low-latency, high-throughput block storage; Amazon EFS for shared POSIX-compliant file systems; and Amazon S3 for scalable object storage with API compatibility. Ensure proper volume sizing, monitor usage proactively, and configure CPU and memory requests accurately to balance performance and efficiency—auto scaling and QoS classes can further optimize resource management. Improve data locality by using local storage, apply caching with intelligent eviction, and design for efficient, asynchronous, and compressed data access patterns.

Monitoring and observability

Monitoring key performance metrics is essential to maintain storage efficiency and application reliability. For Amazon EBS, track IOPS, throughput, latency, burst balance, queue depth, and snapshot performance to avoid degradation—see the Amazon CloudWatch metrics for Amazon EBS for the full list. For Amazon EFS, monitor total I/O, throughput, client connections, metadata operations, burst credits, and Regional data transfers to support effective capacity planning—refer to CloudWatch metrics for Amazon EFS. For Amazon S3, observe request and error rates, data transfer, storage usage, latency, multipart upload efficiency, and access patterns to optimize performance and cost—see Metrics and dimensions.

Security considerations

Strong security practices are critical for Amazon EKS on Outposts. Use AWS Key Management Service (AWS KMS) for Amazon EBS encryption, encrypt Amazon EFS data at rest and in transit, and enable server- or client-side encryption for Amazon S3. Enforce TLS for all data transfers and apply key rotation with compliance controls. Implement least privilege IAM policies, scoped roles, and Kubernetes Role-Based Access Control (RBAC) for granular pod access. Secure traffic with security groups and NACLs, and maintain audit logs for all storage operations.

Cost optimization strategies

Manage storage costs by right-sizing volumes, automating lifecycle policies, selecting appropriate storage classes, monitoring data transfer, and using de-duplication and compression where applicable. Lower operational expenses through automated backups, infrastructure as code (IaC), monitoring automation, leveraging managed services, applying cost allocation tags, and conducting regular usage reviews.

Conclusion

Amazon EKS on Outposts empowers organizations to build hybrid applications with storage options that align to performance, compliance, and data residency needs. By selecting the right storage solution for each workload and leveraging Outposts’ local infrastructure, you can reduce latency, minimize network dependencies, and maintain consistency across environments. As Outposts capabilities continue to evolve, they offer a strong foundation for modern, resilient, and cost-efficient hybrid cloud architectures.

Reach out to your AWS account team, or fill out this form to learn more about running containarized applications on Outposts.

.NET 10 runtime now available in AWS Lambda

Henrique Graca — Thu, 08 Jan 2026 21:01:05 +0000

AWS Lambda now supports .NET 10 as both a managed runtime and base container image. .NET is a popular language for building serverless applications. Developers can now use the new features and enhancements in .NET when creating serverless applications on Lambda. This includes support for file-based apps to streamline your projects by implementing functions using just a single file.

.NET 10 delivers runtime and compiler optimizations including enhancements to the JIT compiler and improvements to Native AOT that reduce executable size and startup time. For details of the .NET 10 features, you can go to the .NET 10 overview.

You can develop Lambda functions in .NET 10 using the AWS Management Console, AWS Command Line Interface (AWS CLI), AWS Toolkit for Visual Studio, AWS Extensions for .NET CLI (Amazon.Lambda.Tools), AWS Serverless Application Model (AWS SAM), AWS Cloud Development Kit (AWS CDK), and other infrastructure as code (IaC) tools.

You can also use .NET 10 with Powertools for AWS Lambda (.NET), a developer toolkit that helps you implement serverless best practices. Use cases include observability, batch processing, AWS Systems Manager Parameter Store integration, idempotency, and more.

This post demonstrates what’s new in the .NET 10 Lambda runtime and how you can use the new .NET 10 runtime in your serverless applications.

File-based C# applications

.NET 10 introduces file-based apps, which are programs contained in a single .cs file, without a .csproj file or a complex folder structure. File-based apps are an ideal way to streamline the development and management of .NET Lambda functions. They are fully supported by the Lambda .NET 10 runtime and associated developer tooling.

Creating C# file-based apps

The fastest way to get started creating a C# file-based Lambda function is to use the Amazon.Lambda.Templates package. Version 8.0.1 of the package adds the lambda.FileBased template as well as updating the rest of the templates in the package to .NET 10.

Install the package by running the following command:

dotnet new install Amazon.Lambda.Templates

Create a new C# file-based Lambda function by running the following command:

dotnet new lambda.FileBased -n MyLambdaFunction

This creates a file in the current directory called MyLambdaFunction.cs with all of the required startup code necessary for a Lambda function. The following is the starting content of the file:

// C# file-based Lambda functions can be deployed to Lambda using the 
// .NET Tool Amazon.Lambda.Tools version 6.0.0 or later. 
// 
// Command to install Amazon.Lambda.Tools 
//   dotnet tool install -g Amazon.Lambda.Tools 
// 
// Command to deploy function 
//    dotnet lambda deploy-function <lambda-function-name> MyLambdaFunction.cs 
// 
// Command to package function 
//    dotnet lambda package MyLambdaFunction.zip MyLambdaFunction.cs 
 
 
#:package Amazon.Lambda.Core@2.8.0 
#:package Amazon.Lambda.RuntimeSupport@1.14.1 
#:package Amazon.Lambda.Serialization.SystemTextJson@2.4.4 
 
// Explicitly setting TargetFramework here is done to avoid 
// having to specify it when packaging the function with Amazon.Lambda.Tools 
#:property TargetFramework=net10.0 
 
// By default File-based C# apps publish as Native AOT. When packaging Lambda function 
// unless the host machine is Amazon Linux a container build will be required. 
// Amazon.Lambda.Tools will automatically initate a container build if docker is installed. 
// Native AOT also requires the code and dependencies be Native AOT compatible. 
// 
// To disable Native AOT uncomment the following line to add the .NET build directive 
// that disables Native AOT. 
//#:property PublishAot=false 
 
using Amazon.Lambda.Core; 
using Amazon.Lambda.RuntimeSupport; 
using Amazon.Lambda.Serialization.SystemTextJson; 
using System.Text.Json.Serialization; 
 
// The function handler that will be called for each Lambda event 
var handler = (string input, ILambdaContext context) => 
{ 
    return input.ToUpper(); 
}; 
 
// Build the Lambda runtime client passing in the handler to call for each 
// event and the JSON serializer to use for translating Lambda JSON documents 
// to .NET types. 
await LambdaBootstrapBuilder.Create(handler, new SourceGeneratorLambdaJsonSerializer<LambdaSerializerContext>()) 
        .Build() 
        .RunAsync(); 
 
// Since Native AOT is used by default with C# file-based Lambda functions the source generator 
// based Lambda serializer is used. Ensure the input type and return type used by the function 
// handler are registered on the JsonSerializerContext using the JsonSerializable attribute. 
[JsonSerializable(typeof(string))] 
public partial class LambdaSerializerContext : JsonSerializerContext 
{ 
}

File-based functions use executable assembly handlers, in which the compiler generates the Main() method containing your function code. Therefore, your code must include the Amazon.Lambda.RuntimeSupport NuGet package and implement the LambdaBootstrapBuilder.Create method to bootstrap the runtime client.

File-based applications also use .NET Native AOT by default. You can disable Native AOT by adding #:property PublishAot=false to the top of the file. For more information on using Native AOT in Lambda, go to Compile .NET Lambda function code to a native runtime format in the Lambda documentation.

Deploying C# file-based apps

To deploy your function using the dotnet CLI with the Amazon.Lambda.Tools extension, pass the .cs filename as an added argument. Native AOT is enabled by default, thus the build must match the target architecture. If you’re building on the same architecture as the target Lambda function and on Amazon Linux 2023, then the build runs natively. Otherwise, Amazon.Lambda.Tools uses a Docker container to build the function.

For example, to deploy for x86_64 (default architecture):


dotnet lambda deploy-function ToUpper ToUpper.cs --function-runtime 
dotnet10 --function-role <role-arn>

Alternatively, to deploy for arm64:


dotnet lambda deploy-function ToUpper ToUpper.cs --function-runtime 
dotnet10 --function-role <role-arn> --function-architecture arm64

Debugging C# file-based apps

Visual Studio Code with the C# Dev Kit support debugging C# file-based applications.

1. Install the test tool.
dotnet tool install -g Amazon.Lambda.TestTool

2. Start the emulator.
dotnet lambda-test-tool start --lambda-emulator-port 5050

3. Configure .vscode/launch.json to attach to the process.

{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "LambdaDebugFile",
      "type": "coreclr",
      "request": "launch",
      "program": "${fileDirname}/artifacts/Debug/${fileBasenameNoExtension}.dll",
      "cwd": "${workspaceFolder}",
      "console": "internalConsole",
      "stopAtEntry": false,
      "env": {
        "AWS_LAMBDA_RUNTIME_API": "localhost:5050/${fileBasenameNoExtension}"
      },
      "preLaunchTask": "build-active-file"
    }
  ]
}

4. Configure .vscode/tasks.json to build the active file.

{
  "version": "2.0.0",
  "tasks": [
    {
      "label": "build-active-file",
      "command": "dotnet",
      "type": "process",
      "args": [
        "build",
        "${file}",
        "--output",
        "./artifacts/Debug"
      ],
      "problemMatcher": "$msCompile"
    }
  ]
}

The configuration uses ${file}, which allows the build task to target whichever C# file is currently active in your editor. This enables seamless debugging across multiple single-file functions.

Lambda Managed Instances

The Lambda runtime for .NET 10 includes support for Lambda Managed Instances, so that you can run .NET 10 Lambda functions on Amazon Elastic Compute Cloud (Amazon EC2) instances while maintaining serverless operational clarity. Therefore, you can use current-generation EC2 instances, including Graviton4, network-optimized instances, and other specialized compute options, without managing instance lifecycles, operating system patching, or scaling policies. Lambda Managed Instances provides access to Amazon EC2 commitment-based pricing models, such as Compute Savings Plans and Reserved Instances, which can provide up to a 72% discount over Amazon EC2 On-Demand pricing. This offers significant cost savings for steady-state workloads while maintaining the familiar Lambda programming model. For more information, go to Lambda Managed Instances.

With Lambda Managed Instances, each function execution environment can process multiple function invokes at the same time. In .NET, Lambda uses .NET Tasks for asynchronous processing of multiple concurrent requests. You should apply the same concurrency safety practices when using Lambda Managed Instances that you would in any other multi-concurrent environment. For example, any mutable state—including shared collections, database connections, and static objects—must be thread safe. For more information, go to .NET runtime for Lambda Managed Instances.

Performance considerations

At launch, new Lambda runtimes receive less usage than existing established runtimes. This can result in longer cold start times due to reduced cache residency within internal Lambda sub-systems. Cold start times typically improve in the weeks following launch as usage increases. Therefore, AWS recommends not drawing conclusions from side-by-side performance comparisons with other Lambda runtimes until the performance has stabilized.

Performance is highly dependent on workload, thus you should conduct your own testing instead of relying on generic test benchmarks. There are a range of features available to reduce the impact of cold starts for Lambda functions that use .NET 10, including SnapStart, provisioned concurrency, Native AOT, and Lambda Managed Instances.

In addition, .NET 10 Lambda workloads might experience slightly lower performance until some added runtime enhancements are released. Go to dotnet/runtime#120288 for details.

Migrating from .NET 8 to .NET 10

To use .NET 10 and the new file-based features, you must update your tools.

1. Install or update the .NET 10 SDK.

2. If you are using AWS SAM, then install or update to the latest version.

3. If you are using Visual Studio, then install or update the AWS Toolkit for Visual Studio to version 1.83.0.0 or later.

4. If you use the .NET Lambda Global Tools extension (Amazon.Lambda.Tools) for the .NET CLI, then update to version 6.0.0 or later to support file-based C#.

To upgrade a function to .NET 10, check your code and dependencies for compatibility with .NET 10, run tests, and update as necessary. Generative AI can help: consider using AWS Transform custom or coding assistants such as Kiro to help with upgrades.

Upgrading using the dotnet CLI

For projects using the dotnet CLI with the Amazon.Lambda.tools extension:

1. Open the .csproj project file.

2. Update the TargetFramework to net10.0.

3. Update NuGet packages Amazon.Lambda.* to the latest versions.

4. If using aws-lambda-tools-defaults.json, then set function-runtime to dotnet10.

5. Run dotnet lambda deploy-function to deploy.

Upgrading using the AWS Toolkit for Visual Studio

To upgrade a function to .NET 10:

1. Open the .csproj project file and update the TargetFramework to net10.0.

2. Update NuGet packages to the latest versions.

3. Right-click the project in Solution Explorer and choose Publish to AWS Lambda.

Upgrading container image functions

Along with the preceding changes, the .NET 8 and .NET 10 runtimes are built on the provided.al2023 runtime, which is based on the Amazon Linux 2023 minimal container image. The Amazon Linux 2023 minimal image uses microdnf as a package manager, symlinked as dnf. This replaces the yum package manager used in .NET 6 and earlier Amazon Linux 2-based images. If you deploy your Lambda functions as container images, then you must update your Dockerfiles to use dnf instead of yum when upgrading to the .NET 10 base image from .NET 6 or earlier base images.

Learn more about the provided.al2023 runtime in the post Introducing the Amazon Linux 2023 runtime for AWS Lambda.

Using the .NET 10 runtime in Lambda

The following sections demonstrate how to use the .NET 10 runtime in Lambda.

The console

On the Create Function page of the Lambda console, choose .NET 10 in the Runtime dropdown menu, as shown in the following figure.

Figure 1: Creating a .NET 10 function in the Lambda console

To update an existing Lambda function to .NET 10, navigate to the function in the Lambda console. Choose Edit in the Runtime settings panel, then choose .NET 10 from the Runtime dropdown menu, as shown in the following figure.

Figure 2: Editing runtime settings to choose .NET 10

Lambda container image

Change the .NET base image version by modifying the FROM statement in your Dockerfile:

FROM public.ecr.aws/lambda/dotnet:10
# Copy function code
COPY artifacts/publish/ ${LAMBDA_TASK_ROOT}

AWS SAM

In AWS SAM, set the Runtime attribute to dotnet10:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Simple Lambda Function
Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      Description: My .NET Lambda Function
      CodeUri: ./src/MyFunction/
      Handler: MyFunction::MyFunction.Function::FunctionHandler
      Runtime: dotnet10

For file-based functions, set CodeUri to the C# file path relative to the AWS SAM template. AWS CloudFormation commands such as deploy-serverless and package-ci package the file-based Lambda function as a .NET executable. The Handler field must be set to the .NET assembly name, which for file-based C# applications is the filename minus the .cs extension:

ToUpperFunction:
  Type: AWS::Serverless::Function
  Properties:
    Handler: ToUpperFunction
    Runtime: dotnet10
    CodeUri: ./ToUpperFunction.cs
    MemorySize: 512
    Timeout: 30

AWS SAM supports generating new serverless .NET 10 applications using the sam init command. Refer to the AWS SAM documentation for details.

Conclusion

AWS Lambda now supports .NET 10 as a managed language runtime to help developers build more efficient, powerful, and scalable serverless applications. .NET 10 language additions include C# 14 features, runtime optimizations, and improved Native AOT support. This release also introduces file-based C# applications for streamlined single-file Lambda functions, and it includes support for Lambda Managed Instances for specialized compute requirements and at-scale cost efficiency.

You can build and deploy functions using .NET 10 using the AWS Management Console, AWS CLI, AWS SDK, AWS SAM, AWS CDK, or your choice of IaC tool. You can also use the .NET 10 container base image if you prefer to build and deploy your functions using container images.

Try the .NET 10 runtime in Lambda today and experience the benefits of this updated language version.

To find more .NET examples, use the Serverless Patterns Collection. For more serverless learning resources, visit Serverless Land.

Building zero trust generative AI applications in healthcare with AWS Nitro Enclaves

Nathan Pogue — Fri, 12 Dec 2025 19:06:03 +0000

In healthcare, generative AI is transforming how medical professionals analyze data, summarize clinical notes, and generate insights to improve patient outcomes. From automating medical documentation to assisting in diagnostic reasoning, large language models (LLMs) have the potential to augment clinical workflows and accelerate research. However, these innovations also introduce significant privacy, security, and intellectual property challenges.

Healthcare data often contains Protected Health Information (PHI), which is governed by strict regulations and compliance frameworks. At the same time, organizations or researchers who have invested substantial time and compute resources into training medical LLMs must protect their proprietary model architectures, weights, and fine-tuned datasets. Traditional deployment models necessitate mutual trust between the model publisher and the healthcare data provider — trust that sensitive data won’t be leaked, and that the model itself won’t be copied, tampered with, or exfiltrated. The absence of a secure and verifiable trust model between model publishers and consumers remains one of the main barriers to scaling generative AI in regulated medical environments.

To address this concern, both parties need a secure environment to publish and consume models without exposing data or intellectual property. Amazon Web Services (AWS) Nitro Enclaves provide isolated, attested, and cryptographically verified compute environments that help protect sensitive workloads. Model owners can encrypt their LLMs with AWS Key Management Service (AWS KMS) and allow only verified Nitro Enclaves to decrypt and run them, making sure that the model can’t be accessed outside the Nitro Enclave. Healthcare organizations and consumers can use this to process sensitive data within their own AWS environment entirely within the Nitro Enclave, helping keep PHI private and contained. Hardware-based attestation provides proof that the Nitro Enclave is running trusted code, so that both sides can exchange information with confidence.

In this post, we demonstrate how to deploy a publicly available foundational model (FM) using Nitro Enclaves for isolated, more secure compute, AWS KMS for model encryption, Amazon Simple Storage Service (Amazon S3) for storing model artifacts and images, and Amazon Simple Queue Service (Amazon SQS) for securely delivering queries, enabling private, privacy-preserving inferences while helping protect both model intellectual property and sensitive patient data.

Solution overview

This solution outlines how to build a more secure end-to-end pipeline that enables zero trust medical LLM publication and inference with Nitro Enclaves. This post demonstrates a guide for setting up an Amazon Elastic Compute Cloud (Amazon EC2) instance with Nitro Enclaves enabled, downloading and encrypting a publicly available FM to an S3 bucket with an AWS KMS key, sending medical text and image-based queries to an SQS queue for processing, and storing results in an Amazon DynamoDB table.

This project is intended solely for educational and demonstration purposes and isn’t suitable for production or clinical use. Its outputs aren’t validated for clinical accuracy and must not be used for patient care or medical decision-making. Before any real-world deployment, make sure that you implement comprehensive security, privacy, and compliance safeguards. These include health data protection controls, secrets management, and regulatory validation. Furthermore, you must consult the appropriate clinical, legal, and security experts.

For demonstration purposes, this solution is deployed in a single AWS account. Ideally, in production, it would be deployed across separate AWS accounts: one for the model owner and one for the model consumer. The model owner can use cross-account AWS Identity and Access Management (IAM) permissions and encrypted model sharing through AWS KMS to securely provide access to their model without exposing the underlying weights or logic. At the same time, the consumer can run sensitive inferences within their own environment, maintaining strict data privacy and zero trust principles. In a real-world implementation, the model provider should also establish a robust entitlement and licensing framework to manage customer access, enabling fine-grained control over who can invoke the model, track usage, and support license revocation to immediately remove permissions from specific customers when necessary.

The following diagram shows the solution architecture:

The steps of the solution include:

Amazon EC2 setup: An EC2 instance is launched with Nitro Enclaves and Trusted Platform Module (TPM) enabled. For this project, a c7i.12xlarge instance with a 150 GB Amazon EBS volume is used to provide the necessary compute resources for running LLMs.
Public FM download: A publicly available FM is retrieved from Hugging Face and stored in an S3 bucket within the model consumer’s AWS account.
Model encryption: The model is encrypted using AWS KMS envelope encryption. Only a Nitro Enclave presenting a valid attestation document can request the decryption key from AWS KMS, which helps prevent unauthorized access to the model weights outside the Nitro Enclave.
Nitro Enclave setup: A Docker image containing the llama.cpp inference runtime is built and deployed inside the Nitro Enclave.
Model decryption and setup: When the Nitro Enclave launched, it requests decryption of the model artifacts using its attestation credentials. Then, the model can be securely decrypted inside the memory of the Nitro Enclave and loaded by the llama.cpp server. This means that the decrypted model weights aren’t visible outside of the Nitro Enclave boundary.
Medical query: Users can submit either text or image-based queries to the model. Queries are sent through vsock, a secure communication channel from the client application to the model server inside the Nitro Enclave. Image queries necessitate that users upload images to an S3 bucket. The upload event triggers an SQS queue, which signals the Amazon EC2 parent to fetch and send the image to the Nitro Enclave image for the medical LLM to process with its multimodal capabilities.
Message history: Each interaction, including the user’s prompt and the model’s response, is logged to a DynamoDB table. This provides a persistent conversation history that enables traceability and auditing while keeping PHI securely stored within the consumer’s account. If necessary, the DynamoDB table can be encrypted and sealed for another layer of security and privacy.

About Google MedGemma 4B

Google MedGemma is a family of medically-optimized LLMs built on Gemma 3, with 4B and 27B parameter variants supporting both text and multimodal versions for medical image inputs. The 4B model offers efficiency and strong performance for multimodal tasks such as report generation and medical Q&A, while the 27B models excel at more demanding scenarios, such as electronic health record interpretation and complex longitudinal data analysis.

MedGemma models are well-suited for automated radiology report generation, clinical triage and documentation, patient education, medical image pre-interpretation, and medical education systems. The 4B model is ideal for portable or resource-constrained deployments, whereas the 27B multimodal delivers maximal performance.

In this project, MedGemma 4B serves as a reference medical LLM, showing how domain-adapted fine-tuning can enhance a model’s ability to interpret, reason about, and respond to complex medical queries. It also provides a foundation for exploring the safe and effective use of LLMs in healthcare applications, while being securely deployed within a Nitro Enclave. However, you can choose to deploy your own medical FM if needed. This is a deeper overview on the 4B model.

Prerequisites

To implement the proposed solution, make sure that you have the following:

The AWS Command Line Interface (AWS CLI) installed on your machine to create the EC2 instance.
AWS permissions with access to EC2 c7i.12xlarge instances and Nitro Enclaves.
Knowledge of Amazon S3, AWS KMS, Amazon SQS, AWS Lambda, and DynamoDB.
Basic knowledge of Nitro Enclaves and healthcare data security.
The GitHub repository cloned to your local machine.

Environment setup

The following sections outline how to set up your environment for this solution.

Create S3 buckets

In this solution, you create two S3 buckets: one for the model artifacts and one for the image inputs.

To create the S3 buckets

Sign in to the Amazon S3 console, choose Create bucket, and follow the prompts to create a new S3 bucket.
For the model artifact bucket, give it a unique name (for example AWSACCOUNTNUMBER-medgemma-model) in the same Region you use for the other project resources.
Repeat the same process for the image bucket (for example AWSACCOUNTNUMBER-medgemma-image-inputs).
Update the S3_BUCKET_NAME variable with your model bucket name in envelope_encrypt_model.sh and run.sh.

Create an SQS queue

When images are uploaded to the S3 image bucket, they are sent to an SQS queue for processing in sequential order by the model running in the Nitro Enclave.

To create an SQS queue

Sign in to the Amazon SQS console, choose Create queue, and follow the prompts to create a new SQS queue.
Choose Standard Queue, provide a name, leave the rest as default, and choose Create queue.
Replace the SQS_QUEUE_URL variable in image_processor.py and lambda_function.py (in the client and assets folder, respectively) with your URL.

Create a Lambda function

For image-based queries, MedGemma 4B expects images encoded in base64 format to be passed in the prompt. To convert the images to this format, a Lambda function is invoked using an Amazon S3 trigger when an image is uploaded to the bucket.

To create a Lambda function

Sign in to the Lambda console, choose Create function, and follow the prompts to create a new Lambda function from scratch.
Choose a name, choose a Python runtime (for example Python 3.13), and paste in the Lambda function code from the assets folder.

Next, update the Lambda function’s IAM role in Permissions under the Configuration tab with access to your S3 image bucket and the SQS queue that you created with inline policy permissions. Attach the following policies:

Amazon S3 policy:

{
 "Version": "2012-10-17",
 "Statement": [
     {
         "Sid": "Statement1",
         "Effect": "Allow",
         "Action": [
             "s3:*"
         ],
         "Resource": [
             "arn:aws:s3:::<IMAGE_BUCKET_NAME>",
             "arn:aws:s3:::<IMAGE_BUCKET_NAME>/*"
         ]
     }
 ]
}

Amazon SQS policy:

{
 "Version": "2012-10-17",
 "Statement": [
     {
         "Sid": "VisualEditor0",
         "Effect": "Allow",
         "Action": "sqs:ListQueues",
         "Resource": "*"
     },
     {
         "Sid": "VisualEditor1",
         "Effect": "Allow",
         "Action": "sqs:*",
        "Resource": "arn:aws:sqs:<REGION>:<ACCOUNT_NUMBER>:<QUEUE_NAME>"
     }
 ]
}

Finally, within the Lambda Designer, add a trigger, choose Amazon S3, and choose your image bucket. You should see the following example when the trigger is enabled.

Create a DynamoDB table

When the queries have been processed by the model for inference, the prompts and responses are logged to a DynamoDB table for auditing and message history purposes.

To create a DynamoDB table

Sign in to the DynamoDB console, choose Create table, and follow the prompts to create a new DynamoDB table.
Give it a partition key named ID as a String type.
Replace the TABLE_NAME variable with the table name and REGION variable with your AWS Region in direct_query.py and image_processor.py files.

Create an AWS KMS key

An AWS KMS key is used to envelope-encrypt the model artifacts before they are uploaded to the S3 model bucket. During encryption, the AWS KMS key policy is configured with conditions that restrict decryption to only those Nitro Enclaves presenting a valid attestation document. This attestation includes platform configuration registers (PCR) hashes that represent the measured state of the Nitro Enclave, which covers the signed Nitro Enclave image, runtime, and configuration. When the Nitro Enclave is launched, it generates an attestation document signed by the Amazon EC2 Nitro hypervisor, proving that its PCR values match the expected trusted measurements defined in the AWS KMS key policy. The key is released only if these PCR hashes align and the attestation is verified by AWS KMS, allowing the Nitro Enclave to decrypt and load the model securely in memory.

To create an AWS KMS key

Sign in to the AWS KMS console, choose Create key, and follow the prompts to create a new AWS KMS key.
Choose Symmetric as the key type and Encrypt and decrypt for the key usage. Make the alias AppKmsKey. Leave the default settings and choose Finish.
Replace the REGION variable in vsock-proxy.yaml in the client folder with your AWS Region.

Create an EC2 instance

Now that the necessary resources are set up, you can proceed to launch the EC2 instance and create the Nitro Enclave image. For this solution, a c7i.12xlarge instance with a 150 GB EBS volume is provisioned.

To launch an EC2 instance with Nitro Enclaves enabled

Within the GitHub repository on your local machine, run ./create_ec2.sh to create the EC2 instance.

cd scripts 
chmod +x create_ec2.sh 
./create_ec2.sh

The script launches an EC2 instance called MedGemmaNitroEnclaveDemo. When the instance is running, you must create an IAM policy and add it to the Amazon EC2 IAM role with necessary permissions to the resources created previously.
Sign in to the IAM console and navigate to Policies, choose Create policy, choose JSON, and paste the following policy, making sure that you update the bucket, queue URL, AWS Region, account number, and table variables:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "s3modelbucket",
      "Effect": "Allow",
      "Action": [
        "s3:*"
      ],
      "Resource": [
        "arn:aws:s3:::<MODEL_BUCKET_NAME>",
        "arn:aws:s3:::<MODEL_BUCKET_NAME>/*"
      ]
    },
    {
      "Sid": "dynamodb",
      "Effect": "Allow",
      "Action": [
        "dynamodb:*"
      ],
      "Resource": [
        "arn:aws:dynamodb:<REGION>:<ACCOUNT_NUMBER>:table/<TABLE_NAME>"
      ]
    },
    {
      "Sid": "sqslist",
      "Effect": "Allow",
      "Action": "sqs:ListQueues",
      "Resource": "*"
    },
    {
      "Sid": "sqsqueue",
      "Effect": "Allow",
      "Action": "sqs:*",
      "Resource": "arn:aws:sqs:<REGION>:<ACCOUNT_NUMBER>:<QUEUE_NAME>"
    }
  ]
}

Give it a name (for example enclave-permissions) and choose Create policy.
Navigate to Roles, choose Create role, choose EC2 as the AWS service for the Trusted entity type, then choose your policy that you created under Add permissions.
Update your EC2 instance to use the role by going to the Security setting under the Actions dropdown, then modifying its IAM role.
You can upload the modified repository to your EC2 instance using SCP. Alternatively, you can transfer the repository through rsync

rsync -avz -e "ssh -i /path/to/directory/sample-for-secure-medical-llm-inference-with-nitro-enclaves/nitro-enclave-key.pem" --exclude='*.pem' /path/to/directory/sample-for-secure-medical-llm-inference-with-nitro-enclaves/ ec2-user@<PUBLIC_IP>:~ 
cd sample-for-secure-medical-llm-inference-with-nitro-enclaves

Make the scripts executable (in client, server and scripts).

chmod +x *.sh

Nitro Enclave setup

With Amazon EC2 loaded with the necessary scripts, you can begin building the Nitro Enclave image. During this process, the Docker container is converted into an Enclave Image File (EIF), which generates cryptographic measurements (PCR hashes) that uniquely identify the code and configuration of the enclave. These measurements are embedded into the AWS KMS key policy, creating a hardware-attested trust boundary that makes sure only this specific, unmodified Nitro Enclave can decrypt and access the model weights.

Run the complete setup script, which sets up the client on the EC2 parent instance and the server running within the Nitro Enclave. You can observe the different scripts in the client, server, and scripts folders.

sudo ./run_complete_setup.sh

The various scripts run to download the MedGemma 4B model, encrypt the model with the AWS KMS key, build a Docker image to run a llama.cpp server, start a Nitro Enclave, and decrypt and run the model. This process takes approximately 10 minutes.
When the Nitro Enclave is running, it runs in debug mode so that you can observe the various startup logs outputted. Wait until llama.cpp server logs are outputted that indicate the server is ready and listening.

Inference examples

When the model is decrypted and running on the llama.cpp server within the Nitro Enclave, you can begin to invoke the model with either image or text-based queries. Open a new terminal session in your EC2 instance. You can navigate to the client folder to run the scripts for queries.

Image-based queries

For inference on medical images, upload an image to your Amazon S3 image bucket. When it is uploaded, run python3 image_processor.py to pass the image from the SQS queue to the Nitro Enclave for processing. The following are examples of image inputs and model outputs.

Brain CT scan:

Case courtesy of Dr Henry Knipe, Radiopaedia.org, rID: 46289

Model response:Chest X-ray:

Model response:Text-based queries

For inference on text-queries, run python3 direct_query.py "<YOUR_MEDICAL_QUERY>" to invoke the model. The following are examples of text-based inputs and model outputs.

Basic usage:

python3 direct_query.py "What are the symptoms of pneumonia?"

Model response:Lab result interpretation:

python3 direct_query.py "Patient has elevated troponin levels (15.2 ng/mL), elevated CK-MB, and ST elevation in leads II, III, aVF. What does this suggest?"

Model response:

Cleaning up

To avoid incurring future charges, delete the resources used in this solution:

Conclusion

You can combine the isolation and attestation capabilities of AWS Nitro Enclaves, the encryption controls of AWS KMS, and the scalability of services such as Amazon S3, Amazon SQS, and Amazon DynamoDB to build a more secure, zero trust pipeline for deploying generative AI models in healthcare. Using Google MedGemma 4B as your reference medical LLM, you can enable privacy-preserving inference where both PHI and model intellectual property remain protected. For more information, consult the following resources:

Orchestrating large-scale document processing with AWS Step Functions and Amazon Bedrock batch inference

Brian Zambrano — Wed, 26 Nov 2025 21:41:51 +0000

Organizations often have large volumes of documents containing valuable information that remains locked away and unsearchable. This solution addresses the need for a scalable, automated text extraction and knowledge base pipeline that transforms static document collections into intelligent, searchable repositories for generative AI applications.

Organizations can automate the extraction of both content and structured metadata to build comprehensive knowledge bases that power retrieval-augmented generation (RAG) solutions while significantly reducing manual processing costs and time-to-value. The architecture not only demonstrates the processing of 500 research papers automatically, but also scales to handle enterprise document volumes cost-effectively through the Amazon Bedrock batch inference pricing model.

Overview

Amazon Bedrock batch inference is a feature of Amazon Bedrock that offers a 50% discount on inference requests. Although Amazon Bedrock schedules and runs the batch job (needing a minimum of 100 inference requests) as capacity becomes available, the inference won’t be real-time. For use cases where you can accommodate minutes to hours of latency, Amazon Bedrock batch inference is a good option.

This post demonstrates how to build an automated, serverless pipeline using AWS Step Functions, Amazon Textract, Amazon Bedrock batch inference, and Amazon Bedrock Knowledge Bases to extract text, create metadata, and load it into a knowledge base at scale. The example solution processes 500 research papers in PDF format from Amazon Science, extracts text using Amazon Textract, generated structured metadata with Amazon Bedrock batch inference and the Amazon Nova Pro model, and loads the final output, including Amazon Bedrock Knowledge Base filter, into an Amazon Bedrock Knowledge Base.

Architecture

This solution uses Step Functions with parallel Amazon Textract job processing through child workflows run by Distributed Map. You can use the concurrency controls offered by Distributed Map to process documents as quickly as possible within your Amazon Textract quotas. Increasing processing speed necessitates adjusting your Amazon Textract quota and updating the Distributed Map configuration. Amazon Bedrock batch inference handles concurrency, scaling, and throttling. This means that you can create the job without managing these complexities.

In this example implementation, the solution processes research papers to extract metadata such as:

Code availability and repository locations
Dataset availability and access methods
Research methodology types
Reproducibility indicators
Other relevant research attributes

The high-level parts of this solution include:

Extracting text from PDF documents with Amazon Textract in parallel, through Step Functions Distributed Map.
Analyzing extracted text using Amazon Bedrock batch inference to extract structured metadata.
Loading extract text and metadata into a searchable knowledge base using Amazon Bedrock Knowledge Bases with Amazon OpenSearch Serverless.

Figure 1. Complete architecture diagram

Prerequisites

The following prerequisites are necessary to complete this solution:

Access to an AWS account through the AWS Management Console and the AWS Command Line Interface (AWS CLI). The AWS Identity and Access Management (IAM) user that you use must have permissions to make the necessary AWS service calls and manage AWS resources mentioned in this post. While providing permissions to the IAM user, follow the principle of least-privilege.
AWS CLI installed and configured. If you are using long-term credentials such as access keys, then follow manage access keys for IAM users and secure access keys for best practices.
Git Installed.
Python 3.13+ installed.
Node and npm installed.
AWS Cloud Development Kit (AWS CDK) installed.

Running the solution

The complete solution uses AWS CDK to implement two AWS CloudFormation stacks:

BedrockKnowledgeBaseStack: Creates the knowledge base infrastructure
SFNBatchInferenceStack: Implements the main processing workflow

First, clone the GitHub repository into your local development environment and install the requirements:

git clone https://github.com/aws-samples/sample-step-functions-batch-inference.git .

cd sample-step-functions-batch-inference

npm install

Next, deploy the solution using AWS CDK:

cdk deploy --all

After deploying the cdk stacks, upload your data sources (PDF files) into the AWS CDK-created Amazon S3 input bucket. In this example, I uploaded 500 Amazon Science papers. The input bucket name is included in the AWS CDK outputs:

Outputs:

SFNBatchInference.BatchInputBucketName = sfnbatchinference-batchinputbucket11aaa222-nrjki8tewwww

Parallel text extraction

The process begins when you upload a manifest.json file to the input bucket. The manifest file lists the files for processing, which already exist in the input bucket. The filenames listed in manifest.json define what constitutes a single processing job run. To create another run, you would create a different manifest.json and upload it to the same S3 bucket.

[
  {
    "filename": "flexecontrol-flexible-and-efficient-multimodal-control-for-text-to-image-generation.pdf"
  },
  {
    "filename": "adaptive-global-local-context-fusion-for-multi-turn-spoken-language-understanding.pdf"
  }
]

The AWS CDK definition for the input bucket includes Amazon EventBridge notifications and creates a rule that triggers the Step Functions workflow whenever a manifest.json file is uploaded.

private createS3Buckets() {
    const batchBucket = new s3.Bucket(this, "BatchInputBucket", {
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      autoDeleteObjects: true,
    })
    batchBucket.enableEventBridgeNotification()

    new cdk.CfnOutput(this, "BatchInputBucketName", {
      value: batchBucket.bucketName,
      description: "Name of input bucket to send PDF documents that Textract will read.",
    })

    const manifestFileCreatedRule = new eventBridge.Rule(this, "ManifestFileCreatedRule", {
      eventPattern: {
        source: ["aws.s3"],
        detailType: ["Object Created"],
        detail: {
          bucket: {
            name: [batchBucket.bucketName],
          },
          object: {
            key: ["manifest.json"],
          },
        },
      },
    })

    return { batchBucket, manifestFileCreatedRule }
  }

The first step in the Step Functions workflow is a Distributed Map run that performs the following actions for each PDF in the manifest file:

Starts an Amazon Textract job, providing an Amazon Simple Notification Service (Amazon SNS) topic for completion notification.
Writes the Step Functions task token to Amazon DynamoDB, pausing the individual child workflow.
Processes the Amazon SNS message when the Amazon Textract job completes, triggering an AWS Lambda function.
Uses a Lambda function to retrieve the task token from DynamoDB using the Amazon Textract JobId.
Fetches the raw results from Amazon Textract, organizes the text for readability, and writes results to an S3 bucket

A key component of this architecture is the callback pattern that Amazon Textract supports using the NotificationChannel option, as shown in the preceding figure. The AWS CDK definition the Step Functions state that starts the Amazon Textract job is shown in the following.

const startTextractStep = new tasks.CallAwsService(this, "StartTextractJob", {
  service: "textract",
  action: "startDocumentAnalysis",
  resultPath: "$.textractOutput",
  parameters: {
    DocumentLocation: {
      S3Object: {
        Bucket: sourceBucket.bucketName,
        Name: sfn.JsonPath.stringAt("$.filename"),
      },
    },
    FeatureTypes: ["LAYOUT"],
    NotificationChannel: {
      RoleArn: textractRoleArn,
      SnsTopicArn: snsTopicArn,
    },
  },
  iamResources: ["*"],
})

The Lambda function that handles task tokens extracts the Amazon Textract JobId from the Amazon SNS message, fetches the TaskToken from DynamoDB, and resumes the Step Functions Workflow by sending the TaskToken:

from aws_lambda_powertools.utilities.data_classes import SNSEvent, event_source

@event_source(data_class=SNSEvent)
def handle_textract_task_complete(event, context):
    # Multiple records can be delivered in a single event
    for record in event.records:
        sns_message = json.loads(record.sns.message)
        textract_job_id = sns_message["JobId"]

        # Get both task token and original file from DynamoDB
        ddb_item = _get_item_from_ddb(textract_job_id)

        # Send both the job ID and original file name in the response
        _send_task_success(
            ddb_item["TaskToken"],
            {
                "TextractJobId": textract_job_id,
                "OriginalFile": ddb_item["OriginalFile"],
            },
        )
        # Delete the task token from DynamoDB after use
        _delete_item_from_ddb(textract_job_id)

def _send_task_success(task_token: str, output: None | dict = None) -> None:
    """Sends task success to Step Functions with the provided output"""
    sfn = boto3.client("stepfunctions")
    sfn.send_task_success(taskToken=task_token, output=json.dumps(output or {}))

The Distributed Map runs up to 10 child workflows concurrently, controlled by the maxConcurrency setting. Although Step Functions supports running up to 10,000 child workflow executions, the practical concurrency for this solution is constrained by Amazon Textract quotas. The startDocumentAnalysis API has a default quota of 10 requests per second (RPS), which means you must consider this limit when scaling your document processing workloads and potentially request quota increases for higher throughput requirements.

const distributedMap = new sfn.DistributedMap(this, "DistributedMap", {
  mapExecutionType: sfn.StateMachineType.STANDARD,
  maxConcurrency: 10,
  itemReader: new sfn.S3JsonItemReader({
    bucket: sourceBucket,
    key: "manifest.json",
  }),
  resultPath: "$.files",
}

Running Amazon Bedrock batch inference

When all of the Amazon Textract jobs finish, the Distributed Map state creates an Amazon Bedrock batch inference input file, launches the Amazon Bedrock inference job, and waits for it to complete.

A Lambda function collects text results from Amazon S3 and creates an Amazon Bedrock batch inference input file with custom prompts.
The workflow starts the Amazon Bedrock batch inference job by calling createModelInvocationJob and sending the batch inference input file as input.
The workflow pauses and stores the task token in DynamoDB.
An EventBridge rule matches completed Amazon Bedrock batch inference events, and upon job completion and triggers a Lambda function. The Lambda function retrieves the task token and resumes the workflow, as shown in the following figure.

A batch inference input is a single jsonl file with multiple entries such as the following example. The prompt in each inference request instructs the large language model (LLM) to analyze the paper and extract metadata. Read the full prompt template in the GitHub repository.

{
  "recordId": "c1b8a3b2086141f963",
  "modelInput": {
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "text": "Analyze the following research paper transcript and extract metadata about code and dataset availability. Extract the following metadata from this research paper transcript:\n\n1. **has_code**: Does the paper mention or link to source code? (true/false) ...... Return only valid JSON matching the schema above. Do not include any text outside of the JSON structure."
          }
        ]
      }
    ],
    "inferenceConfig": { "maxTokens": 4096 }
  }
}

Populating the Amazon Bedrock Knowledge Base

After the batch inference completes, the workflow does the following:

Extracts inference results and creates metadata files based on the Amazon Bedrock inference results (example metadata shown in the following figure).
Starts an Amazon Bedrock Knowledge Base ingestion job.
Monitors the ingestion job status using Step Functions Wait and Choice states.
Sends a completion notification through Amazon SNS.

The following shows the example metadata format:

{
  "metadataAttributes": {
    "has_code": true,
    "has_dataset": false,
    "code_availability": "publicly_available",
    "dataset_availability": "not_available",
    "research_type": "methodology",
    "is_reproducible": true,
    "code_repository_url": "https://github.com/amazon-science/PIXELS"
  }
}

Testing the knowledge base

After the workflow completes successfully, you can test the knowledge base to verify that the documents and metadata have been properly ingested and are searchable. There are two practical methods for testing an Amazon Bedrock Knowledge Base:

Using the Console
Using the AWS SDK to run a query

Testing through the Console

The Console provides an intuitive interface for testing your knowledge base queries with metadata filters:

Navigate to the Amazon Bedrock console.
In the left navigation pane, choose Knowledge Bases under the Build section.
Choose the knowledge base created by the AWS CDK deployment (the name will be output by the AWS CDK stack).
Choose the Test button in the upper right corner.
In the test interface, choose your preferred foundation model (FM) (such as Amazon Nova Pro).
Expand the Configurations column, then navigate to the Filters section.
Configure filters based on the extracted metadata, as shown in the following figure.

Enter a natural language query related to your documents, for example: “Recent research on retrieval augmented generation?”

The console displays the generated response along with source attributions showing which documents were retrieved and used to formulate the answer, filtered by your specified metadata attributes, as shown in the following figure.

Testing via API

For programmatic testing and integration into applications, use the AWS SDK with metadata filtering. The following is a Python example using boto3:

model_arn = "arn:aws:bedrock:us-east-1::foundation-model/amazon.nova-pro-v1:0"

# Query for papers with publicly available code
response = bedrock_agent_runtime.retrieve_and_generate(
    input={'text': "What recent research has been done on RAG?"},
    retrieveAndGenerateConfiguration={
        'type': 'KNOWLEDGE_BASE',
        'knowledgeBaseConfiguration': {
            'knowledgeBaseId': knowledge_base_id,
            'modelArn': model_arn,
            'retrievalConfiguration': {
                'vectorSearchConfiguration': {
                    'numberOfResults': 5,
                    'filter': {"equals": {"key": "has_code", "value": True}},
                }
            },
        },
    },
)

# Display results
print(f"Response: {response['output']['text']}\n")
print("Source Documents:")

for citation in response.get('citations', []):
    for reference in citation.get('retrievedReferences', []):
        metadata = reference.get('metadata', {})
        print(f" Document: {reference['location']['s3Location']['uri']}\n")

The following is the test script output:

Response: Recent research on Retrieval-Augmented Generation (RAG) has focused on enhancing the system's ability to dynamically retrieve and utilize relevant information from a Vector Database (VDB) to improve decision-making and performance. Key innovations include:

1. **Dynamic Retrieval and Utilization**: The system is designed to query the VDB for contextually relevant past experiences, which significantly improves decision quality and accelerates performance by leveraging a growing repository of relevant experiences.

2. **Teacher-Student Instructional Tuning**: A novel mechanism where a Teacher agent refines a Student agent's core policy through direct interaction. The Teacher generates a modified SYSTEM prompt based on the Student's actions, creating a meta-learning loop that enhances the Student's reasoning policy over time.

Conclusion

This solution demonstrates how to combine multiple AWS AI and serverless services to build a scalable document processing pipeline. Organizations can use AWS Step Functions for orchestration, Amazon Textract for document processing, Amazon Bedrock batch inference for intelligent content analysis, and Amazon Bedrock Knowledge Bases for searchable storage. In turn, they can automate the extraction of insights from large document collections while optimizing costs.

Following this solution, you can build a solid foundation for production-scale document processing pipelines that maintain the flexibility to adapt to your specific requirements while making sure of reliability, scalability, and operational excellence. Follow this link to learn more about serverless architectures.

Node.js 24 runtime now available in AWS Lambda

Andrea Amorosi — Tue, 25 Nov 2025 22:19:46 +0000

You can now develop AWS Lambda functions using Node.js 24, either as a managed runtime or using the container base image. Node.js 24 is in active LTS status and ready for production use. It is expected to be supported with security patches and bugfixes until April 2028.

The Lambda runtime for Node.js 24 includes a new implementation of the Runtime Interface Client (RIC), which integrates your functions code with the Lambda service. Written in TypeScript, the new RIC streamlines and simplifies Node.js support in Lambda, removing several legacy features. In particular, callback-based function handlers are no longer supported.

Node.js 24 includes several additions to the language, such as Explicit Resource Management, as well as changes to the runtime implementation and the standard library. With this release, Node.js developers can take advantage of these new features and enhancements when creating serverless applications on Lambda.

You can develop Node.js 24 Lambda functions using the AWS Management Console, AWS Command Line Interface (AWS CLI), AWS SDK for JavaScript, AWS Serverless Application Model (AWS SAM), AWS Cloud Development Kit (AWS CDK), and other infrastructure as code tools. You can use Node.js 24 with Powertools for AWS Lambda (TypeScript), a developer toolkit to implement serverless best practices and increase developer velocity. Powertools includes libraries to support common tasks such as observability, AWS Systems Manager Parameter Store integration, idempotency, batch processing, and more. You can also use Node.js 24 with Lambda@Edge to customize low-latency content delivered through Amazon CloudFront.

This blog post highlights important changes to the Node.js runtime, notable Node.js language updates, and how you can use the new Node.js 24 runtime in your serverless applications.

Node.js 24 runtime changes

The Lambda Runtime for Node.js 24 includes the following changes relative to the Node.js 22 and earlier runtimes.

Removing support for callback-based function handlers

Starting with the Node.js 24 runtime, Lambda no longer supports the callback-based handler signature for asynchronous operations. Callback-based handlers take three parameters, with the third parameter a callback. For example:

export const handler = (event, context, callback) => {
    try {
        // Some processing...
        
        // Success case
        // First parameter (error) is null, second is the result
        callback(null, {
            statusCode: 200,
            body: JSON.stringify({
                message: "Operation completed successfully"
            })
        });
        
    } catch (error) {
        // Error case
        // First parameter contains the error
        callback(error);
    }
};

The modern approach to asynchronous programming in Node.js is to use the async/await pattern. Lambda introduced support for async handlers with the Node.js 8 runtime, launched in 2018. Here’s how the above function looks when using an async handler:

export const handler = async (event, context) => {
    try {
	  // Some processing
        
        return {
            statusCode: 200,
            body: JSON.stringify({
                message: "Operation completed successfully"
            })
        };
        
    } catch (error) {
        throw error;
    }
};

The Node.js 24 runtime still supports synchronous function handlers that do not use callbacks:

export const handler = (event, context) => {
    // Perform some synchronous data processing
    // Return response
    return {
        statusCode: 200,
        body: JSON.stringify(response)
    };
};

And Node.js 24 still supports response streaming, enabling more responsive applications by accelerating the time-to-first-byte:

export const handler = awslambda.streamifyResponse(async (event, responseStream, context) => {
    // Convert event to a readable stream
    const requestStream = Readable.from(Buffer.from(JSON.stringify(event)));
    // Stream the response using pipeline
    await pipeline(requestStream, responseStream);
});

This change to remove support for callback-based function handlers only affects Node.js 24 (and later) runtimes. Existing runtimes for Node.js 22 and earlier continue to support callback-based function handlers. When migrating functions that use callback-based handlers to Node.js 24, you need to modify your code to use one of the supported function handler signatures

As part of this change, context.callbackWaitsForEmptyEventLoop is removed. In addition, the previously deprecated context.succeed, context.fail, and context.done methods have also been removed. This aligns the runtime with modern Node.js patterns for clearer, more consistent error and result handling.

Harmonizing streaming and non-streaming behavior for unresolved promises

The Node.js 24 runtime also resolves a previous inconsistency in how unresolved promises were handled. Previously, Lambda would not wait for unresolved promises once the handler returns except when using response streaming. Starting with Node.js 24, the response streaming behavior is now consistent with non-streaming behavior, and Lambda no longer waits for unresolved promises once your handler returns or the response stream ends. Any background work (for example, pending timers, fetches, or queued callbacks) is not awaited implicitly. If your response depends on additional asynchronous operations, ensure you await them in your handler or integrate them into the streaming pipeline before closing the stream or returning, so the response only completes after all required work has finished.

Experimental Node.js features

Node.js enables certain experimental features by default in the upstream language releases. Such features include support for importing modules using require() in ECMAScript modules (ES modules) and automatically detecting ES vs CommonJS modules. As they are experimental, these features may be unstable or undergo breaking changes in future Node.js updates. To provide a stable experience, Lambda disables these features by default in the corresponding Lambda runtimes.

Lambda allows you to re-enable these features by adding the --experimental-require-module flag or the --experimental-detect-module flag to the NODE_OPTIONS environment variable. Enabling experimental Node.js features may affect performance and stability, and these features can change or be removed in future Node.js releases; such issues are not covered by AWS Support or the Lambda SLA.

ES modules in CloudFormation inline functions

With AWS CloudFormation inline functions, you provide your function code directly in the CloudFormation template. They’re particularly useful when deploying custom resources. With inline functions, the code filename is always index.js, which by default Node.js interprets as a CommonJS module. With the Node.js 24 runtime, you can use ES modules when authoring inline functions by passing the --experimental-detect-module flag via the NODE_OPTIONS environment variable. Previously, you needed a zip or container package to use ES modules. With Node.js 24, you can write inline functions using standard ESM syntax (import/export) and top‑level await), which simplifies small utilities and bootstrap logic without requiring a packaging step.

Node.js 24 language features

Node.js 24 introduces several language updates and features that enhance developer productivity and improve application performance.

Node.js 24 includes Undici 7, a newer version of the HTTP client that powers global ⁠fetch. This version brings performance improvements and broader protocol capabilities. Network‑heavy Lambda functions that call AWS services or external APIs can benefit from better connection management and throughput, especially when reusing clients or using HTTP/2 where supported. Most applications should work without changes, but you should validate behavior for advanced scenarios, such as custom headers or streaming bodies, and continue to define HTTP clients outside of the handler to maximize connection reuse across invocations.

The JavaScript Explicit Resource Management syntax (⁠using and ⁠await using) enables deterministic clean-up of resources when a block completes. For Lambda handlers, this makes it easier to ensure short‑lived objects, such as streams, temporary buffers, or file handles, are disposed of promptly, which reduces the risk of resource leaks across warm invocations. You should continue to define long‑lived clients, for example SDK clients or database pools, outside the handler to benefit from connection reuse, and apply explicit disposal only to resources you want to tear down at the end of each invocation.

Finally, the ⁠AsyncLocalStorage API now uses ⁠AsyncContextFrame by default, improving the performance and reliability of async context propagation. This benefits common serverless patterns such as timers, correlating logs, managing tracing IDs and request‑scoped metadata across async and await boundaries, and streams without manual parameter threading. If you already use ⁠AsyncLocalStorage‑based libraries for logging or observability, you may see lower overhead and more consistent context propagation in Node.js 24.

For a detailed overview of Node.js 24 language features, see the Node.js 24 release blog post and the Node.js 24 changelog.

Performance considerations

At launch, new Lambda runtimes receive less usage than existing established runtimes. This can result in longer cold start times due to reduced cache residency within internal Lambda sub-systems. Cold start times typically improve in the weeks following launch as usage increases. As a result, AWS recommends not drawing conclusions from side-by-side performance comparisons with other Lambda runtimes until the performance has stabilized. Since performance is highly dependent on workload, customers with performance-sensitive workloads should conduct their own testing, instead of relying on generic test benchmarks.

Builders should continue to measure and test function performance and optimize function code and configuration for any impact. To learn more about how to optimize Node.js performance in Lambda, see our blog post Optimizing Node.js dependencies in AWS Lambda.

Migration from earlier Node.js runtimes

We’ve already discussed changes that are new to the Node.js 24 runtime, such as removing support for callback-based function handlers. As a reminder, we’ll recap some previous changes for customers upgrading from older Node.js functions.

AWS SDK for JavaScript

Up until Node.js 16, Lambda’s Node.js runtimes included the AWS SDK for JavaScript version 2. This has since been superseded by the AWS SDK for JavaScript version 3, which was released in December 2024. Starting with Node.js 18, and continuing with Node.js 24, the Lambda Node.js runtimes include version 3. When upgrading from Node.js 16 or earlier runtimes and using the included version 2, you must upgrade your code to use the v3 SDK.

For optimal performance, and to have full control over your code dependencies, we recommend bundling and minifying the AWS SDK in your deployment package, rather than using the SDK included in the runtime. For more information, see Optimizing Node.js dependencies in AWS Lambda.

Amazon Linux 2023

The Node.js 24 runtime is based on the provided.al2023 runtime, which is based on the Amazon Linux 2023 minimal container image. The Amazon Linux 2023 minimal image uses microdnf as a package manager, symlinked as dnf. This replaces the yum package manager used in Node.js 18 and earlier AL2-based images. If you deploy your Lambda function as a container image, you must update your Dockerfile to use dnf instead of yum when upgrading to the Node.js 24 base image from Node.js 18 or earlier.

Learn more about the provided.al2023 runtime in the blog post Introducing the Amazon Linux 2023 runtime for AWS Lambda and the Amazon Linux 2023 launch blog post.

Using the Node.js 24 runtime in AWS Lambda

Finally, we’ll review how to configure your functions to use Node.js 24, using a range of deployment tools.

AWS Management Console

When using the AWS Lambda Console, you can choose Node.js 24.x in the Runtime dropdown when creating a function:

Creating Node.js function in the AWS Management Console

To update an existing Lambda function to Node.js 24, navigate to the function in the Lambda console, click Edit in the Runtime settings panel, then choose Node.js 24.x from the Runtime dropdown:

Editing Node.js function runtime

AWS Lambda container image

Change the Node.js base image version by modifying the FROM statement in your Dockerfile.

FROM public.ecr.aws/lambda/nodejs:24
# Copy function code
COPY lambda_handler.mjs ${LAMBDA_TASK_ROOT}

AWS Serverless Application Model

In AWS SAM, set the Runtime attribute to node24.x to use this version:

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: lambda_function.lambda_handler
      Runtime: nodejs24.x
      CodeUri: my_function/.
      Description: My Node.js Lambda Function

AWS SAM supports generating this template with Node.js 24 for new serverless applications using the sam init command. For more information, refer to the AWS SAM documentation.

AWS Cloud Development Kit (AWS CDK)

In AWS CDK, set the runtime attribute to Runtime.NODEJS_24_X to use this version.

import * as cdk from "aws-cdk-lib";
import * as lambda from "aws-cdk-lib/aws-lambda";
import * as path from "path";
import { Construct } from "constructs";
export class CdkStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);
    // The code that defines your stack goes here
    // The Node.js 24 enabled Lambda Function
    const lambdaFunction = new lambda.Function(this, "node24LambdaFunction", {
      runtime: lambda.Runtime.NODEJS_24_X,
      code: lambda.Code.fromAsset(path.join(__dirname, "/../lambda")),
      handler: "index.handler",
    });
  }
}

Conclusion

AWS Lambda now supports Node.js 24 as a managed runtime and container base image. This release uses a new runtime interface client, removes support for callback-based function handlers, and includes several other changes to streamline and simplify Node.js support in Lambda.

You can build and deploy functions using Node.js 24 using the AWS Management Console, AWS CLI, AWS SDK, AWS SAM, AWS CDK, or your choice of infrastructure as code tool. You can also use the Node.js 24 container base image if you prefer to build and deploy your functions using container images.

To find more Node.js examples, use the Serverless Patterns Collection. For more serverless learning resources, visit Serverless Land

Performance benefits of new Amazon EC2 R8a memory-optimized instances

Tyler Jones — Tue, 25 Nov 2025 19:32:37 +0000

Recently we announced the availability of Amazon Elastic Compute Cloud (Amazon EC2) R8a instances, the latest addition to the AMD memory-optimized instance family. These instances are powered by the 5th Generation AMD EPYC (codename Turin) processors with a maximum frequency of 4.5 GHz. In this post I take these instances for a spin and benchmark MySQL later on, but first I discuss the top things you should know about these instances.

Notable characteristics of R8a instances

Each vCPU on an R8a instance corresponds to a physical CPU core (something we started on 7th generation AMD instances). This means that there is no simultaneous multi-threading (SMT). Each vCPU mapped to a dedicated physical core, which means that you get more predictable and consistent performance because there’s no resource sharing or potential interference between threads, which is particularly crucial for performance-sensitive workloads where consistent latency is essential. When evaluating and adopting R8a instances, make sure that you’re re-evaluating your thresholds for CPU usage. You can likely squeeze more out of each instance’s CPU without impacting any of your workload’s SLA metrics.

R8a instances feature sizes of up to 192 vCPU with 1,536 GiB RAM. The following table shows the detailed specs:

Instance size	vCPU	Memory (GiB)	Instance storage	Network bandwidth (Gbps)	EBS bandwidth (Gbps)
r8a.medium	1	8	EBS Only	Up to 12.5	Up to 10
r8a.large	2	16	EBS Only	Up to 12.5	Up to 10
r8a.xlarge	4	32	EBS Only	Up to 12.5	Up to 10
r8a.2xlarge	8	64	EBS Only	Up to 15	Up to 10
r8a.4xlarge	16	128	EBS Only	Up to 15	Up to 10
r8a.8xlarge	32	256	EBS Only	15	10
r8a.12xlarge	48	384	EBS Only	22.5	15
r8a.16xlarge	64	512	EBS Only	30	20
r8a.24xlarge	96	768	EBS Only	40	30
r8a.48xlarge	192	1536	EBS Only	75	60
r8a.metal-24xl	96	768	EBS Only	40	30
r8a.metal-48xl	192	1536	EBS Only	75	60

Testing MySQL performance using HammerDB

R8a instances are a great choice for MySQL databases, so I thought that would be a great place to showcase some of these instances capabilities. To test MySQL, I used a series of scripts written by my colleagues to track MySQL performance across software versions and different EC2 instances. These scripts are stored in the repro-collection repository, which is an open source, extensible framework for performance testing that addresses real-world workloads rather than micro-benchmarks. It is built to provide a performance measurement reference usable across multiple organizations, and it’s currently centered on MySQL and actively used in discussions with Linux Kernel developers and maintainers. Furthermore, it helps track any performance impacts created by code changes to MySQL. The scripts contained in this repository set up a MySQL database to be tested, and a load generator running the HammerDB benchmark.

For this benchmark I used an r6a.24xlarge instance for the load generator, and an r6a.xlarge, r7a.xlarge, and r8a.xlarge instances for the MySQL database server all deployed in the same AWS Availability Zone (AZ). I chose a single AZ setup to minimize any latency variability from crossing multiple AZs. This is not meant to be a production-like setup, and I highly recommend using multiple AZs for production workloads. Each MySQL instance was tested separately using the same HammerDB load generator. Each test was run three times, and the results were averaged across the three runs. A diagram of the architecture is shown in the following figure:

HammerDB overall results

R8a instances show great results in the HammerDB benchmark for MySQL databases. For HammerDB’s overall score category, R8a instances outscored R7a instances by 55% and outscored R6a instances by 74%.

HammerDB transactions per minute test

R8a instances also showed a notable improvement in this category. When compared to previous generation R7a instances, R8a out performed R7a by 32%. When compared to R6a instances, R8a outperformed by 63%.

HammerDB P99 latency results

R8a instances showed improvement in P99 latency results, showing the efficiency gains driven by the new 5th Generation AMD EPYC CPUs and higher memory bandwidth. R8a shows an 14% latency reduction when compared to R7a, and a 25% latency reduction when compared to R6a.

Conclusion

Built on the AWS Nitro System using sixth generation Nitro Cards, R8a instances are ideal for high performance, memory-intensive workloads, such as SQL and NoSQL databases, as demonstrated by the bench-marking shown in this post, as well as distributed web scale in-memory caches, in-memory databases, real-time big data analytics, and Electronic Design Automation (EDA) applications. R8a instances offer 12 sizes, including 2 bare metal sizes. Amazon EC2 R8a instances are SAP-certified, and providing 38% more SAPS when compared to R7a instances. If you’re still running 6th generation R6a instances, then I highly encourage you to migrate to the 8th generation instances to use their clear price performance benefits. Staying on modern infrastructure is a great way to drive down costs and provide more features for your customers, and there are clear gains to be had based on the testing shown in this post.

Start optimizing your high performance memory intensive workloads today by migrating to R8a instances. Visit the Amazon EC2 R8a instances page to learn more and get started on your upgrades to use the increased price performance of R8a instances today!