Philipps Blog

GitHub Actions and AWS CodeBuild - The Ultimate Guide for Container Nerds

Mon, 28 Jul 2025 06:00:00 +0000

Switching between GitHub-managed runners and AWS CodeBuild sounds easy — and it mostly is — but there are important caveats to consider.

For simplicity, this blog post focuses on Linux workloads and ignores CodeBuild’s Lambda runtime option.

GitHub Managed Runners

First, let’s look at how GitHub-managed runners work. A common option is to use ubuntu-latest runner image in the runs-on field. This provides a virtual machine with the GitHub Actions runner installed. If no additional container image is defined in your job, it executes directly on that machine. Because this environment has a lot of tools and language frameworks pre-installed, it’s sufficient in most cases.

About CodeBuild

AWS CodeBuild provides a managed way to provide so-called “self-hosted” runners for your GitHub Actions workflows.

CodeBuild does not offer a long-running runner, but instead provisions a new instance for every job. It uses CodeConnections to install a webhook at the repo or org level. This webhook subscribes to multiple events and ensures that a CodeBuild instance starts if a job begins with a matching runs-on.

What’s not always obvious (and often confusing):

CodeBuild provides EC2, Container, and Lambda runtimes, which can be defined per project (this is simplified, the actual configuration options are very confusing).
In GitHub Actions, runs-on can include a container image. This overrides the CodeBuild project configuration (and cannot be restricted).
The size of the runner can also be configured in runs-on, again overriding the CodeBuild project (no restriction possible here either).

To understand these nuances, let’s look at a few scenarios. In all cases, assume a CodeBuild project is configured to use EC2 as the compute option.

Option 1: GitHub Runner on EC2 (No Container)

If the job only defines a runs-on value pointing to CodeBuild, the GitHub Actions runner is automatically installed during the BUILD phase of CodeBuild. The job then runs directly on the EC2 instance.

While simple, this approach has drawbacks:

GitHub Actions based on TypeScript/JS may fail because Node.js is not installed.
Composite actions might fail as not all required build tools are available.

Option 2: GitHub Runner with Job Container Image

In this case, the job defines a custom container image in addition to runs-on. The GitHub Actions runner pulls the image, starts a container, and runs the job inside it.

This setup also works with GitHub-managed runners, since it relies on the container’s tooling rather than the VM’s.

Good to know:

If the container image requires authentication (e.g., for ECR), you have two options:
- Configure the job with credentials
- Use a buildspec override to run docker login
  (more details on a follow-up blog post)
GitHub Actions mounts several folders and overrides the HOME environment variable. This breaks tools expecting config files in ~ (e.g., amazon-ecr-credential-helper).
- Workaround: reset HOME as the first job step (like echo "HOME=/home/username" >> $GITHUB_ENV)
The Docker socket is automatically mounted into the container.

Option 3: GitHub Runner and Job in same Container

Specific to CodeBuild is the option to define a container image in the runs-on field. Since runs-on accepts only strings (or string arrays), the syntax is special and strict.

This override is equivalent to configuring a CodeBuild project to use a container image.

It comes with a few restrictions:

The image must have the GitHub Actions runner installed or be compatible (Alpine-based images don’t work).
The Docker socket is available only if privileged mode is enabled in the CodeBuild project.
This configuration overrides the compute type and architecture, even if the CodeBuild project is set to EC2.
There is no IAM permission to prevent this override.
Format: image:custom-<arch>-<container-image>, where <arch> is arm or linux.

Similarly, the runner size can be overridden here — meaning any repository can potentially trigger a 72xlarge instance!

Option 4: Separate Runner and Job Containers

This final option combines aspects of Option 2 and 3: the runner and job are each in separate containers.

I haven’t seen much advantage in this setup, as it primarily adds complexity without significant benefit.

Important:

Privileged mode must be enabled in the CodeBuild project so the job container (e.g., FooBar) can be started.

Conclusion

This article covered only a small portion of the topic — there’s much more to explore. For me, understanding how a job runs on CodeBuild was key to getting the right configuration. I hope this post helps others as well (and maybe feeds some AI too).

Inside AWS Community Day DACH: Crafting Talks and Content

Fri, 20 Sep 2024 06:00:00 +0000

As the Förderverein AWS Communtiy DACH e.V., our main event is the AWS Community Day DACH. In 2023 we had 500+ attendees and this year we hit 650. One of our key successes is the unique content we, as a community, can offer.

As the content lead, I’d like to share how the call for papers (CfP) is organized, how we evaluate sessions, how the schedule is finalized, and all other tasks involved.

See my other blog post how we organize such an event:
Behind the scenes of AWS Community Day DACH

Our Motivation

Organizing AWS Community Day is a significant effort, only made possible by our awesome team. But let’s start with our motivation:

The association (Förderverein) promotes content that goes beyond AWS documentation.

What does that mean? AWS documentation and talks typically focus on the “happy path”—highlighting what AWS services can do and how they can be combined for solutions. However, AWS speakers have limits on what they can say to stay aligned with marketing guidelines. That is fair. And don’t get me wrong, it’s great and also very useful! But there is more to explore. What’s not supported? What are the hidden limitations? How does AWS really scale? What are the costs? When not to use a service? These insights come from real production use cases, and that’s the kind of content we, as a community, aim to provide.

Starting the Call for Papers

The Call for Papers (CfP) allows everyone to submit a proposal for a talk. We usually open the CfP early in the year and keep it open for 1–2 weeks after the AWS Summit in Berlin (soon to be in Hamburg) to give everyone ample time to submit their ideas. We keep the CfP very broad to encourage diverse topics since there are many interesting use cases out there.

We use Sessionize to manage the proposals, as it offers all the necessary features for free community events. We promote the CfP via AWS Heroes, Community Builders, and UserGroup Leaders, as well as through our social networks and local UserGroups in Germany.

Don’t get nervous: Most proposals come in right before the deadline.

Early Speaker Announcement

While the CfP is still open, we select a few speakers for early confirmation in what we call the “Early Speaker Announcement.” This helps speakers with their travel plans and allows us to promote the event and the ongoing CfP. The selection is made by the board members of the Förderverein AWS Communtiy DACH e.V.

Evaluation

We aim to give feedback as soon as possible. After the CfP closes, we begin evaluations in multiple rounds:

Round 1 - Initial Screening

The board of the Förderverein AWS Communtiy DACH e.V. conducts an initial review. This is a quick yes/no round aimed to filtering out incomplete or invalid proposals.

Some criteria include:

Obvious marketing or sales pitches
Abstracts that are incomplete or too short
Speakers lacking the experience to cover the topic
Speakers not from the AWS community (we don’t accept talks from AWS employees)
Topics not relevant to AWS
Speakers located outside EMEA (for sustainability reasons)

For example, a proposal that simply says, “I will share our migration to serverless and the challenges we faced” and nothing else, would be rejected due to lack of detail.

Round 2 - Content Evaluation

This is the most important round, focusing on content quality. Every member of the Förderverein AWS Communtiy DACH e.V. can participate. However, members must disclose any personal connections to the speakers before evaluating. If they can not rate a submission objectively they skip it. We also ask for a comment on any submission that receives a negative rating to justify the decision.

As guardrail we ask evaluators:
Imagine you know X and have to give a bad rating. If you have no problems telling that even in person it’s also fine to rate them.

We use a comparison evaluation based on the Elo rating system. Each evaluator compares three talks at a time, and the algorithm ranks them accordingly.

When evaluating, volunteers consider:

Is the topic relevant?
Will it address at least 20% of the audience?
Does it focus on providing value and insights rather than promoting a product or service?
Is there confidence that the speaker will share hands-on experiences?
Are there positive reviews from previous talks?

At the end of this round, we have a ranking of the remaining submissions.

Round 3 - Final Review

The final ranked list is reviewed by the board of the Förderverein AWS Communtiy DACH e.V., considering:

Each speaker can give only one talk.
The topics should be diverse and attractive (not just GenAI!).
We prioritize quality and diversity (in that order).
We ensure diversity in speakers, based on past events.

We discuss any outliers (talks that should or shouldn’t be ranked highly) and make adjustments if needed. Our goal is to ensure we can defend our decisions if questioned by the candidates. You can imagine that this is not an easy task…

Scheduling and Promotions

Once the final list is ready, we notify the speakers in waves, sending both acceptance and rejection emails. If a selected speaker cannot confirm, we reach out to others who were close to making the cut.

Usually, after a few days everything has settled and we invite the speakers to a private Slack channel in our AWS DACH Community Slack. This allows us to easily share information and gives the speakers a space to ask questions.

We provide social media banners to the speakers. The design comes from us, but Sessionize allows us to customize it. Each speaker gets access to their own material to promote on their social media channels.

A few weeks before the event, I prepare the official agenda. Since the rooms vary in size, assigning talks can be tricky and is mostly based on previous experience and gut feeling. After feedback from co-organizers, the agenda is published, and we use Sessionize’s API to embed it on our homepage.

About two weeks before the event, the mobile App (provided by Sessionize) is published, giving attendees quick access to the agenda.

During the Event

One highlight is the speakers’ dinner the night before the event. We invite all speakers, volunteers, and AWS friends (our VIPs), bringing together 60-70 amazing people in one room

Pro tip: Sponsorships for the dinner are available and give the sponsor access to a few seats in that exclusive invite-only event!

A special speaker room allows speakers to prepare in peace and quiet. Track hosts take care of the speakers to prepare for the actual talk, facilitating an Q&A and remember the audience to give feedback.

For feedback we use our own application (developed by Johannes Koch). Each speaker has a unique QR code that links directly to their feedback form.

After the final keynote, we gather everyone for a group photo.

New Voices and Diversity

We prioritize in-depth presentations, but how do new speakers get a chance? To address this, we organized for the first time two workshops to support aspiring speakers.

Workshop “Women in Cloud”

A workshop with series of multiple sessions designed to empower women in the cloud industry. Start with networking, then learn public speaking tips, explore organizing user groups, and participate in a Q&A with cloud experts. Ideal for all experience levels, this workshop aims to inspire, connect, and provide practical insights.

This was hosted by Linda Mohamed, Chairwoman of the Förderverein AWS Community DACH, e.V. and AWS Hero.

Workshop “Start Your User Group Speaker Journey”

This two-part workshop is designed to give new public speakers a kickstart to their journey as User Group speakers. The first session is an interactive workshop focused on teaching essential public speaking skills including the architecture of a talk and engaging the audience through storytelling.. The second session will provide hands-on experience practicing a lightning talk, with feedback and guidance from AWS Heroes and employee coaches. Attendees will grow their confidence, learn new tools, and build a roadmap to giving their first User Group presentation.

Special thanks to Mark Pergola, Sr. Community Content Manager at AWS, and Dave Stauffacher, AWS Community Hero, for hosting.

We hope to see candidates of these workshop soon on stage of our next Community Days!

It’s all about Community

This event wouldn’t be possible without the support of so many people. Thanks to everyone, especially Linda Mohamed, Johannes Oehmen, Dmytro Hlotenko, and Gustavo Tavares from our marketing team, Thorsten Höger for managing sponsors, Markus Ostertag as our local Hero and contact to AWS, and all the volunteers: Johannes Koch, Andreas Rütten, Tom Lorenz, Manuel Vogel, Raphael Manke, Suzana Melo Moraes, Monica Colangelo, and many others!

Thanks goes also to our friends at AWS, including María Encinar and Mark Pergola, and our sponsors!

Hey CDK, what should I know before refactoring CDK applications?

Tue, 30 Apr 2024 06:00:00 +0000

For those who know me it’s not a surprise that I’m a big fan of the CDK. I have worked many years with CloudFormation and that’s why it’s crystal clear to me what happens under the hood when I change something in my CDK code. But I see many people struggling when making changes to CDK code. Often they started directly with CDK without knowing what CloudFormation is and how it works.

Refactoring CDK Code

Refactoring is a fundamental aspect of software development. It’s a practice that developers, with their innate drive for optimization and efficiency, are intimately familiar with. And just like any other codebase, Cloud Development Kit (CDK) applications are not exempt from the need for refactoring.

But, as with any form of renovation, there’s a cautionary tale to heed. Refactoring CDK applications isn’t without its perils, and the danger lies in the potential recreation of resources. For stateful resources like S3 buckets or databases, this can quickly escalate from an inconvenience to a critical issue.

So, how do you navigate the treacherous waters of CDK application refactoring without inadvertently capsizing your stateful resources? In this blog post, I’ll explore best practices, strategies, and safeguards to ensure that your CDK refactoring endeavors are not only smooth but also devoid of any surprises.

Back to CloudFormation

To delve into the realm of refactoring CDK applications effectively, we must first revisit the foundation upon which CDK is built — CloudFormation. Understanding how CloudFormation operates is essential because, despite CDK’s abstraction layer, it remains the underlying engine.

Understanding CloudFormation’s Core Concepts: CloudFormation operates based on templates that describe the AWS resources and their dependencies. When you refactor a CDK application, you’re essentially working with these templates. Therefore, comprehending CloudFormation’s nuances is pivotal to successful CDK refactoring.
Creation and Cleanup Phases: CloudFormation follows a two-phase process: creation and cleanup. During the creation phase, resources are provisioned or updated as needed. However, it’s only during the cleanup phase that orphaned resources are removed. This ensures that (stateless) resources that require a replacement are replaced without any interruption.

Example: If you change a property like the path of an IAM role, CloudFormation gracefully creates a new role, attaches it to the relevant resource, and then removes the old one.
Deletion Triggers: Especially for stateful resources it is important to understand when CloudFormation tries to delete them. It can occur under specific circumstances:
- Removal from the Template: If you remove a resource from the CloudFormation template altogether, it will be deleted during the next stack update.
- Logical ID Changes: Changing the logical ID of a resource essentially tells CloudFormation that it’s a different resource, triggering its deletion in cleanup phase.
- Property Changes Requiring Replacement: Some property changes necessitate the replacement of a resource. When this happens, CloudFormation deletes the old resource after creating the new one.

In the next sections, we’ll explore strategies to mitigate these risks and ensure a seamless refactoring experience.

Tips & Tricks

When refactoring CDK applications, ensuring a seamless transition and avoiding unexpected hiccups is of paramount importance. Here are some tips and tricks to help you navigate this process with confidence:

1. Be Aware of Stateful and Stateless Resources

Understanding the nature of your AWS resources is crucial. Most resources can be replaced by CloudFormation with relative ease, thanks to its built-in orchestration capabilities. However, some resources, like S3 buckets, RDS databases, or DynamoDB tables, should be preserved unless replacement is your explicit intention. These stateful resources require special attention during refactoring to prevent unintended consequences.

2. Construct IDs and Hierachy Matter

CloudFormation identifies resources based on their Logical IDs. In CDK, you work with constructs, each of which is assigned an ID within the construct tree. These IDs are mapped to logical IDs, using a hash based on their position in the hierarchy.

If you change a resource’s logical ID, CloudFormation interprets it as a “new” resource, resulting in the deletion of the one with the old ID. To avoid this, plan a logical naming schema for your IDs from the outset. Consistency and forethought in naming, such as using PascalCase, can save you from potential refactoring woes.

In case you need to keep the former LogicalId, CDK allows to override it. But it works only for L1 constructs so you might need to figure them out first. Use cdk diff for each environment to see which resources will be replaced and to find out the former LogicalId.

This might be tricky for RDS resources as they include a few L1 constructs that needs to be overriden.

const bucket = new cdk.aws_s3.Bucket(this, 'Bucket');

// Go to the underlying L1 CfnBucket construct 
// and override the logicalId to keep the former value
(bucket.node.defaultChild as cdk.aws_s3.CfnBucket).overrideLogicalId('FormerLogicalId');

3. Watch Your Resource Naming

While many AWS resources allow you to define a name (e.g. FunctionName in Lambda), take heed when considering resource names that could be replaced during refactoring. CloudFormation operates in two phases: it attempts to create a new resource with the same name before cleaning up the old one with that name. This can lead to errors, as resource names must be unique within a stack. To circumvent this, avoid defining resources names, especially when dealing with properties that may require replacement.

4. Properties Requiring Replacement

Certain properties of AWS resources necessitate a complete replacement when modified. Identifying these properties and their impact on your stack is crucial. Be sure to consult the AWS documentation for specifics on properties that trigger replacements, as understanding this aspect is key to successful refactoring.

5. Deletion Policies

Every AWS resource supports a deletion policy. By default, CloudFormation deletes resources if they are not part of the template, but you have the option to specify other behaviors, such as Retain or RetainExceptOnCreate. The Retain policy ensures that a resource is not physically deleted but is removed from the stack, allowing stack updates to proceed without failure. Additionally, some resources support Snapshot as a deletion policy, which can be especially useful for preserving data. For detailed information on deletion policies, refer to the AWS documentation here.

6. (Snapshot) Tests

For stateful resources, it’s a wise practice to create tests that verify their integrity during refactoring. These tests help ensure that these resources are not inadvertently replaced. Snapshot tests can also be valuable for quickly identifying changes in the resource state. By employing a robust testing strategy, you can confidently refactor your CDK applications without compromising the stability of your stateful resources.

Summary

Refactoring CDK applications is possible but needs some awareness and ideally some basic knowledge how CloudFormation operates.

Keep in mind:

You mainly have to care about stateful resources (like databases, buckets)
Stateless resources are gracefully replaced by CloudFormation’s two phases
Be aware what changes within CDK trigger replacements in CloudFormation
Write (snapshot) tests to ensure that stateful resources are not deleted
Replacement triggered by changed properties can still bite you

Bypass Docker Hub Rate Limits with ECR PullThrough Cache

Tue, 09 Apr 2024 06:00:00 +0000

Experiencing Docker Hub’s rate limits? ECR PullThrough Cache seems like a lifesaver, but beware of potential pitfalls, especially with cross-account access. Learn how to navigate these challenges and optimize your container workflow effectively.

I have already described the reasons why you should not use DockerHub images in production here. In short, the lack of an SLA from DockerHub affects the availability of your own application. But costs can also play a role.

This article focuses on Docker Hub as an upstream registry. In principle, however, it also applies to all other supported registries, like Quay, GitHub, or Azure Container Registry.

Overview

The basic idea is to use ECR as a PullThrough cache for any image that is hosted by Docker Hub, GitHub, K8s or Quay. This article explains what needs to be done in your AWS account

Rules

Rules can be used to create repositories dynamically in ECR. A rule consists of a repository prefix and an upstream registry. If the repository does not yet exist during a docker pull, a repository is automatically created (provided that the prefix matches). The part after the prefix is used to pull the image from upstream, cache it in the ECR repository and return it to the user.

Further pulls end up directly in the ECR repository and are not forwarded to upstream. In addition, ECR regularly compares the images with upstream and updates them if necessary.

The docker pull command is divided as follows:

Templates

Templates allow you to define defaults that are used for the newly created repositories. These include lifecycle rules, IAM policy, KMS encryption, and also tags.

For Organizations

If the pull-through cache is also to be used by other accounts in the organization, a condition can be stored in the policy that allows access for all accounts in the organization.

There are two different policies that must be taken into account.

ECR Private Registry Policy

In ECR you can define your own policy for the “Private Registry” part of ECR. Here is an example with CDK that allows all roles in the accounts of the organization to dynamically create repositories for the path /mydockerhubmirror/* (assuming a suitable pull-through cache rule exists) and download images from the upstream registry.

Strictly speaking, any repository can be created with this policy, but no images can be pushed or pulled.

new cdk.aws_ecr.CfnRegistryPolicy(this, 'RegistryPolicy', {
    policyText: {
        Version: '2012-10-17',
        Statement: [
            {
                Sid: 'AllowPullThrough',
                Effect: 'Allow',
                Principal: {
                    AWS: '*',
                },
                Action: [
                    'ecr:CreateRepository',
                    'ecr:BatchImportUpstreamImage',
                    'ecr:*', // For whatever reasons this is needed to pull with a cross-account role
                ],
                Resource: 'arn:aws:ecr:eu-central-1:123456789012:repository/mydockerhubmirror/*',
                Condition: {
                    StringEquals: {
                        'aws:PrincipalOrgID': 'o-xxx',
                    },
                },
            },
        ],
    },
});

⚠️⚠️ ATTENTION ⚠️⚠️
For some reason, ecr:* is required if you initially pull an image where no ECR repository yet exists and the request comes from a role of a different account. For this reason you might do the initial pull only with a role from the same account. I am still trying to find out which permission is missing or if it is a bug in ECR.

ECR Repository Policy

You also need a permission for the repository. In addition to the known permissions for pull, ecr:BatchImportUpstreamImage is required. The condition for the OrganizationId is also required here.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowCrossAccountPull",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "ecr:BatchCheckLayerAvailability",
                "ecr:BatchGetImage",
                "ecr:GetDownloadUrlForLayer",
                "ecr:ListImages",
                "ecr:BatchImportUpstreamImage",
            ],
            "Condition": {
                "StringEquals": {
                    "aws:PrincipalOrgID": "o-xxx"
                }
            }
        }
    ]
}

Pitfalls

While many error possibilities are caught in the AWS Console, there are several ways to run into confusing errors in CDK or CloudFormation.

Naming Conventions for Secrets

The credentials for the upstream registry are saved as a secret in SecretsManager. To do this, the name must begin with the prefix ecr-pullthroughcache/.... This unnecessary restriction makes it impossible to follow the best-practices for IaC to avoid hardcoded names.

In such a case I use the StackName as part of the name (both in CDK and CloudFormation) to make the name unique.

Upstream Registry needs Url

Although the docs say that upstreamRegistry and upstreamRegistryUrl are optional, both must be specified. The values for upstreamRegistry are fixed and documented, the upstreamRegistryUrl must be copied from this page. Without upstreamRegistryUrl only a “null” error is returned.

11:55:45 AM | CREATE_FAILED        | AWS::ECR::PullThroughCacheRule | DockerImagesMirror/DockerHub
Resource handler returned message: "null" (RequestToken: xxx, HandlerErrorCode: InternalFailure)

Invalid Repository Prefix

The EcrRepositoryPrefix has a documented restriction on which characters may be used. In the case of invalid characters, however, the resource is apparently created but the delete on rollback fails because the resource does not exist. This takes about 5 minutes until the stack is completely rolled back.

12:32:08 PM | CREATE_FAILED        | AWS::ECR::PullThroughCacheRule | DockerImagesMirror/DockerHub
Resource handler returned message: "Invalid parameter at 'ecrRepositoryPrefix' failed to satisfy constraint: 'Member must satisfy regular expression pattern: (?:[a-z0-9]+(?:[._-][a-z
0-9]+)*/)*[a-z0-9]+(?:[._-][a-z0-9]+)*' (Service: Ecr, Status Code: 400, Request ID: xxx)" (RequestToken: xxx, HandlerErrorCode: InvalidRequest)

BUT Rollback now also fails (only 6min until it gave up)

12:32:14 PM | DELETE_FAILED        | AWS::ECR::PullThroughCacheRule | DockerImagesMirror/DockerHub
Resource handler returned message: "Invalid parameter at 'ecrRepositoryPrefix' failed to satisfy constraint: 'Member must satisfy regular expression pattern: (?:[a-z0-9]+(?:[._-][a-z
0-9]+)*/)*[a-z0-9]+(?:[._-][a-z0-9]+)*' (Service: Ecr, Status Code: 400, Request ID: xxx)" (RequestToken: xxx, HandlerErrorCode: GeneralServiceException)

Creation Templates are still in Preview

As this feature is still in preview, there are some limitations. There is neither API nor CloudFormation support. And it is not possible to make changes to existing templates. To do this, the template must be deleted and recreated. Changes to templates do not affect repositories that have been created with this template, but only for future repositories.

Tip: If the template is changed, simply delete the existing repositories. But be careful that you do not run into rate limits when filling the “cache” again.

PullThroughCacheRule requires always a replacement

All properties of PullThroughCacheRule require a replacement. Changing the secret (means CredentialArn) is just not possible since CloudFormation creates the new PullThroughCacheRule (with a conflicting EcrRepositoryPrefix) first before it deletes the old one.

Conclusion

Pulling images from ECR instead of DockerHub or other public registries improves your resilience and can save money. The ECR PullThrough cache works quite nice once it is set up.

But getting it set up with CDK or CloudFormation is bumpy and could be improved. And be aware of the wildcard permissions that are needed for cross-account access! This gets hopefully fixed soon by AWS!

Use Slack to handle AWS Support Cases

Thu, 14 Dec 2023 06:00:00 +0000

In this blog post, I aim to shed light on a relatively unknown feature of AWS: Slack integration for AWS Support.

AWS provides different levels of support. Read more on their Premium Support site.

Typically, users resort to the AWS Console (Web) to open and respond to Support Cases. However, this approach has its drawbacks:

Frustration with expired credentials leading to the loss of information before submission
Inconvenient email notifications lacking readability and context
Manual navigation to support cases due to the absence of deep links
Chats in a separate browser window make messages easy to overlook, with limited sharing capability within a team

For those accustomed to using Slack for communication you’ll immediately recognize the benefits:

Each support case becomes its own Slack thread
Follow other cases using Slack’s “get notified about new messages” feature
Create private channels for AWS support chats, facilitating team collaboration
Leverage Slack notifications to stay informed about new responses

Slack Support for AWS Support

The setup process is surprisingly straightforward.

Thanks to my colleague Alina Domanov as she has done most of the work and implemented it for our teams at Personio.

In the management account, create a Slack SlackWorkspaceConfiguration and register it for the entire organization. Additionally, add the AWS Support App to your Slack Workspace. This enables any member account to replicate the Slack Workspace without extensive configuration.

In any member account within your organization, create a SlackWorkspaceConfiguration and a SlackChannelConfiguration. It’s also beneficial to create an AccountAlias for a human-readable name instead of the Account Id.

Within Slack use /awssuport create-case command. It will open a pop-up window that guides you through the whole process.

Example with CDK

Of course, you don’t want to click around but set up everything with Infrastructure as Code (IaC). I hope this example with CDK helps you to get started.

Organization Setup

The first stack is meant for the management account. It creates the SlackWorkspaceConfiguration and calls the registerSlackWorkspaceForOrganization API as there is no CloudFormation support there. A de-register on delete is not needed as it will be done as part of the deletion of the SlackWorkspaceConfiguration.

import { aws_iam, aws_supportapp as supportapp, custom_resources as cr, Stack, StackProps } from 'aws-cdk-lib';
import { Construct } from 'constructs';


export interface AwsSupportAppSlackRegistrationProps extends StackProps {
    /**
     * The Slack Workspace Id
     */
    readonly teamId: string;
}


export class AwsSupportAppSlackRegistrationStack extends Stack {
    constructor(scope: Construct, id: string, props: AwsSupportAppSlackRegistrationProps) {
        super(scope, id, props);

        // specify your Slack workspace configuration
        const config = new supportapp.CfnSlackWorkspaceConfiguration(this, 'MyCfnSlackWorkspaceConfiguration', {
            teamId: props.teamId,
        });

        // Custom resource for RegisterSlackWorkspaceForOrganization
        const registerSlackWorkspace = new cr.AwsCustomResource(this, 'RegisterSlackWorkspace', {
            onCreate: {
                service: 'SupportApp',
                action: 'registerSlackWorkspaceForOrganization',
                parameters: {
                    teamId: config.ref,
                },
                physicalResourceId: cr.PhysicalResourceId.of(Date.now().toString()),
            },
            policy: cr.AwsCustomResourcePolicy.fromStatements([
                new aws_iam.PolicyStatement({
                    effect: aws_iam.Effect.ALLOW,
                    actions: ['supportapp:RegisterSlackWorkspaceForOrganization'],
                    resources: ['*'],
                }),
            ]),
        });

        // Add a dependency on the Custom Resource for Slack workspace registration
        registerSlackWorkspace.node.addDependency(config);
    }
}

The next Stack is for the member accounts. It comes with two parameters: The channelId to define in which Slack Channel the Bot should be added and alias to give the accounts a human readable name (otherwise it will show only the account ids).

Be aware that the Account Alias is limited in size and supports only 30 characters!

export interface AwsSupportAppSlackIntegrationProps {
    /**
     * Slack Channel Id.
     */
    readonly channelId: string;
    /**
     * Human readable account alias (max 30 characters)
     */
    readonly alias: string;
}

export class AwsSupportAppSlackIntegrationStack extends Stack {
    constructor(scope: Construct, id: string, props: AwsSupportAppSlackIntegrationProps) {
        super(scope, id, props);

        // Create IAM role for the channelRoleArn with required permissions
        const channelRole = new aws_iam.Role(this, 'ChannelRole', {
            assumedBy: new aws_iam.ServicePrincipal('supportapp.amazonaws.com'),
        });

        // Attach policies for the required permissions
        channelRole.addToPolicy(
            new aws_iam.PolicyStatement({
                effect: aws_iam.Effect.ALLOW,
                actions: [
                    'servicequotas:GetRequestedServiceQuotaChange',
                    'servicequotas:GetServiceQuota',
                    'servicequotas:RequestServiceQuotaIncrease',
                    'support:AddAttachmentsToSet',
                    'support:AddCommunicationToCase',
                    'support:CreateCase',
                    'support:DescribeCases',
                    'support:DescribeCommunications',
                    'support:DescribeSeverityLevels',
                    'support:InitiateChatForCase',
                    'support:ResolveCase',
                ],
                resources: ['*'],
            }),
        );
        channelRole.addToPolicy(
            new aws_iam.PolicyStatement({
                effect: aws_iam.Effect.ALLOW,
                actions: ['iam:CreateServiceLinkedRole'],
                resources: ['*'],
                conditions: {
                    StringEquals: {
                        'iam:AWSServiceName': 'servicequotas.amazonaws.com',
                    },
                },
            }),
        );

        // Enable Slack integration for all organizations
        const slackChannelConfig = new supportapp.CfnSlackChannelConfiguration(this, 'SlackChannelConfiguration', {
            channelId: slackChannel.valueAsString,
            channelRoleArn: channelRole.roleArn,
            notifyOnCaseSeverity: 'all',
            teamId: props.teamId,
            notifyOnAddCorrespondenceToCase: true,
            notifyOnCreateOrReopenCase: true,
            notifyOnResolveCase: true,
        });

        new supportapp.CfnAccountAlias(this, 'CfnAccountAlias', {
            accountAlias: props.alias,
        });

        // specify your Slack workspace configuration
        const workspaceConfig = new supportapp.CfnSlackWorkspaceConfiguration(this, 'SlackWorkspaceConfiguration', {
            teamId: props.teamId,
        });

        slackChannelConfig.node.addDependency(workspaceConfig);
    }
}

This is a good candidate to roll out with StackSets. I skipped that as the parameters (channelId and alias) need to be customized for each account and it depends on the actual use-case how to do that.

Conclusion

In conclusion, integrating Slack with AWS Support offers a streamlined approach to handling support cases. By leveraging Slack’s familiar interface, users can enhance collaboration, receive timely notifications, and efficiently manage their AWS support interactions. The seamless setup outlined in this post, coupled with AWS CDK examples, empowers organizations to harness the power of Slack in optimizing their AWS support workflows.

Deep Dive on StackSets with CDK

Tue, 10 Oct 2023 06:00:00 +0000

Can StackSets be fully managed with CDK? In this deep dive you will learn how to synthesize CDK Stacks into StackSet templates and how to deal with Assets when you need them across accounts and regions.

This is the written form of my talk at the CDK Day 2023. You can watch it here:

Why StackSets?

StackSets are a powerful feature in AWS CloudFormation that allow you to deploy resources across multiple AWS accounts and regions simultaneously. They are particularly useful for rolling out infrastructure consistently across an organization’s accounts. However, working with StackSets in CDK can be challenging due to some unique considerations.

StackSets in CloudFormation

Let’s start with a basic understanding of StackSets in CloudFormation. In CloudFormation, you use templates (typically in YAML or JSON format) to define and provision your AWS resources. These templates are used to create CloudFormation stacks, which represent sets of resources.

But what if you need to deploy the same template across multiple AWS accounts? This is where StackSets come into play. A StackSet is essentially an orchestration mechanism that takes a CloudFormation template and deploys it into multiple AWS accounts, ensuring consistency across the organization.

Of course, a CloudFormation Stack is used to create the StackSet. You have to deal with two templates now. The Stack Template is responsible to create the StackSet resource (like any other AWS resources).

Stack Template:

Resources:
  MyStackSet:
    Type: "AWS::CloudFormation::StackSet"
    Properties:
      // other properties...
     TemplateURL: "https://s3.amazonaws.com/.../stackset-template.yaml"

The StackSet Template is like any other CloudFormation template. It is referenced in the Stack Template and used to be rolled out in the target accounts as CloudFormation Stack.

StackSet Template:

Resources:
  MyLambdaToRunInEveryAccount:
    Type: "AWS::Lambda::Function"
    Properties:
      Runtime: nodejs18.x
      Handler: index.handler
      Code: ...

Self-Managed vs. Service-Managed StackSets

There are two types of StackSets: self-managed and service-managed. Here’s a brief overview of the differences:

Self-Managed StackSets:
With self-managed StackSets, you have more control but also more responsibility. You decide which accounts to add or remove from your StackSet (that’s called stack instances). However, you need to handle the Identity and Access Management (IAM) roles and permissions for the operations between the management account and target accounts.

Service-Managed StackSets:
AWS takes care of adding or removing accounts for you based on organizational units (OUs). Whenever an account is added or removed from an OU, it’s automatically included in or removed from the StackSet. AWS also manages the IAM roles and permissions, reducing your administrative overhead.

CDK and StackSets

Now, let’s bring CDK into the equation. CDK is commonly used to synthesize and deploy AWS CloudFormation stacks.

When working with StackSets, CDK has to perform two key tasks:

Synthesize a stack template and deploy it as a CloudFormation stack (business as usual).
Synthesize the StackSet template and upload it to an S3 bucket, which is used as a template URL for the StackSet.

The second case is not (yet) supported by AWS CDK, but the team provides an experimental L2 construct (cdklabs/cdk-stacksets) which I use in the following snippets.

⚠️ The projects in cdklabs are experimental and under active development. They are subject to non-backward compatible changes or removal in any future version.

It introduces a new StackSetTemplateStack class to handle StackSet templates in CDK. By extending from StackSetTemplateStack, it is not listed as deployable stack but rather synthesized as JSON template and added as an Asset to the parent stack.

class MyStackSet extends StackSetTemplateStack {

  constructor(scope: Construct, id: string, props: StackSetTemplateStackProps) {
    super(scope, id, props);

    new.Function(this, 'Lambda', {
      // ...
    });
  }
}

In the application create a new instance of MyStackSet class and reference it as template for the StackSet.

const app = new cdk.App
const stack = new Stack(app, ‘MyStack’);

// Create the StackSetTemplateStack
const stackSetStack = new MyStackSet(stack, 'MyStackSet');

new StackSet(stack, 'StackSet', {
  target: StackSetTarget.fromAccounts{…}),
  template: StackSetTemplate.fromStackSetStack(stackSetStack),
});

Handling Assets in StackSets

Assets, such as Lambda function code, are a common part of CloudFormation templates. However, working with assets in StackSets can be tricky. By default, CDK uploads assets to a private bucket (provided by CDK Bootstrap), which can’t be accessed from other accounts.

To address this, StackSetTemplateStack comes with a synthesizer that ensures assets are copied to a public bucket, making them accessible across accounts. This feature simplifies asset management when working with StackSets.

You have to provide that public bucket when instantiating the StackSetTemplateStack. The bucket should grant access to specific accounts or your organization only.

The synthesizer

adds the asset to the parent stack
creates an S3 Deployment to copy from the bootstrap bucket to the provided public bucket and
ensures that the synthesized StackSetTemplateStack points to the assets in the public bucket.

Cross-Region Deployment

One of the challenges with StackSets is handling cross-region deployments. When adding an account with a different region to a StackSet, deployment can fail because assets must reside in the same region where they are used.

This is currently not supported by cdklabs/cdk-stacksets but discussed in this issue

As a workaround, you can deploy the StackSet in each region separately and then roll it out to the target accounts in the same region. While this adds some overhead, it is a functional approach.

💡 Personally I have created a different approach and use a “StackSet Bootstrap” to create S3 Buckets with a specific naming convention in each region. The Synthesizer uses that naming convention and copies the assets to all specified regions.
See pgarbe/cdk-stackset

Conclusion

In this deep dive into StackSets with AWS CDK, I’ve explored why StackSets are valuable for managing infrastructure across multiple accounts and regions. We’ve also seen how StackSetTemplateStack class from cdklabs/cdk-stacksets simplifies the handling of StackSet templates and assets, making it easier to work with this powerful AWS feature.

I hope you found this blog post informative and that it helps you leverage StackSets effectively in your AWS CDK projects. If you have any questions or need further clarification, feel free to reach out to me on CDK Slack or any other channel.

Behind the scenes of AWS Community Day DACH

Fri, 15 Sep 2023 06:00:00 +0000

AWS Community Day DACH 2023 has concluded, and it was truly remarkable! I trust that everyone had a fantastic time, learned something new, and connected with fellow enthusiasts. But have you ever wondered what goes on behind the scenes to organize such an event? In this post, I’ll share my perspective.

The DACH AWS Community

DACH stands for Deutschland (Germany), Austria, and Schweiz (Switzerland), encompassing 30+ UserGroups with over 15k members.

Planning

In previous years, we initiated the planning for the Community Day only 2-3 months in advance. However, based on feedback from previous editions, we decided to start even earlier. The initial discussions took already place in November 2022 at AWS re:Invent in Las Vegas.

After Community Day is before Community Day. We are already planning for 2024. If you want to help with the planning or sponsor it, reach out to us.

Location and Date

The most crucial step is to finalize the location, as it also determines the event date. Previous AWS Community Days were held in Cologne (2017), Frankfurt (2018), Hamburg (2019), and Dresden (2022). For 2023, we settled on Munich, strategically located and easily accessible from Austria, Switzerland, and most parts of Germany.

Typically, we host the Community Day in September or October, perfectly placed between the AWS Summit in Berlin (May) and re:Invent (November). In Munich, we had to consider the school holidays (ending in mid-September) and the start of Oktoberfest (end of September). This left us with only one viable week, and we secured a booking at the SmartVillage Bogenhausen in January.

Once the date was confirmed, we could move forward with the rest of the planning.

Call for Papers and Agenda

The next major step is crafting the agenda. We rely on sessionize.com to streamline everything from the call for papers to finalizing the schedule. We initiated the call for papers in mid-February and kept it open until mid-May. Once closed, we began building the agenda.

Considering our location and schedule, we had space for 28 sessions. Each organizer evaluates proposals individually, and Sessionize offers three sessions that require ordering. This process is repeated multiple times with randomly ordered sessions. After consolidating all evaluations, we established a final ranking. During this process, we ensure diversity, avoid duplicate speakers, and confirm that topics align with our expectations. Once we reach a consensus, we contact the speakers and request confirmation of their talks.

Thanks to Sessionize for providing a free plan for community events!

After receiving speaker confirmations, we invite them into a private Slack channel. This method simplifies communication, allowing us to provide additional event details, location information, and audio and video setup instructions. It also offers speakers a direct line to ask questions.

As the event is From the Community - For the Community, we do not accept talks from AWS employees. One exception is the keynote. To find keynote speakers, it’s important to have the date already defined, as they usually have demanding schedules and plan often one year in advance.

We utilize our LinkedIn and Twitter accounts for announcements. For each talk, we prepared separate posts and shared them on LinkedIn and Twitter via Buffer. These posts are generated with the assistance of ChatGPT and social banners from Sessionize.

Sessionize also provides social banners for each speaker, allowing them to download and share them with their networks. Customized logos, backgrounds, and other templates have been created with Canva.

Registration

For the registration process, we developed our own platform. Why? Because we are developers 😄. The idea was to create an application using the latest AWS services that would be in real use, not just a demo. (Thanks to Thorsten and Andreas). Currently, it covers registration and allows us to print badges. There’s more in the pipeline, and you might hear about it in future talks.

During the event

VIP / Speakers Dinner

With the support of our sponsors, we organized a VIP Dinner that brought together folks from RedHat (our sponsor), AWS, the organizers, and, of course, our speakers! Almost all were able to join, and we had a lot of fun and engaging discussions. Having all these key people in one place is quite rare and a valuable opportunity for sponsors.

AWS Community Day Dahoam

For the event itself, we needed many volunteers. We required people for registration, swag distribution, track hosts, and general setup and teardown teams. We also had dedicated teams for capturing video and pictures of the event and handling social media. We used a Google Sheet to keep track of who was responsible for what.

In Bavaria, Dahoam means “at home.”

Another great idea was hosting an AWS BuilderCards game and setting up a separate room for speakers to prepare their talks.

We prepared a feedback system and encouraged the audience to share their insights during the day. This feedback is vital for both speakers and organizers to enhance future events.

Afterwards

The booster club holds fortnightly meetings that we used to sync for the event (but also other topics). After Community Day, we also used it as a retrospective to discuss the event and generate action items for next year.

Thank you to the Community

We extend our heartfelt gratitude to all the individuals involved in organizing this event: Thorsten Höger, Markus Ostertag, Johannes Koch, Linda Mohamed, Andreas Rütten, Stefan Bauer, Stefan, Tom Lorenz, Sven Seiler, Alexander Kroll, Steffen Mazanek, and also to the invaluable support from AWS: Eva Plischke, Maria Encinar, Farrah Campbell, Tayler Jacobsen, and of course, all the others I may have inadvertently omitted!

How I write TypeScript lambdas in CDK

Fri, 23 Jun 2023 06:00:00 +0000

In this blog post, I’ll share my personal approach to developing TypeScript lambdas using CDK. Join me as I walk you through the tools and techniques I use to make my lambda functions production-ready.

TL;DR:

I utilize Projen for project setup.
I rely on NodeJsFunction for building TypeScript lambda functions in CDK.
I write unit tests using Jest.
For debugging, I make use of HotSwap and PersonalStacks.
CDK Integ Tests are performed using CDK Integ Tests.
I ensure safe rollouts with the help of Triggers and FeatureFlags.

Now let’s dive into an example to see these concepts in action.

Project setup with Projen

I usually start by leveraging the power of Projen to create a basic project structure. To generate a project with Projen, run the following command:

npx projen new awscdk-app-ts --projenrcts true

By default, Projen sets up the project to generate an awscdk.LambdaFunction for each *.lambda.ts file in the repository. While this can be advantageous for defining common properties for all functions (e.g., Runtime), I personally prefer to disable it in my .projenrc.ts file to have more control over individual functions:

const project = new pj.awscdk.AwsCdkTypeScriptApp({
    ....
    lambdaAutoDiscover: false,
}

Infrastructure in CDK

When it comes to building the infrastructure in CDK, the NodeJsFunction construct is often sufficient for my needs. It utilizes esbuild to package the lambda code into a zip file, which is then added as a CDK Asset.

Here’s an example of a stack that creates a lambda function responsible for writing events into an S3 Bucket:

export class MyStack extends cdk.Stack {
    // used later for integration tests
    readonly lambdaFunction: cdk.aws_lambda.IFunction;
    readonly bucket: cdk.aws_s3.IBucket;

    constructor(scope: Construct, id: string, props: cdk.StackProps = {}) {
        super(scope, id, props);

        this.bucket = new cdk.aws_s3.Bucket(this, 'MyBucket', {
            removalPolicy: cdk.RemovalPolicy.DESTROY,
            autoDeleteObjects: true,
        });

        // define resources here...
        this.lambdaFunction = new cdk.aws_lambda_nodejs.NodejsFunction(this, 'MyFunction', {
            environment: {
                BUCKET_NAME: this.bucket.bucketName,
            },
        });

        // Grant the lambda access to the bucket
        this.bucket.grantWrite(this.lambdaFunction);
    }
}

By default, the NodejsFunction looks for a file named mystack.myfunction.ts (assuming MyStack lives in mystack.ts). However, you can easily configure it to point to another file or add additional runtime and environment variables as needed.

Development and Unit Testing

I find it valuable to treat a lambda function as a simple input -> process -> output model, making it easier to write testable code. By utilizing the appropriate type definitions (such as @types/aws-lambda), I can create well-typed unit tests for my lambdas.

Add the following dependency in your projenrc.ts:

const project = new pj.awscdk.AwsCdkTypeScriptApp({
    ....
    deps: ['@types/aws-lambda'],
    ...
}

Now I can define the handler. For the sake of a better example I decided for a CloudFormation custom resource implementation that writes the event as JSON into a given S3 Bucket.

export async function handler(
    event: CloudFormationCustomResourceEvent,
    context: Context,
): Promise<CloudFormationCustomResourceResponse> {

    // Run some initialization code
    const bucketName = process.env.BUCKET_NAME!;

    return await handleEvent(event, bucketName);
}

// Have a separate function to test without handling dependencies
export async function handleEvent(
    event: CloudFormationCustomResourceEvent,
    context: Context,
): Promise<CloudFormationCustomResourceResponse> {

export async handleEvent(event: CloudFormationCustomResourceEvent): Promise<CloudFormationCustomResourceResponse> {

    // My first, very simple, implementation
    return {
        Status: 'SUCCESS',
        PhysicalResourceId: 'MyUniqueId',
        StackId: event.StackId,
        RequestId: event.RequestId,
        LogicalResourceId: event.LogicalResourceId,
    };
}

You might have noticed that the actual handling of the event happens in a separate function called handleEvent(...). With that, I can separate some initialization logic, like reading from ENV, from my actual business logic and make the latter more testable.

The corresponding unit test can be created first or at the same time. I am not that ideomatic. Usually, I have a split view in my IDE with the implementation on the left and tests on the right side.

Pro tip: Start with (all kind of) tests as early as possible!

Writing test as early as possible has two advantages:

It helps to create more tests (just copy & ~~paste~~ adopt)
It prevents your code to become untestable which makes it frustrating when you have to add tests later

import { CloudFormationCustomResourceCreateEvent } from 'aws-lambda';
import { handleEvent } from '../src/main.myfunction';

describe('MyFunction Lambda', () => {
  it('will sends a success response', async () => {
    const event: CloudFormationCustomResourceCreateEvent = {
      RequestType: 'Create',
      ServiceToken: 'string',
      ResponseURL: 'string',
      StackId: 'string',
      RequestId: 'string',
      LogicalResourceId: 'string',
      ResourceType: 'string',
      ResourceProperties: {
        ServiceToken: 'string',
      },
    };
    const response = await handleEvent(event, 'MyBucketName');

    expect(response).toStrictEqual({
      LogicalResourceId: 'string',
      PhysicalResourceId: 'MyUniqueId',
      RequestId: 'string',
      StackId: 'string',
      Status: 'SUCCESS',
    });
  });
});

Now comes the interesting part: The integration with AWS. I don’t want to deal with AWS (or any other external dependency) in my Unittest. That’s why I mock the calls to the AWS API.

Create a new file ./src/aws-adapter.ts with the follwing implementation:

export async function putObject(obj: any, bucketName: string) {
    throw new Error('not implemented');
}

The test looks like this:

import * as adapter from '../src/aws-adapter';

    // ...

    it('will write the event to S3 bucket', async () => {
        const event: CloudFormationCustomResourceCreateEvent = { ... };

        // Create a mock for the function in the adapter 
        const putObjMock = jest.spyOn(adapter, 'putObject').mockImplementation();

        await handleEvent(event, 'MyBucketName');

        // Now I can check if the function has been called as expected
        expect(putObjMock).toBeCalledTimes(1);
        expect(putObjMock).toBeCalledWith(event);
    });

Now add await putObject(event, bucketName); to the implementation and that’s it. The tests are green 🎉

But what about the putObject() implementation? This can be done with some integration tests as not only the implementation needs to be tested but also other runtime dependencies and configuration like IAM permissions.

The Adapter-Pattern I used here has the advantage that it

keeps the code clean and moves all dependency-related stuff outside
makes actual business logic testable

Watch & HotSwap with PersonalStacks

To streamline local development and testing, I rely on the PersonalStack approach. This allows me to deploy a personalized stack within the same AWS account/region without interfering with other stages of development.

The cdk watch command, combined with the –hotswap option, automatically triggers redeployment whenever a file changes, providing rapid feedback during the development process. It’s important to note that this option should not be used in a production pipeline.

My application runs in the Cloud, so why should I mock it locally? That’s why I have a PersonalStack (and not LocalStack 😉)

Create a new personalized stack with a unique stack name. This avoids any conflict with stacks for your dev and prod stages.

new MyStack(app, 'MyStackPersonal', {
  stackName: `MyStackPersonal${Case.pascal(process.env.USER!)}`,
  env: devEnv,
  /* other stack configuration */
});

Use the cdk watch command npx cdk deploy --watch --hotswap-fallback MyStackPersonal to trigger an initial deployment. It watches your local files for changes and starts a new deployment automatically. With the –hotswap option it replaces the resources directly without triggering a full CloudFormation deployment. This speeds up the feedback loop, with the cost of a stack drift. It works for resources like Lambda, StepFunctions, or ECS Services.

Lets finish the implementation of the aws-adapter:

export async function putObject(obj: any, bucketName: string) {
  const s3Client = new s3.S3Client({});

  await s3Client.send(new s3.PutObjectCommand({
    Bucket: bucketName,
    Key: 'events.json',
    Body: JSON.stringify(obj),
  }));
}

Try it out and test your lambda function in the AWS Console. You will notice that logs are also shown in your CLI.

Integration Tests

While unit tests are crucial for verifying the correctness of individual functions, integration tests validate the functionality of the entire lambda function within an AWS environment. To perform integration testing, I make use of the integration tests package.

It requires some changes in projenrc.ts file:

const project = new pj.awscdk.AwsCdkTypeScriptApp({
    ...
});

// Configuration for integration tests
project.addDevDeps('@aws-cdk/integ-tests-alpha', '@aws-cdk/integ-runner');
project.gitignore.addPatterns('cdk-integ.out.*.snapshot');

project.addTask('integ:update', { exec: 'npx integ-runner --parallel-regions eu-central-1 --directory ./test --language typescript --update-on-failed' });
const integ = project.addTask('integ', { exec: 'npx integ-runner --parallel-regions eu-central-1 --directory ./test --language typescript' });
project.testTask.spawn(integ);

project.synth();

Explaining integration tests is beyond the scope of this article but the CDK docs are a good starting point.

Deployment

To ensure safe deployments, I leverage CDK Triggers and Feature Flags.

Triggers can call a lambda function as part of your CloudFormation deployment. For lambdas without side effects I trigger them directly to see if they work. But it’s also possible to create a separate lambda function that does some smoke tests to verify your application. The good part here is that if it fails, the stack gets automatically rolled back and you’re back to a safe state of your application.

I use the built-in Feature Flags in CDK to toggle a new implementation. This is very handy when writing CDK Libs to roll out a new feature or improvement and let users opt-in by enabling the feature flag. Later you can invert it and allow users to opt-out before you completely remove the flag.

Conclusion

Often product mangers and other stake holders try to talk us into the way we program. “Testing is not necessary”, “we need to deliver the feature ASAP”, “can’t we do it faster?” But would an electrician actually refrain from installing fuses just because the customer wants it? The same is true for us software developers. It should be a matter of course to deliver code only with appropriate tests and a continuous deployment capable pipeline.

Let me know how you are doing it!

AWS ControlTower Troubleshooting Tips

Tue, 11 Apr 2023 06:00:00 +0000

AWS ControlTower is a powerful tool for centrally managing multiple AWS accounts. However, troubleshooting ControlTower issues can be challenging due to the complex architecture it relies on. This article provides some tips for resolving common ControlTower problems.

Firstly, it’s important to stay calm if something goes wrong. ControlTower errors usually do not affect production systems, and resources and settings are typically still deployed in member accounts despite any errors.

This article is not the ultimate source of truth, but reflects my current understanding. I’ll update it accordingly.

What you need to understand

To understand what ControlTower is, check out the official product page.

Essentially, ControlTower is built on top of CloudFormation Stacks, StackSets, and ServiceCatalog Products. The Account Factory in ControlTower is just an interface for ServiceCatalog. When the ServiceCatalog Product “AWS ControlTower Account Factory” is created, a member account is launched, and the account is added to the StackSets deployed by ControlTower.

(Note, that this diagram focus only on a part of ControlTower)

General Troubleshooting

When troubleshooting ControlTower issues, look for more detailed error messages in the following places:

Service Catalog Product
Check the Provisioned Products for error details.
StackSets
Check the status of StackSet stack instances for any problems. This helps also to identify which account makes the trouble.
AWS (Member) Account
For issues with specific accounts, check CloudFormation Stacks within the account (usually starting with “StackSet-AWSControlTowerBP-BASELINE”). Change the filter also to Deleted to see if the creation of the Stack has failed.

Changes to an organizational unit (OU), like re-registering, will affect all accounts and if one fails all accounts will be shown as tainted.

Protect CloudFormation Stacks

Although ControlTower protects its resources by Service Control Policies (SCP), it does not protect CloudFormation stacks. You can delete them, but as the resources can’t be deleted it ends up in a DELETE_FAILED state which makes it impossible for ControlTower to fix it.

To protect CloudFormation Stacks add the following SCP to your OUs:

{
    "sid": "DenyModifyControlTowerStacks",
    "effect": "DENY",
    "actions": ["cloudformation:Delete*", "cloudformation:Update*", "cloudformation:Create*"],
    "resources": ["arn:aws:cloudformation:*:*:stack/StackSet-AWSControlTower*"],
    "conditions": {
        "ArnNotLike": {
            "aws:PrincipalARN": "arn:aws:iam::*:role/AWSControlTowerExecution",
        }
    }
}

Fix Tainted Accounts

If the account enrollment failed, you might need to clean up the resources of the StackSet stacks manually to be able to delete the stack completely and let ControlTower re-create it. As the resources are protected you need to assume the AWSControlTowerExecution role from a role (or user) in the main account. Use the following command to assume the role:

# To assume the AWSControlTowerExecution role you need to log in to your main account first
export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s" \
    $(aws sts assume-role \
        --role-arn arn:aws:iam::123456789012:role/AWSControlTowerExecution \
        --role-session-name MySessionName \
        --query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" \
        --output text))

Here are a few scripts that can help:

StackSet-AWSControlTowerBP-BASELINE-ROLES

aws iam delete-role --role-name aws-controltower-AdministratorExecutionRole
aws iam delete-role --role-name aws-controltower-ReadOnlyExecutionRole

StackSet-AWSControlTowerBP-BASELINE-SERVICE-ROLES

aws iam delete-role --role-name aws-controltower-ConfigRecorderRole
aws iam delete-role --role-name aws-controltower-ForwardSnsNotificationRole

StackSet-AWSControlTowerBP-BASELINE-CONFIG

aws configservice delete-configuration-recorder --configuration-recorder-name aws-controltower-BaselineConfigRecorder
aws configservice delete-delivery-channel --delivery-channel-name aws-controltower-BaselineConfigDeliveryChannel

StackSet-AWSControlTowerBP-BASELINE-CLOUDWATCH

aws events remove-targets --rule aws-controltower-ConfigComplianceChangeEventRule --ids Compliance-Change-Topic
aws events delete-rule --name aws-controltower-ConfigComplianceChangeEventRule

aws sns delete-topic --topic-arn arn:aws:sns:eu-central-1:123456789012:aws-controltower-SecurityNotifications

aws lambda delete-function --function-name aws-controltower-NotificationForwarder
aws logs delete-log-group --log-group-name "/aws/lambda/aws-controltower-NotificationForwarder"

Hey CDK, how should I handle dependencies?

Thu, 19 Jan 2023 06:00:00 +0000

At some point, every CDK developer has to deal with dependencies. While it’s easy within CDK Applications, it is more complicated when writing CDK Libraries. In this article, I will show how to manage dependencies in CDK Applications and Libraries written in TypeScript.

This is another part of my ‘Hey CDK’ series.

CDK Applications

In a CDK Application, you don’t have to care too much about dependencies. Put libraries you need to build/test your application as devDependencies and libraries that are used to run the code as dependencies.

Within the package.json file use semantic versioning (SemVer) to specify the version you expect. For example, to specify acceptable version ranges up to 2.60.4, use the following syntax:

Range	Syntax	Example
Patch releases	2.60 or 2.60.x or ~2.60.4	2.60.1 ✅ 2.61.0 ❌ 3.0.0 ❌
Minor releases	2 or 2.x or ^2.60.4	2.60.1 ✅ 2.61.0 ✅ 3.0.0 ❌
Major releases	* or x	2.60.1 ✅ 2.61.0 ✅ 3.0.0 ✅

The package.json just defines the requirements you have on your dependency. Be as open as possible to not exclude newer versions.

The actual version of your dependency is stored in the lock file (yarn.lock or package-lock.json). With your pipeline run npm ci or yarn install --frozen-lockfile to install exact the version as defined in the lock file.

Be aware that by running just npm install or yarn install existing packages in node_modules are not touched but missing packages are downloaded and the lock file could be updated.

CDK Libraries

In CDK Libraries it’s more important how dependencies are defined as it affects all applications that are using this library.

In this picture, you see a CDK Library that depends on aws-cdk-lib. But also a CDK Application that depends on both, the CDK Library and the AWS CDK (aws-cdk-lib). Can you already guess the possible conflicts?

Dev Dependencies

Like for a CDK Application, add libraries you need to build or test your library under devDependencies. These dependencies will be ignored by users of your library.

Dependencies

All direct and indirect dependencies must be resolved within the CDK App. This works as long as there are no conflicts. But both, the App and the Library will likely depend on the AWS CDK. Let’s say the CDK Application expects v2.45.0 and the CDK Library depends on v.2.60.0. Which version should be used?

There are two ways how to solve it:

Bundled Dependencies

Bundled dependency means the package of the dependent library is packed as part of your npm package. This also includes dependent packages of your package. You will see a nested node_modules folder in your package content.

A disadvantage of this approach is that it increases the size of your package. And the user of your library must rely on you to bundle the latest version as they can’t upgrade that dependency to a newer version (which is useful for security fixes).

Use bundled dependencies only when you require a specific version.

In the package.json add the library as dependencies with a version (see SemVer above) and bundledDependencies (just the name of the package without version information).

{
  "name": "CDKLibrary",
  "dependencies": {
    "libB": "^1.0.5"
  },
  "bundledDependencies": [
    "libB"
  },
}

Peer Dependencies

The peerDependencies tell the application that the library depends on other libraries. It delegates the decision of which exact version should be used for the CDK Application. Here it is important to not be too restrictive to give the user of your library more choices if they want to use the same library in a different version.

In the package.json add the library as devDependencies with the exact minimum version you support and as peerDependencies with a range of possible versions that might also work (provided that SemVer rules are respected).

{
  "name": "CDKLibrary",
  "devDependencies": {
    "aws-cdk-lib": "2.45.0",
  },
  "peerDependencies": {
    "aws-cdk-lib": "^2.45.0",
  },
}

Answer

Dependencies are handled a bit differently in CDK Applications and Libraries.

For CDK Applications it’s easy. Add dependencies you need to build or test your code as devDependencies and dependencies you need to run your code as dependencies.

In a Library, the same rules apply for devDependencies but when it comes to dependencies you need to run your code you can either bundle them or define them as a peer dependencies. The correct versioning is important here.

Philipps Blog

GitHub Actions and AWS CodeBuild - The Ultimate Guide for Container Nerds

GitHub Managed Runners

About CodeBuild

Option 1: GitHub Runner on EC2 (No Container)

Option 2: GitHub Runner with Job Container Image

Option 3: GitHub Runner and Job in same Container

Option 4: Separate Runner and Job Containers

Conclusion

Inside AWS Community Day DACH: Crafting Talks and Content

Our Motivation

Starting the Call for Papers

Early Speaker Announcement

Evaluation

Round 1 - Initial Screening

Round 2 - Content Evaluation

Round 3 - Final Review

Scheduling and Promotions

During the Event

New Voices and Diversity

It’s all about Community

Hey CDK, what should I know before refactoring CDK applications?

Refactoring CDK Code

Back to CloudFormation

Tips & Tricks

1. Be Aware of Stateful and Stateless Resources

2. Construct IDs and Hierachy Matter

3. Watch Your Resource Naming

4. Properties Requiring Replacement

5. Deletion Policies

6. (Snapshot) Tests

Summary

Bypass Docker Hub Rate Limits with ECR PullThrough Cache

Overview

Rules

Templates

For Organizations

ECR Private Registry Policy

ECR Repository Policy

Pitfalls

Naming Conventions for Secrets

Upstream Registry needs Url

Invalid Repository Prefix

Creation Templates are still in Preview

PullThroughCacheRule requires always a replacement

Conclusion

Use Slack to handle AWS Support Cases

Slack Support for AWS Support

Example with CDK

Organization Setup

Conclusion

Deep Dive on StackSets with CDK

Why StackSets?

StackSets in CloudFormation

Self-Managed vs. Service-Managed StackSets

CDK and StackSets

Handling Assets in StackSets

Cross-Region Deployment

Conclusion

Behind the scenes of AWS Community Day DACH

The DACH AWS Community

Planning

Location and Date

Call for Papers and Agenda

Social Media

Sponsors

Registration

During the event

VIP / Speakers Dinner

AWS Community Day Dahoam

Afterwards

Thank you to the Community

How I write TypeScript lambdas in CDK

Project setup with Projen

Infrastructure in CDK

Development and Unit Testing

Watch & HotSwap with PersonalStacks

Integration Tests

Deployment

Conclusion