If you want to give your coding tool maximal freedom but you fear for the safety of your computer (you should), then letting it run in a Docker container is a good idea. If your coding tool goes rogue, it only damages your Docker container, which you can restore in an instant rather than destroying your work computer.

I will show you how to

• build a Docker image based on Ubuntu with Opencode and necessary tools installed and configured
• authenticate your LLM for Opencode and use this auth in your Docker container
• mount your local Opencode config files into your container (agents, plugins, …)
• bundle all of that in a bash script for quick access

I assume you have Docker Desktop installed and running. I also assume you are working on a coding project and have it checked out locally using git.

Create a Dockerfile in your coding project

First, you need to create a Dockerfile describing your container.

Let’s walk through it step-by-step.

# Start from the latest official Ubuntu release
FROM ubuntu:latest

# Set common environment variables
ENV DEBIAN_FRONTEND=noninteractive

# Install core tools
RUN apt-get update && apt-get install -y --no-install-recommends \
    # Core utilities needed for setup and the curl install script
    wget \
    vim \
    ca-certificates \
    curl \
    unzip \
    gnupg \
    sudo

We need to be able to interact with GitHub:

# Install GitHub CLI (gh)
 RUN curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | gpg --dearmor -o /usr/share/keyrings/githubcli-archive-keyring.gpg \
    && echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | tee /etc/apt/sources.list.d/github-cli.list > /dev/null \
    && apt-get update \
    && apt-get install -y gh \
    # Clean up APT lists
    && rm -rf /var/lib/apt/lists/*

Time to set up the local user in our container and enable it to connect to the GitHub servers:

# Add the 'ubuntu' user to the sudo group and allow passwordless sudo
RUN usermod -aG sudo ubuntu \
    && echo "ubuntu ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/ubuntu \
    && chmod 0440 /etc/sudoers.d/ubuntu

# Set the existing non-root 'ubuntu' user as the default user
USER ubuntu

# Set the working directory to the user's home folder
WORKDIR /home/ubuntu

# 1. Create the .ssh directory
RUN mkdir -p /home/ubuntu/.ssh

# 2. Add GitHub to the known_hosts file so git commands work non-interactively
# We use ssh-keyscan to fetch GitHub's public key and save it.
RUN ssh-keyscan github.com >> /home/ubuntu/.ssh/known_hosts

Next, we enable OpenCode to use the local config you have on your work computer inside the container. This will give it all the agents, plugins, etc you have configured locally:

# Create the directory structure for the auth file and fix ownership
# This prevents Docker from creating it as 'root' when the volume is mounted.
# This is important to be able to mount the OpenCode auth on docker run
#    -v ~/.local/share/opencode/auth.json:/home/ubuntu/.local/share/opencode/auth.json
RUN mkdir -p /home/ubuntu/.local/share/opencode \
    && chown -R ubuntu:ubuntu /home/ubuntu/.local/share/opencode
  
RUN mkdir -p /home/ubuntu/.config/opencode \
    && chown -R ubuntu:ubuntu /home/ubuntu/.config/opencode

There are multiple ways to get your development project into your container: copy it in, git pull it or link it from your local box into the container. I show the copy method here. You can use . in the copy command as the Dockerfile lives in your project root. Replace “ below.

# copy the project dir into the VM    
COPY --chown=ubuntu:ubuntu . /home/ubuntu/

And last but not least, we install OpenCode AI:

# Install OpenCode AI (Native Binary Method
# https://opencode.ai/docs/
RUN curl -fsSL https://opencode.ai/install | bash

Build and run the Docker container

Simply run docker build -t ubuntu-opencode . in your project root to build your container.

To be able to run it, you need to have a few things set up:

1. You need a GitHub token in your environment (e.g. export GH_TOKEN=). You generate that in the Developer Settings on GitHub.
2. You need your SSH key to be able to mount it into your container. Mine is id_ed25519
3. You need to have your local OpenCode authenticated for the model you want to use by using opencode auth login. This will generate the $HOME/.local/share/opencode/auth.json file, which we mount into our container as well.

Instead of opencode-1 you can use any name for the instance you want to spin up. -h sets it as the hostname so that your bash prompts will show it. Replace “ below.

docker run -dit --name opencode-1 \
    -h opencode-1 \
    -e GH_TOKEN \
    -w /home/ubuntu/ \
    -v "$HOME/.ssh/id_ed25519":/home/ubuntu/.ssh/id_ed25519:ro \
    -v "$HOME/.config/opencode":/home/ubuntu/.config/opencode \
    -v "$HOME/.local/share/opencode/auth.json":/home/ubuntu/.local/share/opencode/auth.json \
    ubuntu-opencode /bin/bash

Bundle Docker handling in a handy shell script

You need to change “ in the script below.

Create a file called run-opencode-docker.sh in your project dir. chmod a+x run-opencode-docker.sh to make it executable and run either directly or with the parameter rebuild if you want to re-create the container after changing your Dockerfile.

#!/bin/bash

# Configuration
CONTAINER_NAME="opencode-1"
IMAGE_NAME="ubuntu-opencode"
DOCKERFILE="Dockerfile"

PROJECT_DIR=""

LOCAL_AUTH_FILE="$HOME/.local/share/opencode/auth.json"

# Function to handle building
build_image() {
    echo "🔨 Building image from $DOCKERFILE..."
    
    if [ -z "$GH_TOKEN" ]; then
        echo "❌ Error: GH_TOKEN is not set. Cannot build."
        exit 1
    fi

    docker build -t $IMAGE_NAME -f $DOCKERFILE .
    
    if [ $? -ne 0 ]; then
        echo "❌ Docker build failed. Exiting."
        exit 1
    fi
    echo "✅ Build successful."
}

# ---------------------------------------------------------
# 0. Check for "rebuild" parameter
# ---------------------------------------------------------
if [ "$1" == "rebuild" ]; then
    echo "Force rebuild requested..."
    build_image
    exit 0
fi

# ---------------------------------------------------------
# 1. Check if Container Exists (Running or Stopped)
# ---------------------------------------------------------
if [ "$(docker ps -q -f name=$CONTAINER_NAME)" ]; then
    echo "🔄 Container '$CONTAINER_NAME' is already running."
    echo "🔗 Connecting to OpenCode..."
    exec docker exec -it -w /home/ubuntu/$PROJECT_DIR $CONTAINER_NAME /bin/bash -c "/home/ubuntu/.opencode/bin/opencode ."
elif [ "$(docker ps -aq -f name=$CONTAINER_NAME)" ]; then
    echo "🔄 Container '$CONTAINER_NAME' exists but is stopped."
    echo "🗑  Removing old container..."
    docker rm $CONTAINER_NAME
fi

# ---------------------------------------------------------
# 4. Auto-Build Image (if missing)
# ---------------------------------------------------------
if [[ "$(docker images -q $IMAGE_NAME 2> /dev/null)" == "" ]]; then
    echo "⚠  Image '$IMAGE_NAME' not found locally."
    build_image
fi

# ---------------------------------------------------------
# 5. Run New Container
# ---------------------------------------------------------
echo "🚀 Starting OpenCode..."

docker run -dit --name $CONTAINER_NAME \
    -h $CONTAINER_NAME \
    -e GH_TOKEN \
    -w /home/ubuntu/$PROJECT_DIR \
    -v "$HOME/.ssh/id_ed25519":/home/ubuntu/.ssh/id_ed25519:ro \
    -v "$HOME/.config/opencode":/home/ubuntu/.config/opencode \
    -v "$LOCAL_AUTH_FILE":/home/ubuntu/.local/share/opencode/auth.json \
    $IMAGE_NAME /bin/bash

echo "🔗 Connecting to OpenCode..."
exec docker exec -it -w /home/ubuntu/$PROJECT_DIR $CONTAINER_NAME /bin/bash -c "/home/ubuntu/.opencode/bin/opencode ."

Now you can run OpenCode on your project inside of your Docker container without risking the AI bricking your work computer. Make sure to always git pull when you start working and git push before you stop the container. You can instruct OpenCode to do this for you by putting this as instructions into your AGENTS.md file.

I’m a huge fan of GitHub actions and organization level secrets, but you can’t see what the current secret is w/i GitHub (w/o deploying a container somewhere with it exported as an ENV var). I tried to keep a copy in my former secret management tool clipperz.is/app but drift is real. It was always too scary to change a secret; I didn’t really know where it was being used and, thus, which applications I should redeploy to ensure proper operations.

This pain of manual “secret management” (syncing between clipperz and GitHub) + the stagnant codebase of clipperz (last commit was Oct 2019), made me realize it was time to take control of the situation. So, I started looking for “off-the-shelf” services.

1Password Secrets Automation ($29 / month) + Keeper Secrets Manager ($10 / month) are probably highly architected and professionally maintained tools, but should I really give up Netflix and/or Amazon Prime to migrate to one of them for my personal projects? And even if I did this, it doesn’t mean that 5 years from now when they double their prices (or go bankrupt and disappear), I’m stuck yet again trying to figure out how best to manage my secrets.

Instead, I spent the last 9 months learning a lot about running my own Vault instance, setting up a few API tokens and automating the whole secret management flow. It all starts from this crontab entry:

32 * * * * /home/ubuntu/my-ca/update_github_secrets.sh

After ingesting the requisite environment variables, it calls vault_update_secrets.py which initially gets the last-modified dates of all my Vault application secrets. Then it goes over to Github and grabs the same data in order to compare which secrets are out-of-sync (meaning updated in Vault where I now manage my secrets).

Updated (or created) secrets are highlighted via Slackbot message with a super helpful pointer to which repos use those secrets (and may need to be redeployed) via e.g. https://api.github.com/search/code?q=org%3Aackersonde+{secret_name}&type=Code. Here’s a diagram to better explain what’s going on:

Vault manages secret versions OOTB so I can rollback w/ confidence if something breaks. The automatic sync is a lifesaver and I no longer stress about rolling over & updating my secrets on a regular basis. The next, logical automation step is auto-deploying the repos identified to be using updated secrets.

If you want to learn more about running your own Vault server, just lemme know and we can chat. If there’s enough interest, I’ll write a separate post on what I’ve learned in the past year.

Since around 2016, this had been my build/deploy pipeline:

You might ask how could that ssh key be comprised. Well, back in the day, CircleCI would happily echo anything you (stupidly) told it right into the publicly available build log including secrets. Yeah, fun times…

Last summer when the new Raspberry Pi 4 was released, my scrooge senses went into overdrive realizing I could recoup the $5 month cost for my Digital Ocean droplet in less than a year and a half.

What I didn’t consider was that it would also take me a year and a half to figure out how-to securely & reliably connect my well established CI/CD pipeline to my home.

I wanted to share how I architected my working setup this summer, but that led me to think even harder about security. How secure was my VPN/ssh deploy from CircleCI? Exposing your home network on the internet is one thing. Bragging about it, quite another. I needed to do an honest assessment and go the extra mile.

Secrets

First things first – I am ruthless about stripping out secrets or any hardcoded sensitive data from my source code. I was always a big fan of using CircleCI secrets and greatly benefited from their shift to Organization wide ones (e.g. not having to copy/paste the same keys into every repository). GitHub employs a mirror copy of this setup.

Since I push all my work to public repositories, I extract all secrets and re-inject at build time. It’s definitely extra effort, but I’m happy (and proud) to share my work online, to inspire others and give back to the amazing community of opensource developers.

Using secrets is good coding practice – not only is it more secure, but it gives you flexibility for deployments in the long term.

VPN

For my deploys from CircleCI, I was connecting via some kind of Cisco IPSec on my router that:

• the manufacture provided no versioning information or update cycle for – basically, it was none of my business
• the credentials were difficult to create on the router via a script so rotating these would be painful – hurdles to automation suck

I had heard a lot about Wireguard and, within an hour of installing it, had successfully connected from my mobile network into my home. Promising! Now to hook it up to CircleCI and away we’d go.

Or so I thought. Not only did CircleCI not have the latest Ubuntu 20.04 available (which has full support for Wireguard packages by default), even after bootstrapping this OS w/i a docker container, I couldn’t get the networking right. It seemed that CircleCI was actively blocking this type of outbound VPN access from docker containers.

As the build was already running well over 5 minutes and wasn’t even close to working, I decided to shop around with other CI/CD providers. First place I went was Semaphore CI. Although I could use an off-the-shelf Ubuntu 20.04, the wireguard connection was also blocked by Semaphore.

Slowly giving up hope, I thought about Github Actions. Now, I don’t like having all my eggs in one basket, but within a few hours of setting up my configuration (which was very comparable to CircleCI), I had a successful ssh connection running ls -lrt on the k3s master node sitting in my hall. I haven’t looked back since (sorry CircleCI \o we had a lot of good years together).

Using the Wireguard container from linuxserverio.io, it’s a breeze to reissue client keys, and I do this on the same monthly cadence as regenerating my deployment SSH certificates:

sudo kubectl exec -it $(sudo kubectl get po | grep wireguard | awk '{print $1; exit}' | tr -d \\n) -- rm -Rf /config/peer_githubActions /config/wg0.conf

As you can tell, I’m running this container as a kubernetes service/deployment. The entire client peer config is uploaded as a secret to my Github org allowing me to access it from any of my repositories during deployment:

- name: Create wireguard tunnel configuration
  run: |
    sudo apt-get install wireguard resolvconf
    echo "${{ secrets.CTX_WIREGUARD_GITHUB_ACTIONS_CLIENT_CONFIG }}" | sudo tee /etc/wireguard/tunnel.conf

To save you some hair pulling, remember to uncomment net.ipv4.ip_forward=1 in your /etc/sysctl.conf on any nodes you’d like to have Wireguard running.

SSH access

Ok, we’ve got rotating Wireguard client creds – time to move on to the main event: ssh access to deploy your new code. Rotating keys here would mean carefully managing each node’s ~/.ssh/authorized_keys file, but why automate yack shaving?

Searching around the Internets led me to Aakash Yadav’s article about SSH certificates which does a great job explaining the why’s and how’s of using them for authentication. With this, I can regenerate the ssh keys used for GitHub deployments every month, create a certificate and upload these artifacts to my organization’s secrets store.

The authentication magic happens because I created a Certificate Authority (CA) keypair, adding the public key to the bottom of every node’s /etc/ssh/sshd_config file with this simple line: TrustedUserCAKeys /etc/ssh/github_deploy_ca.pub and using the private CA key (ca_key) to sign & certify any ssh keypair:

ssh-keygen -t ed25519 -a 100 -f $DIR/id_ed25519_github_deploy -q -N ""
ssh-keygen -s $DIR/ca_key -I github -n ubuntu,user -P "$CACERT_KEY_PASS" -V +5w -z $(date +%s) $DIR/id_ed25519_github_deploy

If these credentials are compromised, they’ll automatically be invalidated after some time period. I chose 5 weeks to ensure my 4 week update cron (shown below) is always one step ahead of the expiration.

Automation

# m h dom mon dow command
50 07 02 * * /home/ubuntu/my-ca/gen_new_deploy_keys.sh > /var/log/gen_new_deploy_keys.log 2>&1

We’ve done our homework when it comes to security, but now it’s time to put our money where are mouth is and automate it. By setting an expiration timer on the cert validity, we’ve forced ourselves to do the right thing. The cron executes this script which rotates both the GitHub wireguard VPN client credentials and ssh deploy key & certificate on the 2nd day of every month.

Updating GitHub secrets is not very straightforward, so I ended up writing this python script todo the heavy lifting.

Figuring out how-to actually get a token to allow this automation was a project in and of itself! Yes, I definitely do recommend you set yourself up with a GitHub organization as this will allow you to share secrets across repository builds.

I’ve created some dummy URLs in the steps below to help you out. You’ll need to replace <your-org> & <your-app> after navigating to them.

0. Create a new GitHub App in your organization. I called mine “Secrets manager” (aka <your-app>):

1. Add the following permissions:

• R/W access to Repository & Org Secrets (update the keys)
• R/W access to Repository Actions (trigger repo workflow CI/CD w/ new keys)

2. To sign access token requests, generate & download a private key for your newly created app:

3. Install the application to your organization:

4. Finally, note the install ID by navigating to your new app and looking at the URL. You’ll need this to generate your Access Token with JWT.

Securely automating your home

Now, I run my homepage, slackbot, pihole, private VPN, transmission, plex media server & wireguard from my 2 node cluster. Deployment performance is snappy and my staging environment is just a kubernetes node label.

Next up: Syncthing and taking back control of my photos.

You are a great software engineer with years of experience under your belt. You’ve seen too many technologies come and go to easily fall for the next hyped framework. You choose the tools which get the job done and have a good chance to keep working for the years to come.

Despite growing experience, your personal growth slowed to a crawl. You’re wondering what the next step should be as many of your peers have chosen to become an engineering manager.

Before you set off down this road, be aware of the implications of such a step: becoming an engineering manager is not a promotion, but a change of careers!

Why becoming an engineering manager is a change of careers

There are three things you have to completely change when switching from engineer to manager. You need to:

• stop negativity
• stop writing production code alone
• stop saying “No” without providing options

Why you need to stop negativity and focus on positive change instead

“Negativity destroys performance in the short term and precludes success and happiness in the long term”, says Mark Divine, a former Navy SEALS commander. This is why it’s so powerful for organizations to focus on making their people happy at work. Making people feel good can help free the workplace from negativity.

Negativity in engineers leads to unnecessary rewrites. I’ve seen it way too often: new engineers look at the existing code and gasp as WTF lights up in their eyes. The first reaction is always: “Oh, this code is so bad – it has to be thrown away and rewritten!”

While almost all code older than three months looks really bad, don’t forget that it’s running a (hopefully) successful business! I’ve seldom seen an engineer appreciate that and approach “legacy” code with a positive attitude to improve it.

As an engineering manager you need to spot negativity and stop it immediately if you experience it. A lot of engineers are proud to be sarcastic and enjoy devastatingly dark humor.

While funny at first, the negativity below the surface slowly erodes a positive attitude towards your team’s tasks and jeopardizes long term success. How often have you seen engineers with a negative attitude create awesome software?

Why you need to stop coding and focus on leveraging others instead

When you write production code alone, it has a 1:1 impact on your business. As an engineering manager you need to increase your impact to 1:n by leveraging others. Focus on how you can make others better, increasing their individual impacts on the business.

The sum of all those improvements is the leverage you have as an engineering manager. Only if the sum of those improvements is greater than your individual 1:1 impact, are you a worthy manager. Spending too much time writing production code on your own not only decreases your business impact, but also the time you’ll spend helping your teams!

Of course, this doesn’t mean stop coding altogether. You need to stay in touch with the daily challenges of your team and the advances of technology to be able to make good decisions in the long run.

But writing production code alone is dangerous. You’ll hardly have enough time to write clean, maintainable code between all the meetings you’re attending. If you write it over a long weekend, you’ll need to train your team mates to maintain and evolve it over time.

Becoming a bottleneck for critical parts of your company’s technical infrastructure means you’ve failed to manage your engineers.

Why you need to stop saying “No” and provide options instead

It’s easy to say, “No, this is not possible; or it’s too complex” as an engineer. While they can get away with it, as an engineering manager, this is no longer an acceptable answer.

Instead of just saying “No”, provide options how to reach the business goals despite the fact that the currently discussed option might not be possible or too difficult. What alternatives are there? Could you reduce scope? Is there an approach that moves in the desired direction with smaller steps? Could you buy a solution?

While the answer ultimately might be that the business idea is not feasible, as an engineering manager you need to make sure that all possible options have been evaluated to make an informed business decision.

The same is true for saying “Yes”.

You need to protect your teams from unreasonable requirements and feature overload. By saying “Yes” too early and too often, you’ll overload your team and steer it away from success. They’ll have to deal with seemingly easy but utterly complex tasks. A lot of engineering challenges look simple from the outside but become ugly if you look more closely.

As an engineering manager you must protect your teams from these unnatural expectations.

If you consider changing careers and become an engineering manager, be aware of these three fundamental changes. If you’re ready to take this step: Congratulations! You’re on your way to helping more and more developers be even more successful while helping your business thrive with your leadership!

Chatbots are all the rage nowadays so I’ll show you how to plug one into your own Slack channel. It’s easy, fun and, best of all, completely free!

I. Preparation

Of course, you’ll need a Slack team which I’ll refer to as yourteam.slack.com throughout this post. The free plan includes up to 10 apps or service integrations & 5GB total file storage for the team which is more than enough for hooking up GitHub, your bot and your CI tool of choice: semaphore, Travis & Circle CI are all natively supported OOTB.

From there, you can create a custom integration for your Bot:

Now, you’ve got an API token which allows your Bot to connect to Slack! Copy this and keep it handy for the next step.

II. Code

I used Norberto Lopes’ Slack API as the connector glue for my bot.

Pay particular attention how I get the Bot to only react when he’s called so it doesn’t get into any weird infinite loops (like an argument with himself).

Now for the fun part: writing your own custom commands so the Bot can do some heavy lifting for you. As a simple prototype, I asked him to fetch the 10 day forecast for my town via the sw cmd. In the last 5 years, I’ve built out many other tasks that he helps me out with.

During baseball season, he even sends me the list of yesterday’s free-to-air MLB games. The possibilities are only limited by your imagination so let loose and get creative!

III. Continuous Deployment

I use GitHub Actions (also free for building public repos) to build and deploy my Bot as a Docker container to a DigitalOcean instance with every git push. It’s much easier than it sounds and the entire process (building a new Golang binary hosted inside a container and deployed to Ubuntu) takes about 1 minute!

Note where the code loads the Slack API token from the environment – I’ve set this up in my build to read from my GitHub organization secrets. Because sometimes tokens have indigestible characters which can get strangely escaped, I’ve base64 encoded some secrets and decode it at docker runtime.

Lemme know if you have any questions or suggestions!

You have to make that release date. You need more time to get the structure of your modules right, but you don’t have it. Hitting the release date is more important than cleaning your code, so you defer the cleanup to make the deadline. You agree to take on technical debt which you’ll have to pay back later. Speeding up now to hit this release date will make you slower hitting the next one. If you’re not careful, over time, development slows down to a crawl and every feature takes weeks instead of days.

To be able to manage technical debt in agile software development, we need to fully understand tech debt. Otherwise, we’ll ruin our software. Some questions to find a technical debt definition are:

• What is technical debt (and what isn’t)?
• Where does technical debt come from?
• When is it ok to create technical debt?
• How does technical debt hurt development?
• Technical debt management: how can I measure technical debt?
• How can I reduce technical debt?
• How can I avoid technical debt?

What is Technical Debt (and what isn’t)?

According to Ward Cunningham, technical debt describes deferred, necessary work:

Technical Debt includes those internal things that you choose not to do now, but which will impede future development if left undone. This includes deferred refactoring.
Technical Debt doesn’t include deferred functionality, except possibly in edge cases where delivered functionality is “good enough” for the customer, but doesn’t satisfy some standard (e.g., a UI element that isn’t fully compliant with some UI standard).

Further describing technical debt, Fowler wrote:

You have a piece of functionality that you need to add to your system. You see two ways to do it, one is quick to do but is messy – you are sure that it will make further changes harder in the future. The other results in a cleaner design, but will take longer to put in place.

And Uncle Bob reminds us that a mess isn’t technical debt:

A mess is not a technical debt. A mess is just a mess. Technical debt decisions are made based on real project constraints. They are risky, but they can be beneficial. The decision to make a mess is never rational, is always based on laziness and unprofessionalism, and has no chance of paying off in the future. A mess is always a loss.

If your code is messy, it’s always bad – if you defer necessary work, it might be a smart business decision. Unfortunately both, creating a mess and taking on technical debt, lead to similar issues over time.

Let’s look at some examples and see what technical debt is, and what it isn’t.

A typical example for incurring technical debt in code is that huge source file every developer fears to touch. Everyone on the team knows that it needs to be broken into more manageable pieces, but it would delay upcoming features by at least two weeks. Touching this module without refactoring it is adding to the technical debt of your product. Every such change will make the eventual refactoring harder and therefore needs to be paid back later – with interest. The interest you’ll have to pay is the added complexity of every change you added. The module grows with every code addition making its refactoring (and even the decision to do so) ever more difficult.

An example for technical debt from the operations side of things is deploying your application in only one availability zone in your datacenter even though you have demanding uptime requirements. Setting up a split environment, which covers multiple availability zones is more work. It might be necessary to defer this additional work to be able to deliver critical features to your clients. In this scenario, technical debt does not materialize in added complexity but in higher risk of downtime. To avoid necessary work right now, you introduce single points of failure which will ultimately lower your uptime.

So, what isn’t technical debt?

Unnecessary complexity, dirty hacks, and unreadable code are not technical debt. There is no good reason for introducing such a mess in the first place. If developers feel that they don’t have the time to write simple, maintainable code they’re clearly lacking experience and professionalism.

“Single brain know-how” is another phenomenon often described as technical debt. But being dependent on a single person to maintain or run a critical part of your application isn’t tech debt – it’s a lethal threat to your business. You could argue that not sharing know-how because that person needs to deliver critical features is technical debt, but I disagree. No single line of code should go live without a second pair of eyes scrutinizing it. Things like pair programming or code reviews are good, professional practice and should not be traded for the illusion of higher delivery speed.

By reading this far, you should now have a clear idea what is technical debt – and what isn’t. We’ll cover the other questions in upcoming articles. Make sure you subscribe to our blog updates below.

Chef enables you to automate your infrastructure. It provides a command line tool called knife to help you manage your configurations. Using the knife EC2 plugin you can manage your Amazon EC2 instances with Chef. knife EC2 makes it possible to create and bootstrap Amazon EC2 instances in just one line – if you go through a few setup steps. In this article, I want to show you how to setup your Chef installation and AWS configuration so that you can easily bootstrap new Amazon EC2 instances with Chef’s knife.

Prepare SSH Access To Your Amazon EC2 Instances

Configure Your Amazon Security Group
As Amazon blocks all incoming traffic to your EC2 instances by default. you’ll need to open the SSH port for knife to access a newly created instance. This is pretty easy. Just login to the AWS management console and navigate to EC2 under Services > Compute. Go to Security Groups and select the default group.

Open the Actions drop-down and choose Edit inbound rules:

Then add a rule for Type SSH with Source Anywhere and save the new inbound rule:

This enables SSH connections from anywhere (0.0.0.0/0) in the world. If you want to limit access to just your home or work network, choose Custom IP instead of Anywhere and enter the corresponding net mask.

When you’re done, your default security group should look like this:

Generate Key Pair in AWS Console
To enable SSH access to your Amazon EC2 instances you need to create a key pair. Amazon will install the public key of that key pair on every EC2 instance. knife will use the private key of that key pair to connect to your Amazon EC2 instances.

Just select Key Pairs under NETWORK & SECURITY in your AWS management console and press Create Key Pair. Give it a name (e.g. knife so you know that this key pair will be used by knife) and press Create. This will create the key pair and download the private key to your local workstation.

Store the downloaded private key knife.pem in ~/.ssh/knife.pem

Prepare your SSH configuration to avoid host key mismatch errors
Open your ~/.ssh/config and add:

Host ec2*compute-1.amazonaws.com
  StrictHostKeyChecking no
  User ubuntu
  IdentityFile  /Users/mm/.ssh/knife.pem

(make sure you fix the path to your home dir)

Now, SSH access to your Amazon EC2 instances will work. Time to move on to the next step.

Configure knife Enabling it to Manage EC2 Instances

Install the knife EC2 plugin
To enable knife to manage Amazon EC2 instances you need to install the knife EC2 plugin.

Either add it to your Gemfile or install it using gem:

$ gem install knife-ec2

The knife EC2 plugin adds the ec2 sub-command to knife, which we’ll use to manage our Amazon EC2 instances.

Tell knife about your AWS credentials
Create a new user for knife in your AWS management console under Services > Administration & Security. You’ll use this user’s AWS credentials (access key ID and secret access key) to manage your Amazon EC2 instances with knife EC2.

Add the AWS credentials of your knife user to your knife configuration file ~/.chef/knife.rb:

knife[:aws_access_key_id] = "..."
knife[:aws_secret_access_key] = "........."

Now, knife should be able to use the Amazon AWS API to manage Amazon EC2 instances.

Choose an AMI for your Amazon EC2 instances
We’re going to instantiate an Amazon EC2 instance running Ubuntu 14.04 LTS.

Look here for Ubuntu 14.04 LTS AMI IDs which you can use to instantiate Amazon EC2 instances with knife.

You need to choose the right AMI for your region, architecture and root storage. Note down the AMI ID (ami-XXXXXXXX) to use it with knife.

Create an EC2 instance using Chef knife
Now, it’s time to use knife to fire up and configure a new Amazon EC2 instance.

$ knife ec2 server create -r "role[ubuntu]" -I  ami-1ed88f69 -f t2.small 
  -S knife -i ~/.ssh/knife.pem --ssh-user ubuntu --region eu-west-1 -Z eu-west-1a

• "role[ubuntu]" is the run_list I want to associate with the newly created node. You can put any roles and recipes you like here
• -I is the AMI ID you selected earlier
• -f is the Amazon EC2 instance type (see Model)
• -S is the name you gave to the SSH key pair generated in the AWS management console
• -i points to the private key file of that SSH key pair as downloaded when the key pair was created in the AWS management console
• --ssh-user the official Ubuntu EC2 AMIs use ubuntu as the default user
• --region eu-west-1 If you want your instances to be deployed in any specific Amazon AWS region, add this parameter and the desired region
• -Z eu-west-1a is the availability zone within your region

ATTENTION: make sure to kill the instance again if not needed anymore

Managing Amazon EC2 Instances With knife

Once you’ve started up at least one Amazon EC2 instance with knife, you can use knife to find running EC2 instances like this:

$ knife ec2 server list --region eu-west-1

(make sure you use the correct --region parameter)

And, if you want to get rid of an instance (terminate instance and delete the corresponding Chef node), it’s as easy as:

$ knife ec2 server delete i-XXXXXXXX --region eu-west-1
$ knife node delete i-XXXXXXXX

(i-XXXXXXXX is the ID of the instance as found in the AWS management console or a knife ec2 server list call)

After getting the initial setup right it’s a breeze to start, list, and stop your EC2 instances at Amazon with Chef’s knife ec2. With just a single command you can instantiate a new server, bootstrap it as a Chef client and run all Chef recipes defined in the run_list. Pretty sweet. What are your experiences with knife and Amazon EC2 (or other cloud providers)? Let us know in the comments…

When inflexible and wasteful software development processes are making your organization inefficient, it’s time to introduce an agile methodology. Kanban vs Scrum then becomes an essential question: Which agile software development methodology is better suited for my own situation? And is Kanban agile? What about Scrum vs agile? Confusion is spreading… Let’s have a look how to sort out all those questions.

Scrum – A Fundamental Shift

Scrum is a well-defined process framework for structuring your work. Introducing Scrum is quite a change for a team not used to agile software development: They have to start working in iterations, build cross-functional teams, appoint a product owner and a Scrum master, as well as introducing regular meetings for iteration planning, daily status updates and sprint reviews. The benefits of the Scrum methodology are well understood: Less superfluous specifications and fewer handovers due to cross-functional teams and more flexibility in roadmap planning due to short sprints. Switching your organization to use Scrum is a fundamental shift which will shake up old habits and transform them into more effective ones.

Scrum Leverages Commitment As Change Agent

The initial introduction of Scrum is not an end in itself. Working with Scrum you want to change your teams’ habits: Take more responsibility, raise code quality, increase speed. As your teams commit to sprint goals, they are intrinsically motiviated to get better and faster in order to deliver what they promised. Scrum leverages team commitment as change agent. It’s amazing to see how much teams demand from themselves – often way more you as a manager ever dared ask for.

Kanban – Incremental Improvements

The Kanban methodology is way less structured than Scrum. It’s no process framework at all, but a model for introducing change through incremental improvements. You can apply Kanban principles to any process you are already running (even to Scrum ). In Kanban, you organize your work on a Kanban board. The board has states as columns, which every work item passes through – from left to right. You pull your work items along through the in progress, testing, ready for release, and released columns. And you may have various swim lanes – horizontal “pipelines” for different types of work. The only management criteria introduced by Kanban is the so called “Work In Progress (WIP)”. By managing WIP you can optimize flow of work items. Besides visualizing work on a Kanban board and monitoring WIP, nothing else needs to be changed to get started with Kanban.

Kanban Leverages Work In Progress (WIP) Limits as Change Agent

For every column (state) on your Kanban board you should define a “Work In Progress”-Limit (WIP Limit). The WIP limit tells you how much work items are allowed to be in a certain state at any given point in time. If a state reaches its pre-defined WIP limit, no new work can enter that state. The whole team has to help clear the filled up state first. Work items trapped in a state will build highly visible clusters on the Kanban board. These clusters make bottlenecks in the progress visible – you can simply look at the Kanban Board to see where your process needs improvements. Making the need for improvement visible challenges your team to change the way they work to avoid such bottlenecks in the future. That’s how WIP limit act as change agent in Kanban.

Kanban vs Scrum

Looking at both agile software development methodologies it should be more clear what to introduce when: If your organization is really stuck and needs a fundamental shift towards a more efficient process, Scrum seems to be more appropiate. If you already have working processes, which you want to improve over time without shaking up the whole system, Kanban should be your tool of choice.

And Scrum vs Agile?

Asking for the differences between Scrum vs Agile or Agile vs Scrum is like asking for the differences between “Water” and “Ice”. Ice is water in a specific physical state. The same could be said about Scrum vs Agile. Scrum is agile in a specific shaping. It is an agile process framework. Scrum and Kanban in software development are both specific shapings of an agile software methodology. While Scrum vs Kanban or Kanban vs Scrum is comparing two agile methodologies, Scrum vs Agile is comparing a concrete example with its fundamental principles.

Steve Jurvetson

Enter the “Platform Team”: a group of build & deploy experts that jumpstart your teams down the road to operational success while providing a safety net. And, no, I’m not referring to a System Administrator with a pager. Instead, I’m suggesting a three-ply construction of automation, containerization and monitoring.

But wouldn’t hiring a full stack developer be cheaper?

The amount of job ads for “Full-Stack Developers” is insane. Hunting for them is like searching for unicorns – the closest you’ll get is an interview with an insane, cone-ornamented horse telling you your search is at an end.

Take the time and effort to teach the folks you have right now. Let them know the most important job is customer satisfaction and this means much more than just writing code.

A platform team is composed of great teachers: test managers who understand ‘release-ready’ test coverage and how to automate it; sysadmins that can point out application bottlenecks and suggest monitoring tools.

How to infect cross-functional teams with the DevOps bug?

The culture of DevOps is founded upon interdisciplinary communication. What minimally was a developer and sysadmin talking about howto deploy and operate an application, now involves product owners and UX folks in the conversation. MVP has replaced monthly releases and the pace only quickens.

With automation becoming ever easier to employ and virtual servers costing pennies during a workday, now’s the time to ready teams for self-service. Here are the first three Self-Service Mantras:

Self-Reliance (aka “We can build it”)

Work with the developer – your customer – to get their requirements right (spoiler: they’ll change during implementation ;). Make the build and deploy commands executable in the online chat room – we use hubot. And if you don’t use an online chat go checkout hipchat or slack. These rooms are essential for improving situational awareness + simplicity + remote deploys.

Attach a minimal test harness to ensure the build really did what you wanted. Is it really the right code version? Does it start correctly and fulfill it’s primary goals? Jenkins + Saucelabs is an inexpensive 1-2 knockout punch for this!

Finally, get those builds auto-deployed to your testing environment. A simple, post-build hook from Jenkins in your chat room with the corresponding URL is a honeypot for product owners keen on signing off features for release. “Houston, we are ‘GO’ for launch!”

Self-Confidence (aka “We know what’s going on”)

1. it takes:
   • 20-30 builds to work out build bugs (git commit hooks, right branches, etc.)
   • 20-30 more builds to perfect test harness (those damn outliers)
   • 20-30 more builds to prove auto deployment (yes, this is the right version)
2. with ~100 builds, the team is confident
3. they know how it works, why it breaks and how to fix it

Self-Responsibility (aka “I care about my users”)

Monitoring used to be black magic, but today, with tools like loggly, datadog and newrelic, it’s incredibly easy to get deep insight on your applications and servers. Most of these support direct alerts into your online chat room too, so the entire team can feel the pain they subject upon their hapless users. Seeing a long list of disturbing, red graphs with the first cup of coffee in the morning turns the daily standup into a Q&A session.

To provide first-level support, cross-functional teams need to be able to dig for root causes. Gathering data from all your monitoring services (you are using more than one right?), you can quickly narrow down potential problems until you’ve found what’s not working. This takes time and experience, so be patient and always available for pair-troubleshooting!

These mantras close the gap between developers and sysadmins and can be unified with the overarching principle “You build it. You run it.” Empowered developers take the time to build higher quality products, because they’re the ones getting the call during the night.

Scarcity lends Focus

We don’t have enough Platform Engineers to embed in each product team, so we loan them out for new projects as “on-site consultants”. Once they’ve gotten the engineers through the three phases above, they move on to the next construction site always with an eye on improving and adding to the company’s architectural platform and engineering excellence. This style of teaching full-stack development is much more rewarding and infinitely more successful.

There’s something to be said for time-boxing new technical initiatives. They tend to be streamlined applications that only do what’s needed. Since there’s not as much code or moving pieces involved, they’re also easier to maintain.

As you widen developers’ horizons to the world of automation and virtualization, a few may decide to pick up the standard and become platform engineers themselves. It’s a win-win situation for your employees and your business.

And what about those unsettling questions about your release process? With development teams responsible for building, shipping and operating their applications, it will suck a whole lot less!

About a year ago, I was tasked with greatly expanding our url rewrite capabilities. Our file based, nginx rewrites were becoming a performance bottleneck and we needed to make an architectural leap that would take us to the next level of SEO wizardry.

In comparison to the total number of product categories in our database, Stylight supports a handful of “pretty URLs” – those understandable by a human being. With http://www.stylight.com/Sandals/Women/ you have a good idea what’s going to be on that page.

Our web application, however, only does URLs like http://www.stylight.com/search.action?gender=women&tag=10580&tag=10630. So, nginx needs to translate pretty URLs into something our app can find, fetch and return to your browser. And this needs to happen as fast as computationally possible. Copy/paste that link and you’ll see we redirect you to the pretty URL. This is because we’ve discovered women really love sandals so we should give them a page they’d like to bookmark.

We import and update millions of products a day, so the vast majority of our links start out as “?tag=10580”. Googlebot knows how dynamic our site is, so it’s constantly crawling and indexing these functional links to feed its search results. As we learn from our users and ad campaigns which products are really interesting, we dynamically assign pretty URLs and inform Google with 301 redirects.

This creates 2 layers of redirection and doubles the urls our webserver needs to know about:

• 301 redirects for the user and bots: ?gender=women&tag=10580&tag=10630 -> /Sandals/Women/
• internal rewrites for our app: /Sandals/Women/ -> ?gender=women&tag=10580&tag=10630

So, how can we provide millions of pretty URLs to showcase all facets of our product search results?

The problem with file based, nginx rewrites: memory & reload times

With 800K rewrites and redirects (or R&Rs for short) in over 12 country rewrite.conf files, our “next level” initially means about ~8 million R&Rs urls. But we could barely cope with our current requirements.

File based R&Rs are loaded into memory for each of our 16 nginx workers. Besides 3GB of RAM, it took almost 5 seconds just to reload or restart nginx! As a quick test, I doubled the amount of rewrites for one country. 20 seconds later nginx finished reloading now running with 3.5GB of memory. Talk about “scale fail”.

What are the alternatives?

Google searching for nginx with millions of rewrites or redirects didn’t give a whole lot of insight, but I eventually found OpenResty. Not being a full-time sysadmin, I don’t care to build and maintain custom binaries – had someone else packaged this?

My next search for OpenResty on Ubuntu Trusty led me to lua-nginx-redis – perhaps not the most performant solution, but I’d take the compromise for community supported patches. A sudo apt-get install lua-nginx-redis gave us the basis for our new architecture.

As an initial test, I copied our largest country’s rewrites into redis, made a quick lua script for handling the rewrites and did an initial head-to-head test:

I included network round trip times in my test to get an idea of the complete performance improvement we hoped to realize with this re-architecture. Interesting how quite a few URLs (those towards the bottom of the rewrite file) caused significant spikes in response times. From these initial results, we decided to make the investment and completely overhaul our R&Rs infrastructure.

The 301 redirects lived exclusively on the frontend load balancers while the internal rewrites were handled by our app servers. First order of business would be to combine these, leaving the application to concentrate on just serving requests. Next, we set up a cronjob to incrementally update R&Rs every 5 minutes. I gave the R&Rs a TTL of one month to keep the redis db tidy by automatically dropping unused rewrites. Weekly, we run a full insert which resets the TTL. And, yes, we monitor the TTLs of our R&Rs – don’t want all of them disappearing over night!

The performance of Lua and Redis

We launched the new solution in mid-July. To give some historical perspective, here’s a look at our pageviews over the last 18 months:

And our average response time during the same period:

As you can see, despite rapidly growing traffic, we saw significant improvements to our site’s response time by moving the R&Rs out of files and into redis. Reload times for nginx are instant – there are no more rewrites for it to load and distribute per worker – and memory usage has dropped below 900MB!

Since the launch, we’ve double our number of R&Rs. Checkout how the memory scales:

Soon, we’ll be able to serve all our URLs like http://www.stylight.com/Dark-Green/Long-Sleeve/T-Shirts/Gap/Men/ by default. No, we’re not quite there yet, but if you need that kinda shirt…

We’ve got a lot of SEO work ahead of us which will require millions more rewrites. And now we have a performant architecture which supports it. If you have any questions or would like to know more details, don’t hesitate to contact me @danackerson.