Dave Konopka

Encrypted Amazon EC2 boot volumes with Packer and Ansible

2016-03-18T00:00:00+00:00

At the end of 2015 Amazon added support for encrypted EBS boot volumes. EBS storage volumes had offered optional encryption for some time before that. Now it’s possible to encrypt an AMI and bring up EC2 instances with fully encrypted starting volumes.

I set out recently to use encrypted EBS boot volumes for a HIPAA compliant project at ReactiveOps. It’s very easy to convert an existing AMI with an unencrypted boot volume to use encryption. I hit a few snags though building encryption into my automated AMI generation workflows using Packer and Ansible tooling.

Encrypt an existing AMI boot volume

Once you have an AMI with an encrypted boot volume any EC2 instance you launch using that AMI will have its boot volume encrypted. There’s nothing extra to do at instance launch time.

If you already have an AMI with an unencrypted boot volume it’s easy to encrypt it. This feature piggybacks on the ability to encrypt EBS volume snapshots while copying them. You can enable encryption for an AMI by copying it using the CLI, API, or the Amazon web panel and enabling encryption.

Amazon’s announcement post contains detailed instructions for copying an AMI using either the CLI or the web panel. After you make the copy you will have a new and separate AMI. Any EC2 instance you start from this new AMI will have an encrypted boot volume. There’s nothing special to do at instance launch.

Marketplace AMI’s

The first hitch I ran into involved AMI’s from the AWS Marketplace. For most projects I start with a base operating system AMI maintained by the OS’s official backing organization. Canonical publishes Ubuntu images. CentOS.org publishes CentOS images.

Unfortunately Amazon doesn’t allow AMI’s with a product code to be copied. All marketplace images have a product code. This makes it impossible to encrypt a marketplace sourced AMI by copying it into your account. The alternative is to stand up an EC2 instance from the marketplace AMI and create an AMI from the instance. The resulting AMI copy can then itself be copied and encrypted. That’s a lot of copying.

Side note: Ubuntu publishes a “Cloud Image” library of their official AMI’s. These AMI’s are separate from the AWS Marketplace. They can be copied with encryption freely unlike the marketplace versions.

Packer amazon-ebs encryption is not possible

I tend to use Packer’s amazon-ebs to build AMI’s. I had hopes for a Packer native option to produce an encrypted boot volume from a marketplace AMI. Though you can set an encryption field on volumes using the ami_block_device_mappings this doesn’t produce an encrypted EBS snapshot for the boot volume. It does not appear there is a way to do this natively with Packer. I had to resort to following up Packer AMI builds with a separate process to copy the resulting unencrypted AMI to an encrypted format.

Creating encrypted AMI’s with Ansible

The following Ansible playbook copies an unencrypted AMI and encrypts it. It tags the resulting AMI with Encrypted='true'. Ansible 2.0 or greater is required.

Ansible has an ec2_ami_copy module. Due to limitations in Boto though this module does not yet support enabling encryption on copy. There are pending PR’s for bringing this feature to the module. Until those are released, this code shells out to the AWS CLI to create the AMI copy. You’ll need to configure the AWS CLI with appropriate credentials on your machine to use this playbook.


---

- name: Create an encrypted copy of a given AMI
  hosts: localhost
  connection: local
  gather_facts: False
  vars:
    aws_region: "us-east-1"
    unencrypted_ami_id: "ami-XXXXXXXX"

  tasks:
    - name: Find the latest AMI by tags
      ec2_ami_find:
        owner: self
        ami_id: ""
        no_result_action: fail
        region: ""
        sort: name
        sort_order: descending
        sort_end: 1
      register: latest_ami

    - set_fact: unencrypted_ami_name=""

    - name: Check if encrypted copy of AMI already exists
      ec2_ami_find:
        owner: self
        name: ""
        ami_tags:
          Encrypted: "true"
        no_result_action: success
        region: ""
      register: latest_ami_encrypted

    # Proceed with encrypting only if no encrypted copy already exists
    - block:
        - name: Encrypt AMI by copying it
          shell: "aws ec2 copy-image --source-region '' --region '' --encrypted --source-image-id '' --name ''"
          register: ami_copy

        - set_fact: ami_copy_json=""
        - set_fact: encrypted_ami_id=""

        # Retry until ami is available for tagging
        - name: Confirm encrypted AMI id.
          ec2_ami_find:
            owner: self
            ami_id: ""
            no_result_action: success
            region: ""
            state: pending
          register: confirm_ami_encrypted
          until: confirm_ami_encrypted.results|length > 0
          retries: 4
          delay: 15

        - name: Tag encrypted AMI
          ec2_tag:
            region: ""
            resource: ""
            state: present
            tags:
              Name: ""
              Encrypted: "true"
          with_items: confirm_ami_encrypted.results

      when: latest_ami.results|length == 1 and latest_ami_encrypted.results|length == 0

Terraform conditionals. Sort of.

2016-03-07T00:00:00+00:00

Terraform doesn’t have support for conditionals on resources. There’s nothing like Ansible’s when statement to conditionally create Terraform resources based on a boolean variable value. At least not yet.

I value the declarative simplicity that Terraform configuration enforces by design. However, the lack of conditionals makes it difficult to create reusable, DRY Terraform modules. Luckily, there is a workaround. Terraform resources offer a count parameter that can be used as an indirect conditional. Before I give a usage example, let me explain my use case.

Why do I want conditionals?

At ReactiveOps where I work we created a Terraform module to manage client Amazon VPCs. We have a standard pattern of VPC architecture that includes public and private subnets. Our TF VPC module sets up all the subnets, route tables, and NAT EC2 appliance instances needed for that pattern. This helps us setup clients rapidly on AWS with a best-practices VPC design.

Amazon recently released NAT Gateways as an alternative to NAT EC2 instances. NAT Gateways offer availability and network throughput benefits over EC2 instances. I want to add an option to our TF module to launch a VPC with either NAT Gateways or NAT EC2 instances. We have existing customers using NAT EC2 instances, so we need to support both paths.

Our VPC TF module creates a few resources specific to NAT EC2 instances:

An EC2 instance per availability zone.
A security group with rules for the EC2 instances.
Route table entries pointing egress traffic at the EC2 instances.

Using NAT Gateways we won’t need any of the above listed resources. We will still need the VPC, subnets, route tables, new route table entries, and the NAT Gateways.

Without conditionals one option would be to isolate the NAT portions of the VPC module to two separate modules. Then we could include the VPC module and the appropriate NAT module for a given project. In this case though the VPC module is not very usable on its own without one of the two NAT modules.

Another option would be to create two separate VPC modules duplicating all the shared resources with separate resources alongside two different NAT approaches. This duplication of VPC resources would be unwieldy to support.

Using count as a conditional

Enter the count parameter. This parameter is typically used to create more than one instance of a resource. If the value is set to 0 though none will be created.

With this in mind, we can add two variables to our module’s variables.tf file:


variable "nat_instance_enabled" {
    description = "set to 1 to create nat ec2 instances for private subnets"
    default = 0
}

variable "nat_gateway_enabled" {
    description = "set to 1 to create nat gateway instances for private subnets"
    default = 0
}

Resources that are specific to one approach can then set a count value using one of the flag variables. When either of these variables is set to 0 the related resources will not be created. When either is set to 1 the related resources will be created.

Below is a security group only needed by the EC2 instance approach:


resource "aws_security_group" "nat" {
 count = "${var.nat_instance_enabled}"
 ...
}

Some resources in our VPC module already set a count to create multiple instances. For these, we can use the flag variables as multipliers. Anything multiplied by zero is zero which means the resources will or will not be created depending on the multiplier.

Below are the actual instances required by the EC2 instance approach:


resource "aws_instance" "nat" {
  count = "${var.az_count * var.nat_instance_enabled}"
  ...
}

When the module is used we can flip either flag to 1 or 0 to pick between the two NAT approaches using a single Terraform module.


# variables.tf
...
variable "nat_instance_enabled" {
  default = 0
}

variable "nat_gateway_enabled" {
  default = 0
}
...

# terraform.tfvars
...
nat_instance_enabled = 0
nat_gateway_enabled = 1
...

# main.tf
module "vpc" {
  ...
  nat_instance_enabled = "${var.nat_instance_enabled}"
  nat_gateway_enabled = "${var.nat_gateway_enabled}"
  ...
}

Caveats

One caveat here is Terraform doesn’t offer variable validation. So you have to be careful not to set nat_instance_enabled to 100 or you would create 100’s of resources. A terraform plan run should make this kind of mistake obvious.

This is admittedly a simplistic example of a conditional. It doesn’t cover conditionally setting parameters or other more complex scenarios based on variable value ranges. The Support use cases with conditional logic thread up on the Terraform repo covers a few of the more complicated use cases. I’m hopeful that thread ends up forming the basis of a kick-ass Terraform conditional feature.

Forgot about Amazon API Gateway models

2016-03-04T00:00:00+00:00

In my post on building a serverless URL shortener I neglected to mention API Gateway models at all.

A model defines a JSON schema for data passed in or out of API methods. Models are assigned to method requests/responses. They play a role in data transformations that happen within mapping templates. Models are not used for validating input data. That seems like a logical use for them so perhaps they will serve this role in the future.

It’s easy to forget about API Gateway models. You can stage a functioning API without creating any models contrary to misleading web panel error messages (more on this below). Also, Swagger API exports don’t seem to include any sign of models you do create. That’s why if you import the Swagger definition included with the serverless URL shortener repo you’ll get a misleading popup error message in the AWS web panel:

You need to define a Model to select before adding a status code output. You can define models on the Models page.

I’ve been able to stage API’s without a model despite this error message. You can get rid of the error though by creating a model. Within the API, click Resources -> Model. Add a model named “Token”.

Add the following schema which defines an object with url and token properties.

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id": "http://jsonschema.net",
  "type": "object",
  "properties": {
    "url": {
      "id": "http://jsonschema.net/url",
      "type": "string"
    },
    "token": {
      "id": "http://jsonschema.net/token",
      "type": "string"
    }
  },
  "required": [
    "url",
    "token"
  ]
}

Using custom domains with Amazon’s API Gateway

2016-02-26T00:00:00+00:00

In my last post Build a serverless URL shortener with AWS Lambda and API Gateway services I walked through creating a URL shortener service using Amazon’s API Gateway and Lambda services. One of my goals for that project was to use a custom domain instead of the randomized URL provided by API Gateway. This turned out to be trickier than I expected.

API Gateway supports custom domains but it requires an SSL certificate. And there are no integrations with Route 53 or the new AWS Certificate Manager. You’ll need to do a few things by hand to get started. Amazon’s docs on the subject are thorough. The following is what I did to get coolstory.me working with my URL shortener service.

SSL Certificate Required

You can’t get started adding a custom domain to an API Gateway service without a valid SSL certificate for the domain you want to use. If you’re experimenting with API Gateway and hesitant to commit to purchasing a certificate consider using a free provider such as Let’s Encrypt or StartSSL.

I created a free certificate with StartSSL. If you’re familiar with generating SSL certificates you can skip this next part. If not you’ll need to do a few things:

Have a custom domain registered and resolvable.
Validate ownership of your domain with your provider of choice. StartSSL provides a wizard to do this.
Generate a private key and certificate signing request using openssl on your computer:

> mkdir coolstory-ssl
> cd coolstory-ssl
> openssl genrsa -out coolstory.me.key 2048
> openssl req -new -sha256 -key coolstory.me.key -out coolstory.me.csr

openssl req will ask a series of questions. The most important one is the domain you want to use. Make sure to use exactly the domain you will use.
Submit a request for a new certificate with your provider of choice. StartSSL provides a web wizard for this step as well. You will need the CSR file you generated to do this.

Once you have the certificate you can add the custom domain under the API Gateway web panel. You’ll need the private key you generated and used to create the CSR. Create a new custom domain under the API Gateway tabs: APIs » Custom Domain Names.

Pointing Your Domain at an API Gateway Service

Once you create the custom domain name entry with API Gateway you will need to map it to an API stage. The example below shows how coolstory.me maps to the /v1 endpoint of my redir service.

The example also shows a CloudFront distribution domain name. This is domain is different than the invocation URL API Gateway gives you when you deploy a service stage. You can map a CNAME from you domain to this CloudFront domain.

Using an Apex Domain

Since this project is a URL shortener I wanted to use an apex domain coolstory.me not a longer subdomain like links.coolstory.me. Route 53 supports pointing apex domain records at internal service aliases. This works with API Gateway custom domain CloudFront distribution domains.

Since I have my domain DNS zone hosted with Route 53 I was able to create a link to the CloudFront distribution domain for the A record.

Build a serverless URL shortener with AWS Lambda and API Gateway services

2016-02-24T00:00:00+00:00

I’ve had my eye on Amazon’s Lambda and API Gateway services for a few months. Running a web application without the costs or headaches of maintaining servers is attractive. Of course, no platform is without tradeoffs. So to get myself familiar with the finer points of serverless apps I decided to launch a simple project: a URL shortener service.

With a few hours of work and a few dollars per month in cost I got a decent prototype working capable of handling millions of requests per month. This post walks through the setup along with the hitches I hit. There’s also an accompanying code repo that you can use to try this out yourself.

Project Overview

Here were the basic requirements for the project:

One HTTP endpoint to accept a JSON POST containing a short token and destination URL. These values would need to be stored somewhere.
A second HTTP endpoint to take a short token via GET, lookup a corresponding destination URL, and return a 301 redirect.
The redirection endpoint should be accessible without authentication while the POST should require authentication.
Define everything via code as much as possible to make it reproducible.
Put the service on a custom domain.

To accomplish this I needed a few components:

A front-end to take in HTTP requests.
A back-end to do something with the requests and generate responses.
A datastore to keep all the associated short tokens and destination URL’s.

Let’s take a closer look at each of these components. At the end of the post I’ll layout some gotchas and how much this all costs.

Front-end: Amazon API Gateway

At the outset I wasn’t sure Amazon API Gateway would handle everything needed for the front-end. It would need to support both application/json and text/html content as well as setting custom response codes and headers. Different content types are supported. And support for mapping Lambda function results to response headers was added at the end of 2015. API Gateway looked like it would fit the bill.

Creating an API

An API Gateway service is composed of resources and methods. Resources are essentially URL endpoints. Methods correspond to HTTP methods like GET, POST, PUT, etc.

The URL shortener’s endpoints are limited. There’s one GET to turn a provided short URL into a redirect. And a POST to take in short token and destination URL associations for storage.

Method Components

Each API Gateway method has four components. This is an example for the /{token} endpoint GET method that accepts a token and returns a redirect:

Method Request defines the incoming request including any parameters that get pulled from the request path, query string, and headers. It also supports enabling authorization for the particular method.
Integration Request maps the request to a backend. In this case I’m using Lambda Function but it also supports proxying to another HTTP service or mocking responses for development purposes. Here you pick which Lambda function will handle a single request and map request parameters to a backend data payload.
Method Response defines a collection of response status codes and headers that your service supports.
Integration Response maps return data from Lambda function execution to appropriate response codes. This can be done using regex matchers. More detail to follow on that. It also allows you to set custom response headers and setup templates to transform Lambda results.

Mapping Lambda Results to Response Headers

Part of the magic for the URL shortener is mapping Lambda function results to a response header. I needed to set the Location header to a destination URL provided by a Lambda function based on a short token lookup.

Below is the Integration Response settings for the /{token} endpoint GET method. Note the Location header mapping to a value in the Lambda function’s return JSON.

Requiring Authentication

The url shortener’s POST method requires authentication to control who can post entries. One of the authentication options API Gateway offers is an API key. API keys can be generated and associated with applications. When enabled on a method, all requests to that method must include a valid API key header to execute:

x-api-key: bkayZOMvuy8aZOhIgxq94K9Oe7Y70Hw55

This approach allowed me to require authentication on the URL POST and leave the GET endpoint open to the world.

Managing API as Code

One of my goals was to manage this project in code instead of clicking through the Amazon control panel. API Gateway offers the ability to export and import application configuration in Swagger format.

Swagger is a specification aimed at describing and documenting RESTful API’s in JSON or YAML format. A variety of services can consume Swagger files to visualize, test, or implement an API service.

I have to concede that getting started with Amazon’s API Gateway service was much easier using the web based control panel. I ultimately was able to export the service definition into a YAML file. I was also able to apply changes and clone the API using the aws-apigateway-importer tool provided by Amazon.

All that said, I created the service first by clicking through the web panel. It greatly helped with my understanding of how API Gateway applications are structured. I recommend using the web panel as a starting point.

Back-end: Amazon Lambda

Lambda supports Java, JavaScript, and Python languages. Lambda also supports uploading and executing binaries so other languages are possible. I’m most familiar with JavaScript so I started there.

Function Inputs and Outputs

Lambda functions are executed on demand. Unlike EC2, AWS ensures the compute environment whenever the function is executed. There’s no server to maintain and capacity scales as needed.

Each function can take data in the form of parameters, execute functions, and return data. Functions can import external libraries for things like connecting to other AWS services.

Lambda handler functions receive two parameters when invoked: event and context. When executed by an API Gateway event includes parameters translated from the incoming request. In the case of a token lookup, the token value from the URL is mapped to a JSON property named token. The context argument provides runtime information about the function’s execution environment.

URL Shortener Functions

The URL shortener has two simple functions:

redir_lookup_token Takes in a token value and returns a corresponding destination URL.

redir_post_token Takes in a short URL and destination URL association and stores it.

Handling Errors with API Gateway

The Lambda function’s context argument also includes methods for generating a response. It offers succeed, fail, and done methods. The last method is a helper wrapper for the first two. It takes two parameters and if the first is anything other than null it triggers a fail.

API Gateway offers regex pattern matching for mapping fail error messages to appropriate response status codes. This allows for translating backend failures to meaningful client responses. Various response status codes can be defined and associated with a regex pattern. If the Lambda function returns an error message matching one of the patterns, the client response will return the matching status code.

Each response code has its own template mapping. This allows response content to be tailored individually to each response condition. For the purposes of the short URL lookup, a 301 is the default response code for a successful request. If someone attempts a lookup for a token that doesn’t exist then a 404 should be returned.

The following shows an error condition in the token lookup Lambda function (line 7) and the corresponding 404 message response from the API Gateway.

API Gateway response statuses

Lambda function error result handling

Managing Lambda Functions with Apex

Lambda functions can be created directly in the AWS web panel or uploaded via zip file package. There are frameworks that help manage the coding and deployment of Lambda applications from a local development setup.

I considered Serverless, an extensive framework for building applications with Lambda and API Gateway. I also looked at Apex, a minimal Lambda function manager.

Apex provides a lightweight structure for organizing function code and metadata. It offers a CLI command for deploying and executing Lambda functions. I decided on the simpler approach of Apex for this project. With Apex each function is represented by a directory with a JSON metadata file and a JavaScript file containing the function code.

As you write code you can deploy changes using the apex deploy command.

> apex deploy
   • deploying                 function=post_token
   • deploying                 function=lookup_token
   • created build (1.1 kB)    function=lookup_token
   • created build (1.2 kB)    function=post_token
   • config unchanged          function=post_token
   • code unchanged            function=post_token
   • config unchanged          function=lookup_token
   • code unchanged            function=lookup_token

Once uploaded, you can execute functions from your laptop with sample input using the apex invoke command.

For small projects Apex’s structure is easy to get working quickly and it doesn’t get in the way. The project mentions plans to include API Gateway management in the future.

Datastore: DynamoDB

In keeping with the serverless theme, DynamoDB was a natural fit for storing the short and destination URLs. With DynamoDB instead of standing up a server you specify allotments of read and write capacity. Amazon ensures the allotted read and write capacities are available. You can scale up or down capacity units as needed.

DynamoDB tables are schemaless. Primary indexes can be a single value or a composite of two fields. An important limitation of composite keys is that they are hierarchical. This means that the second field is selectable in the context of the primary field. Selecting it on its own results in expensive table scans. Since the URL shortener primary index is a single short token string this was not a concern.

How much does this cost?

So how much does this all cost? The following examples assume a working example of roughly 1 million hits over a month. Prices do not include data transfer costs which will vary.

Service	Explanation	Cost
API Gateway	per 1 million requests per month	$3.50
Lambda	(hits * typical execution seconds) * (memory/1024) * $.00001667	$1.04
DynamoDB	1 read, 1 write per second	$0.58
	Grand Total	$5.12/month
	Under the AWS Free Tier this would be free.

What are the gotchas?

API Gateway response times can vary greatly. There is support for response caching though which greatly improves latency.
Importing API Gateway from Swagger doesn’t create required Lambda permissions. I had to manually reselect each Lambda function through the web panel to apply the appropriate permissions for a service newly cloned from Swagger.
API Gateway Swagger exports don’t seem to include any model schemas you create. The web panel seems to show an error message if you have no models.
- Quick followup post on creating a model: Forgot about Amazon API Gateway models
API Gateway custom domains require SSL. There’s no connection with AWS Certificate Manager of Route 53 at this point. Certificates must be manually posted.
Custom domains are pointed at API Gateway using a CNAME record. This means using a subdomain which isn’t ideal for short URLs.
- I wrote a followup post explaining how to do this: Using custom domains with Amazon’s API Gateway

On-call rotations are a team sport

2015-06-08T00:00:00+00:00

No one ever says: “I want to ack PagerDuty alerts when I grow up”. Lost sleep. Diverted focus. Scrambling under pressure. There’s no shortage of downsides to being on the frontline for a business’s technology.

Still, a well run on-call rotation is key to the proper care and feeding of any company’s technology stack. A healthy on-call rotation shines a bright light on weaknesses and implements improvements with urgency. It requires a team with sharpened communication practices to do it well.

Over time, how your team handles on-call can make all the difference between a business taking flight or drowning in technical debt.

You built it, you bought it

Limiting on-call rotations to a single person or a relatively small handful of people in an organization is an anti-pattern. This practice tends to shift problems away from the people who can act on them. When the people fielding alerts aren’t empowered to make improvements they will ultimately optimize for silencing alerts rather than addressing underlying issues. Those people will also end up burning out in short order.

Teams owning alerts for the systems that they create and/or maintain is a much healthier pattern. Teams will be much more likely to spend cycles on performance improvements and bug fixes when they’re experiencing the live fire of their work. They will roll lessons learned into improving their development process for new features. Over time this practice will also enhance a team’s ability to collect metrics and monitoring points relevant to the business.

Turn on the flood lights

Hidden problems tend to fester until they turn into emergencies. Creating a culture that prefers exposing vulnerabilities over hiding them will keep the problem scopes smaller. Pushing the issues out in the open for all to see will lessen the chances they end up swept under a rug.

Metrics are like a light switch. Tracking the occurrences and nature of on-call alerts will show which areas of a system need attention. Metrics will also reveal patterns that may not be apparent to the people fielding alerts.

A shared on-call log is one way to collect and expose these metrics. Many monitoring tools along with alerting services like PagerDuty expose some sort of API. These API’s can be mined for alert history data. Regular review of this type of alerting data in a team setting will help prioritize attention.

A chat room is another great place to shine a light on issues. Most monitoring systems can send alerts to a variety of chat services. The squeaky wheel gets the grease. Putting a stream of alerts front and center for the entire company to see will make it much harder for everyone to ignore squeaky wheels.

Build a culture of improvement

Knowing is half the battle. The other half is actually fixing stuff.

A sustainable on-call process includes time allotted for making improvements. In some situations this work may be doable during a person’s on-call rotation. In other cases it may require time after the rotation to follow-up on issues. Consider what works within your teams’ responsibilities and plan accordingly.

Solidify your team’s process for ticketing issues as they arise. Problem tickets must contain enough information to be actionable. Groom the backlog of issue tickets regularly. Work improvement tickets into the team’s pipeline regularly. Anything less will end up becoming a vanity exercise producing little more than an ever increasing ticket backlog.

Pay particular attention to systemic issues. Problems in core infrastructure tend to replicate out, compounding the number of alerts. Be on the look out for false positive and unactionable alerts. Prioritize fixing these type of issues quickly before recurring alerts result in pager fatigue across a team.

Use metrics to publicize the improvements your team makes. Wins are contagious. Every time you speed up a page load by a high percentage or eliminate a flaw that has been waking up an engineer for days, shout it from the roof tops. Sharing these kind of measured wins will reinforce a culture of improvement throughout your organization.

Fewer beeps. More sleeps. And much happier customers.

Few people get excited about taking on an on-call rotation. Handled properly though, teams can use on-call rotations to build a culture of improvement. The visibility that comes from metrics combined with clear pathways of action can be a powerful force for positive change in any organization.

The secret to organizing a tech meetup

2015-06-05T00:00:00+00:00

A little over three years ago Hector Castro brought up an idea he had for starting a DevOps meetup in Philadelphia. We both agreed that existing user groups weren’t covering many topics bridging the gap between developing and operating software. So we organized the first Philly DevOps meetup in July of 2012 and we’ve hosted regular monthly meetups since then.

Every so often somebody asks: I want to start a meetup. What do I do?

There’s plenty to talk about in response: Meeting space. Food. Speakers. Sponsorships. Scheduling. Web sites. Attracting and engaging community members.

But there’s really only one answer that matters…

Do It.

Seriously. JFDI.

Meetups are a time and a place for a community to assemble. There’s very little you can do in isolation to orchestrate a successful meetup. Logistics are important but they only take you so far. The real development of a meetup happens when people start coming together and talking to each other.

Set a topic. Book a room. Announce a date.

Your first meetup talk doesn’t need to be perfect. You can deliver a brief talk on something you’ve worked with. You could even pose an open topic for group discussion. Pick something and move forward with it.

Finding meeting space can be painful. Not many organizations are willing to offer space to the public in the evenings. Even fewer are willing to do so for free.

We held our first meeting in a local coworking space that we reached out to via their website. Soon after we moved to a room in a bar/restaurant. We had to guarantee a minimum bar tab in exchange for using the space. There were some nervous moments the first few months but we generally covered the tab minimum.

Ask around with your contacts for space connections. Maybe your employer would be willing to donate a conference room for an evening. Keep in mind, the space you start with doesn’t have to last forever. All you need is someplace adequate to get started.

Once you lock in a topic, space, and date you can begin gathering your community. Announce the first meetup to your contacts. Share it with your coworkers. Post it to your social media accounts. Ask other similar meetup organizers if they would share it within their communities. Be sure to keep your communication genuine and avoid spamming people at all costs.

Get to know the community

Meetups thrive on interaction. High quality talks are a valuable resource. Well known speakers draw attendees. Still, the interactions people have with each other are what bring them back month after month. Without interaction people will drop off quickly after attending one or two events.

Ask people what they’re interested in from the meetup. Ask them what they’re working on. Listen to what they say. Encourage them to consider giving a talk to the group. Over time you’ll build connections that will sustain the meetup.

Build in time for socializing. Find a bar/restaurant nearby. Announce that everyone will head there to continue conversations after the meetup. Encourage people to follow over. Clarity and repetition help people feel more comfortable engaging with each other.

Be inclusive

The best tech meetup communities are a continuum of experience and skill levels. There’s always something to learn from each other. It’s helpful to organize a range of content that speaks to different skill levels.

Echo chambers are pointless. A diversity of backgrounds and experiences helps everyone advance. To this end, we enforce a code of conduct for Philly DevOps events and communication channels. We do not tolerate harassment in any form. We encourage you to be kind to others. No one can insult or put down other attendees. We expect professional behavior. Anyone interested in DevOps should feel safe and comfortable attending a meetup.

The secret is: There is no secret

There really is no secret to organizing a tech meetup. If you’ve decided there is a gap you want to fill in your community, start doing it. Get people together as quickly as possible. Listen to what they have to say. Encourage people to connect with each other. You will work out all of the logistics that matter along the way.

Going remote, even when you’re on-site

2015-06-01T00:00:00+00:00

Over the past few years I’ve worked on opposite extremes of the remote/on-site workplace spectrum. My current employer maintains top notch office facilities complete with chef prepared lunches, bio-walls, lego murals, and slides. Previously, I worked with a fully distributed company of 75+ people that had a strong culture and no physical office space at all.

There are definite trade-off’s to both types of working environments. Whether your team is on-site or on-the-beach, I’ve learned that teams can find serious benefits in adopting remote-first communication patterns.

Remote-first means interacting with your team members as if you all aren’t in the same location at the same time. Even if you happen to be in the same place at the same time, consider these scenarios.

Instead of hitting up coworkers in their cubicles, try having a conversation in a community chat room. People can come along later, read the chat logs, and learn from a conversation that otherwise would have been lost between two people.

Stop handing over half the week to all-hands, hour-long blocks of in-person meetings in conference rooms. Discard conventions like blocking off 30 - 60 minutes and filling the time. Establish a common way to advertise availability amongst your team. Agree to use Google Hangouts for impromptu group video chat as needed. Doing this you can be on the other side of the building or the country and collaborate effectively with your team.

Ditch the obligatory after-the-fact employee to manager status updates. Expose activity streams from actual sources of work: Github pull requests, continuous integration/deployment job results, ticket/card updates, and service status alerts. These activity streams are transparent to everyone on the team. They’re also automatic and give a much more accurate picture than any one person’s recollection of what happened over the course of a week, month, or year.

Remote-first is a transition to an asynchronous culture. It certainly doesn’t eliminate the need for personal communication. It actually elevates the value of communication by destroying artificial roadblocks associated with engaging your fellow team members.

Back to my current on-site work life today. I’ll share one way my team has chosen to experiment with remote-first communication.

I work a team of six people managing technical operations for a variety of web applications and core services. The entire company already communicates and shares activity streams heavily over HipChat.

Still, my team leans on in-person meetings for weekly planning and daily status updates. This has served us well but as the team grows these meetings take considerably longer. It also leaves us feeling disconnected when one of us works remotely for a day.

We’ve decided to replace our daily in-person stand-up status meeting with two things. First, we each commit to updating a list of what we completed yesterday and what we plan to complete today by the start of each day. We’re using a service called iDoneThis to share everyone’s lists.

Second, we’ll each commit to being available during a half hour period in the morning. During this half hour any of us can call for a stand up, on HipChat, Google Hangouts, or in person, if there’s something that requires discussion. If there’s nothing specific to discuss, there will be no meeting.

This new approach aims to cut down time spent reading our status updates aloud. It also aims to keep us informed regardless of where our workspace is.

Time will tell how this change works for the team. Willingness to experiment and make iterative improvements is key. Even though we regularly share the same space I expect the extra flexibility will help us better deliver on both our individual and team commitments.

Repeatable, randomly ordered RSpec test runs on TeamCity

2013-03-22T00:00:00+00:00

Running a suite of tests in the same order always can mask unintentional dependencies between tests. You can end up with a test that succeeds in a full run of the suite but fails by itself.

One way to shake out these dependencies is to run tests in a random order. RSpec has an –order option that you can use to randomize test order for each run.

rspec --order rand

Recent versions of RSpec enable random ordering by default in generated spec_helper.rb files.

It’s important though to be able to reproduce a specific run order to track down issues when they pop up. RSpec accepts a seed value to initiate a specific order.

rspec --order rand:1234

At work we wanted to randomize test order on our CI server. We run our projects on TeamCity which does a nice job of parsing and reporting test run results. It does this through a custom Rake wrapper that does not output the order seed into the build logs.

A coworker Mark Starkman pointed us to this blog post with a handy shell one-liner for converting a Git commit hash into a seed numeric value.

$(git rev-parse HEAD | tr -d 'a-z' | cut -b 1-5)

Armed with this, I created a Command Line build step that populates an environment variable at runtime with a commit hash generated number. TeamCity recognizes a special format to populate parameters like environment variables at runtime.

Next up I updated the Rake step that handles RSpec runs to pass the environment variable in as the seed value.

Now each build log on our CI server includes the command needed to reproduce the test run order on any developer system.

RSpec run fails, returns exit code 0

2013-03-09T00:00:00+00:00

My team recently setup CI for a few Rails projects using TeamCity at work. When new commits hit a project’s repo if an RSpec test run passes the CI setup deploys the code to a Heroku site used by our QA team. One way or another some specs started failing on a project but the code still deployed to Heroku. That’s not how it’s supposed to work.

TeamCity looks at each command’s exit code to determine if a CI step fails or succeeds. If the exit code is 0 everything is copacetic and it continues on to the next step. Anything other than 0 means shit’s broke so it stops the process and logs failure.

It turns out the command rspec spec was returning exit code 0 on the CI server even when specs were failing.

After some research I found this Ruby bug which seemed to explain the behavior: at_exit bug with exception handling

The suggested fix solved our problem. I put the monkey patch into place in the project’s spec_helper.rb file and rspec spec exit codes are coming back “1” with failing specs and “0” with no fails on the server. TeamCity now stops short of sending broken code off to Heroku.