Deploying packages with Helm can require a lot of ceremony on the command line. It's not senseless ceremony: there are use cases where the flexibility of Helm is helpful. Yet, if you're anything like me, you want as little typing as possible when you're ready to throw a thing at Kubernetes. This post is about some simple tooling I created to achieve that.

This post is part 3 in my series about running my own Kubernetes deployment using Digital Ocean managed Kubernetes. If you're interested in what else is involved in my setup, be sure to check out part 1 and part 2.

Deploying something to my cluster using Helm loosely involves a Helm invocation that looks something like this:

helm upgrade deployment_name \
  chart_name \
  -f override_values.yaml \
  --tls \ # if you don't have the env var set
  --install \
  --namespace target_namespace

This is fine until you've got to keep everything straight for more than two deployments or if you're not doing deployments frequently. At work, we built out a good amount of tooling to manage Helm for us. Honestly, though, I didn't need anything near that complex for my personal cluster. I settled down one evening not long ago, stretched my bash muscles, and came up with something "good enough" for me.

Matt's k8s-specs repo

I have a git repository that I've imaginatively named k8s-specs. The structure is pretty straightforward:

k8s-specs/
├── deploy.sh
├── fdn
│   ├── CHART_NAME
│   ├── dependencies.yaml
│   └── values.yaml
├── ghost-test
│   ├── CHART_NAME
│   ├── dependencies.yaml
│   └── values.yaml
├── mysql
│   ├── CHART_NAME
│   ├── dependencies.yaml
│   └── values.yaml
├── nfs
│   ├── CHART_NAME
│   └── values.yaml
└── traefik
    ├── CHART_NAME
    └── values.yaml

Under k8s-specs, each folder is a thing I want to deploy. The name that's used for the deployment name in Helm is the same as the folder name. Each folder has up to three items in it:

1. The CHART_NAME file that is a valid reference to the Helm chart I want to use. This is the only file that's required.
2. The dependencies.yaml file containing any Kubernetes resources that must be added before the Helm chart is run. I typically use this to deploy secrets or something similar.
3. The values.yaml file that overrides the default settings for the Chart for my deployment.

This turns out to be a pretty excellent setup. It can even support a chart actually living in one of these folders, if needed.

The deploy.sh script is a few lines of bash that does the required legwork to make things happen in the Kubernetes cluster. It takes up to two arguments: the first is name of the folder you want to deploy, the second is the namespace in Kubernetes you want to deploy it in. The second argument is optional and only matters for initial deployments. Subsequent deployments of a chart will always go to the namespace the initial deployment targeted.

The source of my deploy script is:

#!/bin/bash

NAME=$1
CHART_NAME=$(cat $NAME/CHART_NAME)
NAMESPACE=${2:-default}

echo "Name:" $NAME
echo "Chart:" $CHART_NAME
echo "Namespace: " $NAMESPACE

echo "Ensuring dependencies are installed..."

if [[ -f "$NAME/dependencies.yaml" ]]; then
  echo "Applying dependencies..."
  kubectl apply -f $NAME/dependencies.yaml
else
  echo "No dependencies to apply."
fi

echo "Running helm"
helm upgrade $NAME $CHART_NAME -f $NAME/values.yaml --tls --install --namespace $NAMESPACE

Now whenever I have changes I want to deploy to the blog, I just drop into my k8s-specs repository on my terminal and type ./deploy.sh fdn to get rolling.

I've been using this setup for many weeks now and have found it to age pretty well so far. This is more than I can say for a lot of Chef cookbooks I've written in a prior life.

After you've got your initial Kubernetes cluster up and running the next step is to decide what core cluster services you want to install. I'll walk through what I've deployed in my DigitalOcean cluster. This post is Part 2 of my experience running a personal Kubernetes Cluster on DigitalOcean's managed Kubernetes platform. If you missed part 1, you can find it here.

The core services I chose to deploy to my Kubernetes cluster are:

• Sealed Secrets — A secure secret implementation for Kubernetes.
• Tiller — The server-side component for the Helm package manager.
• Traefik — An HTTP server that serves as my edge proxy.

Below I'm going to walk through what each of these do in detail and some specifics about how I use each of them.

Sealed Secrets

Kubernetes ships with a Secrets API that gives cluster users the hooks to protect credentials and other sensitive information. Secrets themselves, however, don't provide any kind of encryption on the data provided to them. The Kubernetes default "Opaque" secrets are just bags of keys with base64-encoded values. A base64-encoded secret isn't something you'd want to put into, say, GitHub. Cluster users essentially have two options for secure secret storage.

The first is to manage secrets totally apart from version control. The second option is to deploy tooling that handles the encryption and decryption of secrets into their base64-encoded counterparts for Kubernetes.

I chose to use Bitnami's Sealed Secrets Controller.

When you deploy the controller to your cluster it generates a public and private key pair for encryption and decryption, respectively. It also defines a new resource type, sealedsecret that can be used from kubectl and other tooling. A sealed secret is encrypted using the public key the controller generates. When you push a new sealedsecret resource to your cluster, the controller decrypts it using its private key and creates a corresponding Kubernetes secret that can be mounted into a Pod.

The YAML file for the sealed secret is safe to check into GitHub alongside your other Kubernetes resources because it can only be decrypted with the private key in your cluster. If it were to fall into the hands of an unsavory foe, it wouldn't do them much good unless they are the NSA.

If you are deploying the Sealed Secrets Controller on a cluster with many users, I recommend using role-based access control to block individual users from interacting with the secret resource from kubectl. This will prevent users from being able to access the base64 encoded versions of secrets. It will also serve as a useful reminder that folks should be using sealedsecret in your production cluster (because they won't be able to directly create the less secure secret object).

Installing Sealed Secrets

The repository for the Sealed Secrets Controller has great installation instructions. Just a few simple kubectl commands stand between you and the ability to create sealed secrets to your heart's content.

You'll also want to install kubeseal on your local machine to create sealed secrets. You can download the latest binary for Linux and macOS from their releases page on GitHub or install it using Homebrew:

$ brew install kubeseal

Creating your first sealed secret

Creating a sealed secret is a two-step process:

1. Locally create a Kubernetes Opaque secret that you would like to seal (but don't push it to the cluster)
2. Run that secret through kubeseal to do the encryption, and output it to YAML so you can store it somewhere. (You could also output it to JSON, but meh.)

In Bash that process looks like so:

kubectl create secret generic \
  --dry-run \
  my-secret-name \
  --from-literal KEY1=VALUE1 \
  --from-literal KEY2=VALUE2 \
  -o json | kubeseal --format yaml

To make this a bit easier, I've written a shell script that lives on my path that I've named ezseal:

#!/bin/bash

set -e

kubectl create secret generic \
  --dry-run \
  "$@" \
  -o json | kubeseal --format yaml

This allows me to get the same result by running:

$ ezseal my-secret --from-literal KEY1=VALUE1 --from-literal KEY2=VALUE2

The resulting YAML can be pushed as a sealedsecret resource to your cluster. Once the controller picks up that the sealed secret is there, a corresponding secret resource will appear in your cluster with the decrypted values.

Tiller and Helm

Helm is a popular way to distribute applications that can be deployed to a Kubernetes cluster. Tiller is the cluster-side component that the helm command line tool requires. To use helm, you need Tiller.

One option for deploying Tiller is to deploy it on your local machine and configure it to talk to a remote Kubernetes cluster. This option is viable because Tiller persists everything it needs in the cluster itself. It's a perfectly fine option if you don't want to run Tiller remotely. I decided to run Tiller in my cluster for little other reason than I wanted to.

Note: The default Tiller configuration does not enforce TLS on Tiller's API endpoint. Any multi-user, production deployment of Kubernetes should take care to enable TLS on Tiller.

Helm's Documentation is pretty comprehensive. I won't waste my effort talking about how to install Tiller and the helm client. I will, however, share a few things I've done for configuring my helm client locally.

Setting up Tiller's Key Infrastructure

Helm's documentation walks you through setting up Tiller's TLS using the raw openssl commands to create a Certificate Authority and sign the certificate. Each user should have their own client certificate they use to authenticate to Tiller, so you can get into a situation where whoever administrates these keys does a lot of key signing.

You have a few options here to make this easier with out-of-the-box tools.

The first is to use mkcert. The mkcert utility was designed for local development certificates, but it also makes running a public key infrastructure for something like Tiller a lot easier because it can issue and sign certificates from a CA with a single command. If you're going to have mostly one person managing certificates for a Tiller install, you might want to consider this option.

The second is to look at deploying something like Vault to manage the PKI for Tiller and other services. The deployment of Vault is not a small undertaking. However, at a certain scale it is a preferable way to manage these certificates.

Defaulting Helm to TLS locally

The helm client does not default to using TLS. If you attempt to use helm against a Tiller setup that has TLS turned on without providing the --tls flag, you will get an opaque error message.

I fixed my forgetfulness by adding the following environment variable to my shell profile:

export HELM_TLS_ENABLE=true

Now my local helm client will default to using TLS when it connects to Tiller.

Traefik

Kubernetes cluster admins can choose to deploy an Ingress Controller that's capable of processing Kubernetes ingress definitions and routing incoming requests that match that definition to the proper backend. At the end of the day, an ingress controller is a reverse proxy with some wiring that permits it to update its configuration when new ingress definitions are added to Kubernetes.

My proxy of choice is Traefik. (Apparently, pronounced "traffic" because letters don't mean anything.)

A few things about Traefik make it my go-to solution for an edge proxy:

• The simplicity of its configuration is second to none.
• It has built-in support for Let's Encrypt.
• It has built-in support for integrating with Kubernetes.
• It also can be employed as a service mesh proxy
• It has a built-in dashboard and a slew of useful metrics

I used the official Helm chart to deploy Traefik to my cluster. The README there does a good job of laying out the various options for the deployment.

Conclusion

My cluster is a pretty simple one. A full deployment at a multi-person organization could be much, much more complex. But for one dude running a blog that occasionally gets some readers these cluster services work pretty well.

Until recently I was running my blog on Linode using Docker Swarm. This week I migrated over to Kubernetes on DigitalOcean.

After doing a comparison between running Kubernetes and Swarm a few years back, I decided in favor of the latter. Swarm is incredibly lightweight and, at the time, it was different enough from what I was doing day to day to be interesting. Further, I'm cheap. I could effectively run my blog, its database, and a few other apps on a $5/mo Linode instance. There weren't good options, besides GKE, for managed Kubernetes at the time, either. I didn't want to manage a Kube control plane.

Fast forward a few years: we're in a very different situation. When DigitalOcean announced their Kubernetes offering, I decided to take it out for a spin. I decided that, even though it's more expensive, there's more learning value in running my blog on Kube than there is in running it on Docker Swarm. Let's be honest, I'm not writing this blog because I expect to get famous. Running this site is all about learning value for me. This is part one of what I've learned so far.

Pricing Your Expectations

I don't use Amazon Web Services or Google Cloud for my personal projects. The utility pricing those providers have is great for micro-usage or companies that have the budget to write big checks. I, however, am always going to strongly favor predictability. Even though I am probably spending more on DigitalOcean than I would at Amazon or Google, it would take a lot to cause that dollar amount to increase.

I never expect to end up on the front page of Hacker News, but it's nice to know that if I do I don't have to be too concerned with what the impact on my wallet is going to be.

When configuring a Kubernetes cluster on DigitalOcean you configure what are called Node pools.

For my personal cluster I opted for the two of the Flexible Nodes. My selection was $15/month for 2vCPU and 2GB of memory for each node.

In addition to this, you pay a per GB cost for each Volume you create on DigitalOcean. When you deploy a Kubernetes cluster on DigitalOcean, this is all abstracted behind the Kubernetes concept of PersistentVolumeClaims.

When you create a new PersistentVolumeClaim in Kubernetes with the class do-block-storage - DigitalOcean will create a Volume on the backend. These volumes ensure that data outlives any individual node if that node gets destroyed. At $0.10 per GB reserved, my usage adds about $1 per month to my bill. (Note: You pay for GB reserved, not GB used. So if you reserve 10 GB, that's what you pay for.)

I also have a LoadBalancer service declared in Kubernetes for Traefik - the service responsible for routing incoming requests to the right backend running in the cluster. On DigitalOcean Kube, this creates a DigitalOcean Load Balancer, which costs $10/month. You could save this cost by using a Floating IP and host ports in Kubernetes, but Floating IPs don't automatically move when a node goes down.

Finally, I've subscribed to DigitalOcean Spaces for $5/month. I'm using Spaces to host image assets for the blog. Previously, image assets were loaded from the blog's server directly. Unfortunately, there was a performance hit to doing this. Images are many times larger than the text that you transfer from my server. Using Spaces I can offload that work to DigitalOcean and, bonus, I can utilize their Content Delivery Network (CDN) to get the images to load really quickly no matter how far away you are from my actual servers.

Totaling everything up:

• Compute Nodes — $30/month
• Load Balancer — $10/month
• Volumes — $1/month
• Spaces — $5/month

Total cost: $46/month.

That's a pretty sharp increase from the $5/month that I was paying for my Swarm node. But this is all about the learning experience for me.

I've been learning a lot more from running on Kubernetes than I learned from running on Docker Swarm.

I made the attempt this weekend to switch to FastMail from Google for my custom domain. It lasted less than an hour. Here's why.

By way of (brief) introduction, FastMail is a direct competitor to Google GSuite. It is great for folks who are privacy concious. Their plans are affordable, and their UI is slick. Included is an importer tool that will allow you to move all of your email from any IMAP-capable account (which includes Google) into your new FastMail account.

I came incredibly close to making the migration. I had MX records on my primary domain switched over, did some initial tests, and was beginning to switch over some of my secondary domains before I was stopped by an incoming email. This was an email that I wanted to save for later, but alas: I could not. Unlike Google Inbox and the "new" Gmail, I cannot snooze emails in FastMail and have them reappear in my inbox later on. My only options were to leave the email in my inbox or to send it to the archive. This single feature was my dealbreaker.

A few months back I attempted to switch away from Google's interface and toward a desktop email client. I tried to use Airmail and Apple Mail. I realized pretty quickly that snoozing emails and having them reliably reappear in my inbox was an essential workflow for me. Without this work flow a lot of things get missed when I inevitably declare inbox bankrupcy. Both Apple Mail and Airmail offer or can be extended to offer that feature, but they didn't work effectively unless the application was actively running on my machine. Given the frequency with which I would snooze something on my computer and need to see it on my phone later on, I couldn't tolerate that limitation.

Therefore, for the moment, I'm locked into the Google life. However, the limited time I spent tinkering with FastMail looked interesting. There are other apps that provide the functionality I'm looking for (e.g. Newton), but the idea of a service that proxies my email through their servers makes me a bit leery... it doesn't feel like that should be required to get the effect I'm looking for. Also, if I have to go through their servers to get an experience that makes sense, have I really gained much by not being in the Google ecosystem? (Especially given I'll be paying an annual fee for the pleasure?)

I got an itch a few days ago to try out an idea. This was, in part, motivated by a few conversations about Helm templates for Kubernetes. Helm Charts rely on weaving templating into your Kubernetes object definitions. The templating that Helm uses hails from the same tradition as ERB templates, mixed PHP/HTML pages, etc.

I started thinking on what it would be like if I could template Kubernetes definitions using something resembling Lift snippets - where the template and the instructions that lead to the final result are totally separated. And, since I'm working on getting a bit more familiar with Go - I decided to build it using Go. What I ended up with is Snipper.

Getting Snipper

I'll start by saying that Snipper is very much a proof of concept.

Snipper is available on GitHub. Pre-built binaries are available on the releases page, or you can get it using go install if you have a Go development environment set up.

Once it's installed you can use it by providing a template YAML file - which is just a regular old YAML file. You can then invoke snipper with the template and any number of transformers like so:

$ snipper template.yaml transformer1.yaml transformer2.yaml...

The resulting YAML will get sent to standard out.

Example

Let's say that I've got this Pod definition pulled mercilessly from the Kubernetes documentation.

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    app: myapp
spec:
  containers:
  - name: myapp-container
    image: busybox
    command:
    - 'sh'
    - '-c'
    - 'echo The app is running! && sleep 3600'

I can use snipper to modify this pod to my heart's content. Let's say that I would like to add a team label to the container. I might define this in the transformer YAML file like so:

'metadata:labels:team': Farmdawg Nation

Transformer YAML files follow some special conventions. Specifically:

• The top-level keys are all in "selector format"
  • : separates selectors. metadata selects the metadata node at the root. metadata:labels selects the labels under metadata, but would not select the labels under spec if there were one.
  • Ending a sector with + causes snipper to append a value to a string or a list instead of replacing the value
  • Using [] as a selector selects all array members at the level you're selecting at. spec:containers:[]:command:[] would select all components of the command array for all of the containers above.
• Selectors are only parsed at the top level of keys. Anything nested inside the selectors are assumed to be values that you would like to drop into your template.

These rules are still heavily in flux. They will likely change, but let's take a few more examples of things we can do.

We could suffix the pod name with the number 2:

'metadata:name+': 2

We could change the image to alpine:

'spec:containers:[]:image`: alpine

We could add an additional container:

'spec:containers+':
- name: myapp-hello
  image: hello-world

And of course, a single transformer file can have multiple rules:

'metadata:labels:team': Farmdawg Nation
'metadata:labels:alerting': enabled
'spec:containers+':
- name: myapp-hello
  image: hello-world

All of these instructions remain separate from the actual template itself. You no longer have to litter a single file with the template and what needs to happen to it. With Snipper, these concepts are separate.

But what about logic?

You'll notice that there aren't any logic constructs. That's actually an intentional decision. Snipper by itself isn't a total replacement for a templating engine. However, when combined with an orchestration layer that conditionally applies transforms, you wouldn't be far off. I have not (yet) written such an orchestration layer, but I could imagine a tool that defines, say, a:

• pod_template.yaml
• local.yaml (transforms for local deployment)
• staging.yaml (transforms for staging cluster deployment)
• prod.yaml (transforms for production cluster deployment)

What's next?

I'm not sure. Part of this was scratching an itch. Part of it was wanting to develop some better informed opinions about Go. What I do next with this project will depend heavily on whether or not folks find it useful / interesting. So, if you find it useful / interesting please star it on GitHub, file bugs and feature requests, or share with your friends.

In December of 2010 I wrote a piece titled "What Net Neutrality Means For You" shortly after the FCC took its initial action on Net Neutrality. In light of the recent FCC ruling to scrap the subsequent 2015 rule that permitted the FCC to enforce Net Neutrality principles, I thought it would be fitting to revisit that topic with some thoughts from this past week.

About discouraging investment

On Thursday the FCC voted to roll back the rules that were enacted under the Obama Administration. The Associated Press summarizes what the expected impact will be in the short and long term in this article. Additionally, the AP outlines the central objection of the telecommunications companies:

The big telecommunications companies had lobbied hard to overturn the rules, contending they are heavy-handed and discourage investment in broadband networks.

Wired poked some holes in this argument. The study cited in the FCC proposal to roll back the rules was industry funded, and asserted that there's a correlation of decreased spending after the rules were enacted. Setting aside the post hoc ergo propter hoc nature of that argument, Wired illustrated that the reality is a mixed bag at best. For example:

• Comcast made statements to their shareholders that the Title II classification was not negatively impacting their business, but they did have to delay a streaming service (sounds like that rubbed them the wrong way)
• AT&T has been working to lean on software for improved performance, decreasing their hardware investment accordingly
• Sprint has been cutting spending since before the rules were passed, but the then-CTO admitted that the Title II classification wouldn't impact them
• Charter has consistently been increasing spending since 2014 to increase speed

... and the list goes on.

From my vantage point I've actually seen a surprising amount of investment in infrastructure in the Atlanta area. Comcast and AT&T have both launched residential gigabit services within months of Google Fiber announcing their intent to move in. Comcast has also increased data caps for residential customers over the last year or two without me noticing.

It sure doesn't feel like investment has been hindered since 2015.

About that light-touch approach

Ajit Pai, the FCC chairman, seems quite proud of his work. From the AP article linked above:

"What is the FCC doing today?" asked FCC chairman Ajit Pai, a Republican. "Quite simply, we are restoring the light-touch framework that has governed the internet for most of its existence."

Let's briefly discuss some other things that have been true for most of the internet's existence.

Pai conveniently neglects to mention here that for most of the internet's existence there has been a strong delineation between content providers and internet service providers. He also doesn't mention that for most of the internet's existence it's been mostly irrelevant in Presidential elections. He doesn't mention that for most of the internet's existence citizens have not used it to access social or government services, educate themselves, or exercise their First Amendment rights.

Did you know that in 2014 the country of Estonia launched an E-Residency program? While primarily designed for businesses who wish to have an EU presence, the program in Estonia is certainly a forbearer of what is to come as governments look for ways to become more accessible and efficient.

The internet has become very much like the streets through which we travel to do the important business of life. All signs indicate that it will become even more so in the years to come. This has not been true for most of the internet's existence. As that happens, having good rules for how the companies that own the pavement can behave is ultimately required for a successful society to continue to build on top of it.

Chairman Pai is either laughably ignorant that the reality around him has changed, or he's willingly ignoring it for political convenience. Either option should disqualify him from having a voice in the matter.

This way be dragons

Now, as in 2010, there are very good reasons for the FCC, or some other entity, to monitor and referee how privately-held monopolies riddled with conflicts of interest steward the foundation of our digital society.

Though internet providers have claimed they are not interested in blocking the services we value, the AP rightly notes:

But such things have happened before. The Associated Press in 2007 found Comcast was blocking some file-sharing services. AT&T blocked Skype and other internet calling services — which competed with its voice-call business — from the iPhone until 2009.

The astute reader would rightly observe that there's a razor thin difference between censoring a competing service and censoring a competing opinion.

A path forward

There are proven strategies for managing the ever-increasing network demand that an ever-increasing population with an ever-increasing variety of content will produce. There are proven strategies for dealing with monopolies that misbehave. There are proven strategies for dealing with conflicts of interest.

We need only use them.

It's being widely reported this weekend that a former engineer involved in the Volkswagen diesel scandal, James Liang, has been sentenced to a 40-month prison term. I've seen a bit of shock and outrage on Twitter over this, including a quickness to conclude that Liang was sent to prison for just doing something his boss told him to do. It seemed to me that there are some out there that have bought into the school of thought that if you're ordered to do something, your butt should be totally covered. As a software engineer, this is a belief I would like to speak against and dispel for the good of myself and my craft.

Background

If you've been living under a rock, or just aren't quite sure what the emissions scandal was about, this summary from the Car and Driver Blog¹ does a good job of summarizing what happened:

Volkswagen installed emissions software on more than a half-million diesel cars in the U.S.—and roughly 10.5 million more worldwide—that allows them to sense the unique parameters of an emissions drive cycle set by the Environmental Protection Agency. [...] [T]hese so-called “defeat devices” detect steering, throttle, and other inputs used in the test to switch between two distinct operating modes.

In the test mode, the cars are fully compliant with all federal emissions levels. But when driving normally, the computer switches to a separate mode—significantly changing the fuel pressure, injection timing, exhaust-gas recirculation, and, in models with AdBlue, the amount of urea fluid sprayed into the exhaust. While this mode likely delivers higher mileage and power, it also permits heavier nitrogen-oxide emissions (NOx)—a smog-forming pollutant linked to lung cancer—up to 40 times higher than the federal limit.

To elaborate slightly: This means there was a group of people at Volkswagen that installed software specifically designed to lie to the test devices. The result of this lie is that there were (and probably still are) cars on the road that emit an excess of a chemical linked to lung cancer.

The one qualm that I have with Car and Driver's depiction is their use of the word "installed." Please excuse me if I'm giving the average reader of my blog too little credit, but I want to make it clear to everyone that installing this kind of software isn't anywhere near the same thing as installing Microsoft Office on your PC. You'd be hard pressed to find pre-written emissions defeat software anywhere online, much less software that just happens to work with the new car engine you're designing.

I actually searched for "emissions defeat software" while writing this blog post. The results are best summarized by this screen shot from GitHub's project search screen:

Why does this matter?

To install this software in their cars, it means that Volkswagen had to design the software first. Liang's own admissions² corroborate this (empahsis mine):

According to Liang’s admissions, when he and his co-conspirators realized that they could not design a diesel engine that would meet the stricter U.S. emissions standards, they designed and implemented software to recognize whether a vehicle was undergoing standard U.S. emissions testing on a dynamometer or being driven on the road under normal driving conditions (the defeat device), in order to cheat the emissions tests.

The design element of this scandal is the element I find most troubling. It means that there was an ongoing, intentional, directed effort to subvert U.S. law by the leadership and engineering teams at VW. There were likely planning meetings, task tracking of some sort,³ deadlines, and concrete deliverables that these teams agreed on in advance. Throughout this process, there was no voice that prevailed that said "Hey, perhaps this isn't okay."

Just an "Engineer"

The piece that I've seen circulating the most is a piece from Reuters that characterizes Mr. Liang as a "former engineer."⁴ The reaction I saw is nicely summarized by the Tweet below:

Every. Single. Software developer. Must. Take. Note.

YOU can go to jail for the code YOUR BOSS tells you to write. https://t.co/3MZGHqI4dy

— Ted Neward (@tedneward) August 25, 2017

I saw a few retweets and other takes that had about the same flavor to them. This will be news to nobody, but "Oh man, that could be me" is a visceral fear that's hard to combat. If I had to guess, I'd say this fear is motivating many of the reactions I've personally seen.

However, based on my reading of the Reuters article, the relevant Justice Department press release, and some well-educated guesses informed by my professional experience, I think the tweet above is largely a false equivalence. It paints the picture that Liang was convicted for just writing some code.

It's unlikely that every single software engineer at Volkswagen had the level of involvement it seems Liang exercised over Volkswagen's diesel (non-)compliance. Indeed, the Justice Department release from 2016² states: "Liang admitted that during some of these meetings [with regulators], which he personally attended, his co-conspirators misrepresented that VW diesel vehicles complied with U.S. emissions standards and hid the existence of the defeat device from U.S. regulators." Liang here is directly complicit in misleading U.S. regulators. At the risk of over-generalizing my own experience, I'll state that folks who are front-line engineers are rarely in the room with regulators.

Yet, I want to set aside that false equivalence and discuss something more important. Even if Liang himself is not a member of the population Ted Neward's tweet above appeals to, the front-line software engineers, do we really believe they should be totally free from responsibility in this scandal?

The Ethics of Engineering

I, like many of my contemporaries, strongly prefer the job title of "Software Engineer" to describe myself over the practically interchangeable term "Software Developer." I don't see my job as developing software. My job is to solve problems, and I will most often use or develop software to do so.⁵

I cannot know what happened in the back rooms at VW. Perhaps ethical concerns were raised. Perhaps jobs were threated by management. Perhaps every front-line engineer involved had defendable reasons why they couldn't resign their position on ethical grounds and ultimately helped create the deception that lead to the sentencing of Mr. Liang and others.

I would not fault someone for doing the wrong thing for the right reasons. I likewise do not think that the front-line software engineers at VW should be held legally responsible in this specific case. In spite of that, however, I doubt the ethical intelligence of anyone who would suggest, without further exonerating evidence, that those individuals bear no moral responsibility for the deception and health risk they helped create.

In our roles as problem solvers, whether or not someone is paying us for the work we're doing, we are ultimately responsible for whether or not our solution to the problem at hand will make the world a better place. This conviction is what drives the conscience of the ethical engineer - software or otherwise - and it plays out in our decisions big and small.

This conviction is simultaniously burdensome and empowering.

It is burdensome when we're at odds with whatever organization we're currently employed by and we're asked to do things that go against our ethical code. It is burdensome when we're told doing the right thing is too expensive and we should just do the cheap thing and be happy. It is burdensome when our peers or bosses reply, "But everyone else is doing X, so you should just get over your objections and implement X!" Or, worse still, when those in power threaten to fire us if we do not captiulate.

Yet even in burdensome organizations, this conscience is empowering when it motivates us to do even the smallest things to make the world better. It empowers us to speak up in the meeting and say, "I disagree with this." It empowers us to work to ensure that the new app we're building is accessible to someone using a screen reader. It empowers us to help customer support execute more informed, coordinated responses when things go wrong. It empowers us to go to the extra effort of making our code easier to understand for the next engineer who comes along. It empowers us to work to protect end-user user information of our own accord instead of waiting for a security event to force our hand.

We all have the choice of whether or not to live in line with this conscience as we go about our problem solving. I have good days and bad when it comes to listening to my own convictions about the right thing to do. Even still, the truth is that we're uniquely empowered to make the world a better place every day, by a thousand paper cuts if we're so forced. On those occasions when we're lucky to work with an organization that backs ethical efforts, the possibilities for improving the world we're in are endless.

But this is not the work of those happy to punch a clock and write off ownership of what they've done. We have to first take up our mantle of responsibility as engineers, or else we will squander our unique, individual opportunity to make the world better by ethically doing what we do well.

Footnotes

1. You can find that blog post here here.
2. Justice Department Press Release, 9/2016
3. I find myself idly wondering if they were using JIRA, though I kind of doubt it. It would just be really amusing to me if they were because it seems like everyone uses JIRA. I also find myself idly wondering if the people who could reason their way to committing fraud like this may understand JIRA better than I do. Really, I just find myself confused at JIRA a lot these days and I wanted to vent about that a little.
4. Reuters article here
5. When I'm really lucky I'll solve problems by deleting software. Those are good days.

I've been giving a lot of thought to online communities as of late. I've been a bit discouraged at some behavior I've seen in the Scala ecosystem recently. Enough so that I felt the need to subtweet about it.

I’m always really disappointed when really prominent folks in the Scala community are mean to people.

— Matt Farmer (@farmdawgnation) June 29, 2017

I truly believe that open source software is an essential, important piece of maintaining innovation velocity in the technology industry. I've advocated heavily for this idea everywhere I've worked, and even written about it for prior employers on occasion. I volunteer my time to help maintain open source software that I've benefited from in the past. And yet, as much as I love and support open source development, I have to admit that the biggest risk to the continued health of the open source ecosystem is the behavior of the people involved in open source development.

Nothing that I witnessed on these most recent occasions was particularly egregious, but after reflecting on my frustrations for a few weeks I wanted to share some thoughts for constructive community interactions within technical communities. The individuals who triggered these thoughts will probably never read them, and probably wouldn't care if they did. But perhaps someone else will find them useful. If nothing else, writing them down should be cathartic.

Without further ado:

1. Assume you're not the smartest person in the room. Odds are, you probably aren't.
2. Assume the other person means well and has put thought into their point of view. Being belittling, especially when people are volunteering their free time to participate, ultimately discourages people from participating in your community or using your software. Over time this kills open source projects.
3. Disagreement isn't the end of the world. Agreeing to disagree is totally fine. Your open source contributions aren't life or death. Don't be overly dramatic when something happens that you disagree with, because it'll probably be fine. This, too, kills projects over time.
4. Be patient with new community members. New community members that don't understand what is going on are a good sign. It means your project is gaining visibility and getting new blood.
5. Decide what's accepted behavior and what's not, and enforce that norm. GitHub has rightly started allowing maintainers to add codes of conduct to their projects. Create one. And enforce it. Lift has been doing this for a while (though we haven't yet taken advantage of the new GitHub feature). I plan to do the same for my other major projects.

TL;DR: Just be nice, and encourage other participants in your community to do the same. At the end of the day you can have the most well thought out, perfect software out there. Yet it won't matter if nobody uses it because they don't want to deal with you.

End of song.

As I write this I'm burning off the last of the data I purchased from Karma earlier this year. For reasons that are understandable they've decided to discontinue their Refuel plan and, instead, favor plans that charge a monthly fee or a monthly fee plus the cost of data used.

While I understand that Karma needs to stop paying for customers that are never using their devices, it still seems like the underlying problem with their data arrangement is the "Never Expires" bit. I'm confused as to why the answer to this problem wasn't to change the "Never Expires" policy and leave the rest of the structure in place.

Perhaps I am a strange customer, but I would be willing to pay for data that expires in a certain amount of time than to accept a $3 monthly fee plus automatically incurring $10 per GB as I use data. The fact that the total cost of my data expenditure was known up front without billing me monthly was the killer feature of the Karma platform for me.

Under the current offerings there isn't much that distinguishes Karma from the data service I can get through a cell provider. Furthermore, the resurgence of "Unlimited" data plans on many of the major cell networks and new offerings like Verizon's Pop Data make a Karma subscription even harder to justify. If I were to use Pop Data, I can work for an entire day on my LTE connection for $24, likely have better coverage, and still have my standard LTE data pool to fall back on afterward. If I were to use Karma's Drift program, I could easily end up using 3GB (or more) over the course of the day (Docker images can be large) and spend $30+ after the monthly subscription fee.

I wish Karma well, but I'm not sure they've made the right move here. As a small company in the ocean with big fish like Verizon, Sprint, AT&T, and T-Mobile, you've got to distinguish yourself to win out. Karma's old pricing structure put the service in a class of its own while the new structure just seems like a small spin on the same thing every other company in the space is doing. Furthermore, no matter what plans Karma offers, they're never really going to replace any of the cell providers. I'll still need to buy data from them for my phone. Karma is always an extra on top of my cell plan whereas if I need to only use a few hundred MB to do a quick errand I can easily slot that into my existing data allowance with tethering.

Perhaps they'll still manage to find a productive, profitable niche with these new plan structures. However, based on what I know today that niche won't include me.

Image above is used under the terms of CC Attribution 2.0 Generic. Source is here.

JSON is everywhere. That means that properly supporting all of JSON's flexibility is an important priority for any web framework in 2017. For a typed language, like Scala, this type of problem can get interesting because JSON can do a few things that aren't quite as intuitive to do in an actual Scala application. One of those is supporting collections with many diverse types.

In JSON, like many other data formats, you can have arrays that help store collections of information. Such as a list of my favorite dishes to eat:

["spaghetti", "pizza", "pb&j", "pimento cheese sandwich"]

As you can tell, I make sure that all of the major food groups are covered in my diet. I could also create a tabular data structure and describe a health rating of 0-10 of each of these foods:

[
  ["spaghetti", 6],
  ["pizza", 4],
  ["pb&j", 7],
  ["pimento cheese sandwich", 8]
]

This is a totally valid JSON data structure. The inner data structure here is called a heterogenous array because the types of the different positions are different. However, it's one that's a bit challenging to represent in Scala without additional libraries. Unless you use a Tuple.

The data structure above can easily be modeled in Scala using a Seq[(String, Int)], but until recently Lift-Json couldn't interpret this data structure without a custom serializer and deserializer defined by the developer.

We recently merged a pull request that changes that by implementing a feature we're calling Array Tuple Extraction. This feature will ship with Lift 3.1.0-M3 and will be disabled by default for backwards-compatibility reasons. This blog post will walk through what has changed, the limitations of our new Tuple extraction, and how to use it.

Current Tuple Support

Lift-Json has always supported tuples in one form or another, but it has not, until now, supported heterogenous arrays. In Lift 3.1.0-M2 and earlier, a Seq[(String, Int)] would be decomposed and serialized into the following JSON data structure:

[
  {"_1": "spaghetti", "_2": 6},
  {"_1": "pizza", "_2": 4},
  {"_1": "pb&j", "_2": 7},
  {"_1": "pimento cheese sandwich", "_2": 8}
]

If those _1 and _2 members in the object look familiar to you, it's because they come from the constructor for the [Tuple2](http://www.scala-lang.org/api/current/scala/Tuple2.html#(_1:T1,_2:T2):(T1,T2)) class that Scala would use to model this at runtime.

One of Lift-Json's strengths is that its method of looking at the declaration of a class when figuring out how to serialize it means it can do a lot of things automatically without intervention. The above data structure will correctly serialize and deserialize through Lift-Json with no extra work from the developer using the library.

Unfortunately, this isn't super semantically correct with regards to how JSON thinks about arrays. Many REST JSON APIs require you to submit a heterogenous array as a part of a payload for certain requests. Until recently, this is a need that Lift-Json users would need to manually craft the AST representation of the data structure or write a custom serializer/deserializer for this data.

Array Tuple Extraction

The new Array Tuple Extraction functionality allows developers using Lift-Json to correctly represent Heterogenous Arrays as mixed-type Tuples in Scala without any additional serializer/deserializer work.

As with many changes you might want to make to Lift-Json's extractor, all you need to do is provide a Formats instance as an implicit val that turns the feature on. You can do this, for example, by declaring the following at the top of a class:

val formats = new DefaultFormats {
  override val tuplesAsArrays = true
}

This will activate the new tuple extractor and decomposer that will represent a Tuple of (Thing1, Thing2) as a JSON array of [Thing1, Thing2].

There are, however, a few caveats with this functionality:

• Scala primitives don't yet work reliably. If you're going to serialize/deserialize Tuples with your application using the array extractor you should ensure you're using Java's boxed types (java.lang.Integer, java.lang.Boolean, etc). This has something to do with the limitations of JVM reflection on Scala primitive types, but we haven't determined the best solution.
• Back-compat support is provided, but not two-way. The array extractor can correctly interpret Tuples that were serialized in the old format, but if you want to write Tuples in that format again, you'll need to change your Formats to disable the feature first.
• The implementation is a bit... dodgy. The Lift-Json extraction code is the code that takes actual, factual Scala datatypes and turns them into an AST for translation into JSON. This is the point at which we start handling Tuples differently. Unfortunately, the extractor is quite old at this point and challenging to reason about. I've started a speculative rewrite of the extractor using Scala runtime reflection, but don't expect to see feature parity with the main extractor for awhile.

Final Notes

My hope is that this feature turns out to be really useful to folks using the Lift Framework and Lift-Json to push JSON over a wire. If you have any questions, find some bugs, or just want to tell us what you think: drop us a line on the Lift Mailing List.

Last week was my first week at MailChimp. I want to share a few specific, unorganized thoughts coming out of that week about my experience as a first-week employee of the company.

• The on boarding experience at MailChimp was intentionally thought out. This is a big difference from my past positions where on boarding was more ad-hoc. Historically, I've felt like a burden as a new employee. MailChimp's approach goes a long way toward making me feel valued from Day 1, which is a much better emotional foot to start off on.

• Humility is one of the core values of the company, and it shows in my interactions with the coworkers I met last week. Throughout the week members of every department sat with the new hires and answered our questions about their share of what we do. They spent time actually getting to know us and asking what departments we were in and what we'd be doing. Furthermore, that includes IT and legal, departments that in my past experiences tended to be a bit more aloof from the rest of the company.

• Intentional discussion about how we each communicate is a part of the orientation process. MailChimp uses Birkman assessments to get a sense of how each of the new hires communicates. The company also does regular sessions with teams to help ensure that people are communicating effectively.

• I didn't have any time in my first week to write code. This isn't a one size fits all thing — MailChimp is larger than most other organizations I've worked with, but after experiencing the intentional learning time and discussion up front, I think I'm going to be less of an advocate for "first pull request on the first day" than I have been in the past. A week long series of sessions certainly wouldn't make sense for a small start up, but in reflecting on my past experiences I think that my first few months at my previous jobs would have been less anxiety filled with some intentional time to listen and learn.

• This is the first company I've worked for that handles security so well. I won't go into details for Obvious Reasons™ — but the security policies established are sensible for our threat level without being overbearing. IT has also done a great job of making security accessible to the less technical members of the organization. I don't fall into that category, obviously, but I still noticed how smooth the process was and how well written the documentation is. Should both of those fail, the IT folks are also super easy to reach and willing to answer questions.

• This is the most diverse company I've ever worked for. I'm not qualified to speak on how we got to the point we're at now, but technology companies have a really lousy track record with diversity and I'm really proud to work for one that is taking it seriously. A pastor friend of mine once shared this quote with me: "The best criticism of the bad is the practice of the better." I can't recall the original author, but it rings true in this situation.

TL;DR: I'm really excited for week two, and that's a fantastic feeling. It's also been about three weeks since I've touched any code in any significant capacity — so I'm starting to get that old itch to fire up my editor again and get into the weeds on something challenging.

I've been thinking a lot about technical debt lately. It's something that every codebase has, but also something that we still fail to fully consider the cost of. As such, I thought it would be appropriate to walk through some of the ways technical debt has affected my life in the last 24 hours.

VM File sharing

This is something that always comes with a lot of promises, but never seems to deliver. And, to boot, when it fails it always fails in really interesting ways.

Today, the file sharing in Parallels failed me. At Domino we recently added LevelDB as a dependency. I've been largely working on another project so it had been a minute since I had compiled the main application and tried to get it working. Things didn't go so well.

It seems that if you attempt to use LevelDB over a Parallels file share you'll get an error that things can't be set up correctly. All of that will boil down to:

IOException: Invalid argument

Let me count the ways in which this is a frustrating condition to find yourself in:

1. What the hell kind of error message is "Invalid argument."
2. What the hell kind of error message is "Invalid argument."
3. VMW file systems are typically supposed to be a substitute, and ideally something that performs better than, some network file share like NFS or SMB. In all cases that I tried, except Parallels, they have abysmal performance (but that's probably an entirely different blog post).
4. What the hell kind of error message is "Invalid argument."

X11 / XQuartz / Remoting

After this experience I decided a more conventional Linux experience might serve me better. As a result, today I tried (yet again) to get X11 remoting working. I had tried this on a company issued Lenovo earlier. After failing to use Linux as a primary OS I reverted to using Windows - because my experience with Linux reached a point where that seemed like a more reasonable option. (Perhaps that's a different blog post.)

Anyway, we arrive at today where I attempt to redo the same experience on my Mac box using XQuartz. I got XQuartz installed, I got Atom installed on my VM, and then I ssh'd into the Ubuntu VM from the XQuartz session and started Atom.

Performance was abysmal, but not intolerable. So I decided to give typing in a new document a try.

This is a st ofybord

Hm. That doesn't seem right. If you're looking at this and you think "Whoa the 'e' key got mapped to backspace" then you're a cleverer person than I, because it took me a little bit to figure out what was going on.

After much Googling, tinkering around inspecting X events, trying to figure out ways to work around this, attempting to connect to the X server without a wrapping session, I then proceeded to give up.

I don't quite know what the underlying cause of the technical debt here is. I don't know if it's in X11, ssh, XQuartz, or something else. It's not something that happens in the xterm terminal, so I'm inclined to think that it's some weird X11 interaction with Atom.

I don't quite know how I would classify this technical debt. However, I feel confident in attributing it as technical debt in spite of my lack of knowledge with X11. (Or maybe perhaps because of what I do know of X11?)

Installing kubuntu on Parallels

After the above failed, I attempted to install Kubuntu in a Parallels VM and start emulating an entire Linux environment with KDE as my preferred alternative to the mainline Ubuntu Unity.

The installer crashed before it even got started. So, there's that.

Conclusion

My day has the distinct flavor of a lot of shit that should just not be possible. It has the distinct flavor, in my mind, of technical debt.

So, next time you're trying to figure out how to weigh trade offs in your project please imagine that your project will be wildly successful. And then imagine that there's going to be someone like me. Staring at his screen. Asking his software to just please finally work.

I do not love transitive dependencies. However, they seem almost unavoidable. Consider HTTP libraries. The odds are that if you're writing software in 2016, there's a decent chance you're writing software that will want to talk to some other software on the Internet. Furthermore, there's also a decent chance you'll use HTTP to talk to that other software.

The good news here is that you don't have to keep re-inventing the wheel. You have plenty of reusable libraries to choose from in order to accomplish this communication. Pull them in to your Super Cool Open Source Project™, and your project then can speak HTTP with relatively little effort from you! Being able to re-use code in this manner is one of the major strengths of today's software world. It's also a significant weakness.

I've spent some of my free time lately developing a project I've named Giterrific. As a part of that project I've had to wrestle with some of the problems inherent in transitive dependencies and, in the process, have developed a solution I'm calling the Pluggable Dependency Pattern that I believe more open source projects should consider adopting. This blog post will outline the issues we ran into attempting to integrate Giterrific into a proof of concept project at Domino, my employer, and how I used this pattern to overcome those issues.

This pattern may not in fact be new to you. I've seen it appear sporadically in other projects. Yet in implementing it myself I learned some useful guidelines for integrating it into my projects, and became convinced this is something other projects should be discussing.

Transitivity

To set up the story that I'm about to tell you, it's helpful to ensure we all start with an understanding of transitive dependencies.

A transitive dependency is a dependency that exists because of transitivity. If you're writing project A, and project A depends on project B. In turn, project B depends on project C. Therefore, C is a transitive dependency of A.

If this sounds fimiliar to transitivity from a Mathematics class, then you're thinking of the right thing. It works the same way!

Dumpster Fires, the Mother of Invention

Giterrific aims to do one thing well: expose information about private Git repositories in a simple, straightforward JSON API. Further, it also aims to make that easy to integrate with your existing code. To that end, I've provided a library named giterrific-client that provides Scala API bindings for calling out to the Giterrific server.

The abstraction here is pretty simple. You make a call to a function I've defined that is designed to retrieve commit information for a particular git branch. Behind the scenes, giterrific-client will make an HTTP request to your installation of Giterrific and return the result. To accomplish this, I used my favorite HTTP abstraction: Databinder Dispatch. This was a simple 48 hour project over a weekend. The following week I handed this off to a coworker who I thought could find use of this at Domino.

Can you guess what happened?

If you guessed dumpster fire, then you guessed correctly.

At Domino, we're using the Play Framework for a lot of our development work - much like a lot of other Scala shops. The Play Framework version we're using depends on a library called "Async HTTP Client." It's a Java library that makes speaking HTTP easier.

For giterrific-client, I used Databinder Dispatch to help me speak HTTP easier. Dispatch in turn uses Async HTTP Client, also. The problem: Dispatch uses a different version of Async HTTP Client than Play. Moreover, it uses a totally incompatible version. This meant that with the way I'd architected giterrific-client, there's no way that any relatively recent Play application could use it.

This is why transitive dependencies are so obnoxious. It's really trivial on any project to create a situation where you have conflicting dependencies. To explain another way consider the following example:

• Your project A1 depends on B1 and C1.
• B1 depends on X1.
• C1 depends on X2.
• A1 therefore depends on X1 and X2.
• X1 and X2 will attempt to occupy the exact same space, and conflict. As a result either B1 or C1 will not behave as they were intended to.

So what do we do to fix this problem? Well, typically in this scenario project A1 would get no say in what B1 or C1 pull in transitively. However, if it could, it could easliy tell B1 "Hey, I want you to use X2 instead of X1 - and here's how you talk to it."

Enter HttpDriver

To resolve the issue, I tweaked the architecture of Giterrific to use what I'm calling The Pluggable Dependency Pattern.

I created an interface I've named HttpDriver. Previously, Giterrific's client code would talk directly to Dispatch when it wanted to make an HTTP request. Now, it'll talk to an HttpDriver. By default, this will still talk to Dispatch under the hood, but it can be configured to talk to whatever HTTP library you'd prefer to use.

For Giterrific 0.2.0, I ship four different pre-built drivers: the default Dispatch driver, a driver for Play 2.4, a driver for Play 2.5, and a driver for Finagle HTTP. If one of them doesn't work for you, it's quite trivial to swap out which driver is being used or implement your own HttpDriver from scratch. Since I've done this, we've successfully been able to implement a Giterrific POC in our Play application at Domino with minimal fuss.

The caveat here is that the underlying HTTP libraries are no longer declared as transitive dependencies. The project that wants to use Giterrific has to pull in Dispatch, Finagle, or Play themselves.

From Transitivity to Pluggability

Surely, making everything pluggable on every open source project would become very annoying. Transitive dependencies are a huge convenience until they blow up in some unexpected way. Arguably, this is an avoidable problem by using a microservice architecture instead of a monolith architecture. However, I think in spite of that there's space for reasonable people to disagree when it comes to "the best fit" for a certain underlying service.

Some folks make heavy use of some core Finagle features throughout their architecture, for example. If they get a benefit from that and want to use Giterrific, why should my personal preference for Databinder Dispatch handicap them?

Pluggability, I think, has a few specific cases that it will work very well for. Specifically, cases where:

1. The task you're trying to accomplish is fairly common with a lot of angles.
2. The task you're trying to accomplish is sufficiently complex to warrant concern over what transitive dependencies you'll pull in.

There are two examples of areas ripe for pluggability. Specifically, HTTP interaction and JSON serailization/deserialization. Both of these tend to be pretty common tasks with different angles to them (which socket library gets used, which reflection method does the serialization algorithm use) and both can require some complex dependencies in their own right to accomplish their end goal.

Furthermore, I think that it's important that the project which wishes to be pluggable should have its abstraction for pluggability self-contained. So, for example, I think it's unlikely (based on my current thinking) that I'll ever make HttpDriver its own library. Doing so would just move the transitive dependency risk to a different link in the chain. Therefore, each project I implement this pattern in shall have it's own HttpDriver that will define the minimum interface that project needs in order to speak HTTP.

Final Thoughts

This was a useful pattern for me to fix a real probem I ran into. As always, your mileage may vary. Either way, I would love to hear your thoughts on this. Reach out to me on Twitter at @farmdawgnation and let me know your thoughts. I'll update this post with any interesting exchanges that result.