Source Patch

Using Let’s Encrypt certificates on managed clusters with OpenShift Hive

2022-12-06T08:39:00.007-08:00

(You can see the source code behind this article in this GitHub repository)

Using Let’s Encrypt certificates on managed clusters with OpenShift Hive

Overview

OpenShift Hive is a Kubernetes cluster management platform for provisioning and configuring Kubernetes clusters at scale.

As a management platform, it can create new OpenShift clusters across all major Cloud providers and perform limited management operations on clusters it creates, such as hibernating clusters that are temporarily not in use.

OpenShift Hive creates and manages Kubernetes clusters.

Note: I use OpenShift Hive through Red Hat Advanced Cluster Management for Kubernetes. I tried to generalize the writing and instructions so that they work when running OpenShift Hive in isolation, but have not validated the instructions using that environment.

Signed API endpoint certificates

In a production environment, cluster clients need to trust that they are connecting to the actual API endpoint for the cluster and not to a system impersonating the cluster.

The straightforward way of enforcing that requirement is to sign the OpenShift API endpoint with a certificate authority that clients equally trust.

The challenge for using trusted certificates on managed clusters is two-fold:

Currently, OpenShift Hive does not expose APIs to sign (or apply signed) certificates for managed clusters
OpenShift Hibe is also a client to the managed clusters it creates, as it continuously interacts with the clusters to assess their health and perform management functions (such as the hibernation task mentioned earlier.)

Once you sign the API endpoint for the managed cluster, OpenShift Hive reports internal errors about not trusting that cluster’s API endpoint, preventing it from managing the cluster.

OpenShift Hive cannot initially trust certificates added to existing managed clusters.

The solution (credit to Andrew Butcher) is to inform OpenShift Hive about the signing authority for the managed cluster, which can take place in two different ways:

If the signing authority is shared across all (or many clusters,) modify the CA (certificate authority) database for all clusters in the Hive configuration.
Modify the CA database for the individual managed cluster.

This article explores that second approach since there is no guarantee that a single Let’s Encrypt intermediary signer is the same across multiple clusters. If you want to explore that configuration for other purposes (for example, if your organization owns the signing certificate,) then you must use the set-additional-ca.sh script to replace the current CA database (if any) with a new one.

Signing the API endpoint

The first step is to acquire the certificate using one of the various ACME clients.

For this article, I have a self-managed OpenShift cluster created in an AWS account so that we can use the ACME shell script client.

The following snippet assumes you have the AWS CLI credentials in place.

Note: The examples use Shell scripting and rely on the :? construct often to ensure that variables next to that symbol are actually defined. For example, echo ${a:?} fails if the variable named a is not defined or empty.

    # Assuming AWS CLI logged in with an id authorized to interact 
# with AWS's Route 53 service

cluster_domain=# type the name of the cluster domain here, such as mycluster.mylabdomain.com

cluster_api=$(oc get Infrastructure cluster -o jsonpath='{.status.apiServerURL}')

# These two lines remove the preceding "https://" and 
# the ":port_nnumber" suffix from the URL
cluster_api="${cluster_api//*:\/\//}"
cluster_api="${cluster_api//\:*/}"

cert_dir=/tmp/certificates
mkdir -p "${cert_dir}"

git clone https://github.com/acmesh-official/acme.sh.git -b 3.0.0 --depth 1
cd acme.sh

# Note that the technique below was chosen for the simplicity of the example
# since the purpose of this tutorial is to explain how to interact 
# with the resulting cluster. It connects directly to AWS Route 53 
# with the provided credentials, so you should study the CLI 
# documentation for more secure alternatives.
./acme.sh \
    --issue \
    -d "*.apps.${cluster_domain:?}" \
    -d "api.${cluster_domain}" \
    --server letsencrypt \
    --dns dns_aws \
    --force

./acme.sh \
    --install-cert \
    -d "*.apps.${cluster_domain}" \
    -d "api.${cluster_domain}" \
    --cert-file ${cert_dir}/cert.pem \
    --key-file ${cert_dir}/key.pem \
    --fullchain-file ${cert_dir}/fullchain.pem \
    --ca-file ${cert_dir}/ca.cer

  

At this point, we have the following files:

Certificate	File
Server certificate	${cert_dir}/cert.pem
Key for the server certificate	${cert_dir}/key.pem
Full chain for the certificate authority	${cert_dir}/fullchain.pem
Signing certificate for the server certificate (this is the most important file in the subsequent sections))	${cert_dir}/ca.cer

Now execute the OpenShift instructions to apply the new certificate to the managed cluster. You should be logged in as a cluster administrator in the managed cluster.

    # https://docs.openshift.com/container-platform/4.11/security/certificates/api-server.html

oc delete secret api-certs \
    --ignore-not-found=true \
    -n openshift-config

oc create secret tls api-certs \
    --cert="${cert_dir}/fullchain.pem" \
    --key="${cert_dir}/key.pem" \
    -n openshift-config

oc patch apiserver cluster \
      --type merge \
      --patch="{\"spec\": {\"servingCerts\": {\"namedCertificates\": [ { \"names\": [  \"${cluster_api:?}\"  ], \"servingCertificate\": {\"name\": \"api-certs\" }}]}}}" 


  

The cluster may take several minutes to reconfigure internal resources before all master nodes respond with the new certificate.

You can wait for the complete availability of the API endpoints with the following command:

    oc wait ClusterOperator kube-apiserver \
    --for=condition=Progressing=False \
    --timeout=1200s
oc wait ClusterOperator kube-apiserver \
    --for=condition=Available=True \
    --timeout=1200s

  

At this point, Hive will likely start complaining about the new API certificate. You can confirm that by running the following command against the cluster running Hive:

    # Assumes you are logged in to the cluster running Hive as an 
# administrator

managed_cluster_name=#type the managed cluster name here

oc describe ClusterDeployment "${managed_cluster_name:?}" \
    --namespace "${managed_cluster_name}"

The output can be somewhat verbose, but you may see the following fragment somewhere in that output:

    ...
    Last Probe Time:          ...
    Last Transition Time:     ...
    Message:                  Get "https://api.mycluster.mylab.com:6443/api?timeout=32s": x509: certificate signed by unknown authority
    Reason:                   ErrorConnectingToCluster
    Status:                   True
    Type:                     Unreachable
    ...

Updating Hive’s configuration

With the certificate signer in hand, it is time to update Hive’s configuration for that cluster.

Conceptually, we want to locate the kubeconfig file for the managed cluster and update the list of trusted certificate authorities with the signing certificate from the previous section.

OpenShift Hive cannot initially trust certificates added to existing managed clusters.

The first step is to identify the Kubernetes secret containing the kubeconfig file using the following commands:

    managed_cluster_name=#type the managed cluster name here

oc get ClusterDeployment "${managed_cluster_name:?}" \
    --namespace "${managed_cluster_name}"

secret_name=$(oc get ClusterDeployment "${managed_cluster_name}" \
        --namespace "${managed_cluster_name}" \
        -o jsonpath='{.spec.clusterMetadata.adminKubeconfigSecretRef.name}')

echo ${secret_name:?}

With the secret name in hand, it is time to dump the kubeconfig contents into a local file so we can modify it and then update it. One could edit the downloaded content for this section with a text editor, but you can use the scripted approach described below.

Note that since the file contents use the YAML format, the snippet below relies on the yq utility:

    WORKDIR=/tmp/hive-ca
mkdir -p "${WORKDIR}"

kubeconfig_yaml="${WORKDIR}/${managed_cluster_name}-kubeconfig.yaml"
ca_crt="${WORKDIR}/${managed_cluster_name}.crt"

oc extract secret/"${secret_name:?}" \
    --namespace "${managed_cluster_name:?}" \
    --keys=kubeconfig --to=- > "${kubeconfig_yaml:?}"

yq -r .clusters[].cluster.certificate-authority-data "${kubeconfig_yaml}" \
    | base64 -d > "${ca_crt}"

# Shows original CA certs for the managed cluster API server
cat "${ca_crt}"

  

At this point, the original API certificates are stored in the file designated by the ca_crt variable.

The next step is to generate a new CA file with the recent signing certificate. Note that one could argue in favor of outright replacing the contents of the kubeconfig CA data, but since there is no documentation about their contents, it is safer to combine the contents rather than risk finding out some of the contents were helpful for other reasons.

    # This is the CA file for the signing certificate for the cluster-a
additional_ca_file=${cert_dir}/ca.cer

# Merges the existing CA with the new CA
ca_new_crt_bundle="${WORKDIR}/${managed_cluster_name}-new.crt"
cat "${ca_crt}" "${additional_ca_file}" \
    | openssl base64 -A > "${ca_new_crt_bundle}"

# Generates a new kubeconfig file
ca_new_crt=$(cat "${ca_new_crt_bundle}")

yq -i "del (.clusters[0].cluster.certificate-authority-data)" "${kubeconfig_yaml}"

yq -i ".clusters[0].cluster.certificate-authority-data = \"${ca_new_crt}\"" "${kubeconfig_yaml}"

  

After the previous block, we have an updated version of kubeconfig for the cluster stored in the file location designated by the kubeconfig_yaml variable, containing the new signing certificate. The next step is to update the original Secret resource with that new configuration.

Note that there are 2 fields in the secret requiring updates:

kubeconfig
raw-kubeconfig

    kubeconfig_64_yaml="${WORKDIR}/${managed_cluster_name}-kubeconfig-64.yaml"
openssl base64 -A -in "${kubeconfig_yaml}" > "${kubeconfig_64_yaml}"
kubeconfig_64=$(cat "${kubeconfig_64_yaml}")

oc patch Secret "${secret_name}" \
      --namespace "${managed_cluster_name}" \
      --type='json' -p="[{'op': 'replace', 'path': '/data/kubeconfig', 'value': '${kubeconfig_64:?}'},{'op': 'replace', 'path': '/data/raw-kubeconfig', 'value': '${kubeconfig_64:?}'}]"

# It is a good idea to wait for the ClusterDeployment resource
# to report being ready before proceeding
oc wait ClusterDeployment "${managed_cluster_name}" \
    --namespace "${managed_cluster_name}" \
    --for=condition=Ready=true \
    --timeout=600s 

  

Assuming everything was executed successfully, OpenShift Hive should now recognize the new certificate for the managed cluster.

You can validate that Hive can connect to the cluster again by running the same oc describe command from earlier:

    oc describe ClusterDeployment "${managed_cluster_name}" \
    --namespace "${managed_cluster_name}"

The command output should contain the following excerpt arount the `Unreachable` condition:

    ...
    Last Probe Time:          ...
    Last Transition Time:     ...
    Message:                  cluster is reachable
    Reason:                   ClusterReachable
    Status:                   False
    Type:                     Unreachable
    ...

Conclusion

OpenShift Hive does a great job creating new clusters in supported Cloud providers. Still, it currently lacks primitives for certain management operations, such as adding a signed certificate to the API endpoints of the managed clusters.

This article shows how to locate and update the kubeconfig for the managed cluster, using a concrete example and code samples to effect the change.

Multi-cloud DNS delegation between GCP and AWS

2022-11-29T14:06:00.013-08:00

(You can see the source code behind this article in this GitHub repository)

Multi-cloud DNS delegation between GCP and AWS

Overview

The use case is hosting services in one cloud provider while having the DNS domain for those services managed on a different cloud provider.

This technique is useful when you group system resources in a hybrid multi-cloud environment under the same DNS domain.

For the example in this article, we already have a DNS domain hosted using AWS DNS (Route 53) on AWS, and needed to place a set of OpenShift Container Platform (OCP) clusters in GCP while using a sub-domain of that original DNS domain.

This diagram illustrates the desired outcome on both DNS zones, with AWS Route 53 handling the initial requests for names in the primary domain and deferring the queries for the sub-domain to the DNS resolvers in GCP.

Component diagram of the solution in this article.

There are two underlying assumptions for this type of solution:

The cluster administrator has considered the implications of making the DNS names of the new cluster publicly available. Note that this is unrelated to whether the cluster endpoints are accessible publicly.
The cluster administrator is responsible for ensuring clients have a proper network path to the network hosting the endpoints. For instance, if the new OCP cluster is created on a private subnet, clients outside that subnet may be able to resolve the DNS name for the cluster’s console or API server. Still, they will not be able to establish a network connection to those endpoints.

DNS record types

It may be helpful, although not strictly necessary, to understand a bit more about DNS record types as you follow along this tutorial.

For this tutorial, these are the most relevant record types:

NS: Delegates a DNS zone to use the given authoritative name servers.

A: Returns a 32-bit IPv4 address, most commonly used to map hostnames to an IP address of the host, but it is also used for DNSBLs, storing subnet masks in RFC 1101, etc.

SOA: Specifies authoritative information about a DNS zone, including the primary name server, the email of the domain administrator, the domain serial number, and several timers relating to refreshing the zone.

Create the DNS Zone in GCP

The first step is to create a non-authoritative DNS Zone in GCP, using a sub-domain of the hosted domain in Route 53.

The hosted domain in Route 53 is “cloudpak-bringup.com,” whereas the sub-domain will be “gcp.cloudpak-bringup.com”

Click on the “Create Zone” button.
Use cloud-pak-bring up-lab as the zone name.
Use gcp.cloudpak-bringup.com as the DNS name.
Click the “Create” button.
Creation form for the DNS zone

The following listing shows the alternative command if using the GCP CLI:

    project_id=# type your GCP project id here
gcloud dns \
    managed-zones create cloud-pak-bringup-lab \
    --project=${project_id:?} \
    --description= "Test clusters tied to the Bringup Lab activities." \
    --dns-name="gcp.cloudpak-bringup.com." --visibility="public" \
    --dnssec-state= "off"

  

Once the DNS zone is created, you should see the following screen:

DNS Zone in GCP Console

Set aside the list of GCP DNS servers

Click on the “NS” entry named “gcp.cloudpak-bringup.com” to see the resource record set details.

Write down the list of name servers under the “Routing data” table. That list is needed later when configuring the DNS zone in AWS.

List of DNS names for the new DNS zone

The list contains the following records in our example, but your DNS record may have a different list of DNS servers.

ns-cloud-e1.googledomains.com.
ns-cloud-e2.googledomains.com.
ns-cloud-e3.googledomains.com.
ns-cloud-e4.googledomains.com.

Create delegation records in Route 53

Locate the hosted zone in the Route 53 page of the AWS Console.

Authoritative DNS Zone in AWS

Create Name Server record

The name server record informs AWS Route 53 to delegate all requests for a sub-domain of the hosted zone to a list of name servers.

In our case, we want to associate the “gcp.cloudpak-bringup.com” sub-domain with the list of name servers obtained from the DNS zone created earlier in GCP.

Click on “Create Record” and choose the “Simple routing” strategy if prompted for the routing strategy.
Type gcp as the record name
Select “NS - Name servers for a hosted zone” as the “Record Type.” Note that this option is grayed out until you fill out the “Record name” field.
Under "Value," paste the list of DNS name servers from the GCP DNS zone.
Click the "Create records" button.

Creation of the delegated NS record for the DNS sub-domain in AWS

(optional) Restrict CAs that can create certificates for the domain

Create an additional record to indicate which CA’s can create certificates for the sub-domain.

Click on “Create Record” and choose the “Simple routing” strategy if prompted for the routing strategy.
Use gcp as the “Record name” field, matching the sub-domain used when creating the DNS zone in GCP.
Pick “CAA – Restricts CAs that can create SSL/TLS certificates for the domain” as the “Record Type.”
Choose the appropriate issuer value for the CA authority. For example, for “Let’s Encrypt,” use 0 issue "letsencrypt.org"
Keep the other default values.
Click the "Create records" button.

Creation of the CAA record for the DNS sub-domain in AWS

Validating DNS configuration

Once the DNS records are configured in both cloud providers, it is a good idea to validate the setup before attempting to create the OpenShift cluster (or whatever other service you want to respond to in the new sub-domain.)

Note that some DNS settings typically take a few moments to propagate across the various DNS servers on the Internet.

This example command queries the authority for a pseudo hostname in the new sub-domain:

dig some-non-existent-service.gcp.cloudpak-bringup.com soa +noall +authority

The command results should indicate that GCP is the authority for the sub-domain, like in the following snippet:


; <<>> DiG 9.10.6 <<>> some-non-existent-service.gcp.cloudpak-bringup.com 
soa +noall +authority
;; global options: +cmd
gcp.cloudpak-bringup.com. 300 IN  SOA ns-cloud-e1.googledomains.com. 
cloud-dns-hostmaster.google.com. 1 21600 3600 259200 300

Create the OpenShift cluster

With the DNS resolution configured in both clouds, it is time to create the OpenShift cluster in GCP.

There are multiple methods for creating an OpenShift cluster on GCP Cloud. The most basic methods are listed in the OpenShift Container Platform documentation.

There are more sophisticated mechanisms, such as Red Hat Advanced Cluster Management for Kubernetes.

Whatever method you choose, use the new sub-domain as the baseDomain field: gcp.cloudpak-bringup.com.

Once the cluster creation completes, you should see two new “A” type DNS records for the OCP endpoints in the GCP console:

*.apps.clustername.cloudpak-bringup.com
api.clustername.cloudpak-bringup.com

OCP cluster routes for the new cluster

With those DNS records in place, you can now access the resulting cluster endpoints using their DNS names and not resort to local alterations.

OCP console hosted in the new DNS sub-domain

Conclusion

This type of DNS delegation is a common solution for hybrid multi-cloud arrangements where you want to share the primary DNS domain name for all systems spread across the different clouds.

The detailed instructions should be sufficient to generalize the solution to different DNS providers, paying attention to the placement of the appropriate “NS” DNS records in the respective DNS zones.

Running Knative services on AWS spot instances

2022-10-19T06:00:00.017-07:00

(You can see the source code behind this article in this GitHub repository)

This article is a technical exploration of several technologies to reduce the cost of running workloads in Kubernetes.

The exploration goes into how the characteristics of each of these technologies favor certain types of workloads and ultimately combines all these components into a deployment strategy that utilizes their core strengths.

The general approach works better with containers designed using serverless principles, such as reduced internal state, lightweight footprints, and quick startup times.

Components

This section lists the core components of this article, a brief overview, and pointers for further reading.

OpenShift Container Platform
AWS and Spot Instances
Knative

Note: I struggled with the decision to use OpenShift over EKS for this tutorial. On the one hand, using AWS as the cloud provider makes EKS a more natural choice. On the other hand, I have easy access to OpenShift licenses, which also makes the article more readily portable to GCP and Azure. If you prefer using an EKS cluster, replace the OpenShift portions with the instructions in this AWS tutorial, where "Spot Managed node groups" are the rough functional equivalent to the concept of OpenShift machine sets.

OpenShift Container Platform

OpenShift Container Platform (OCP) is a Kubernetes distribution from Red Hat.

OCP adds a rich management interface around the Kubernetes runtime, making it easier to install the open-source components used in this article and, most importantly, visualize the resources and browse their metrics. I made a special effort to avoid using OCP-specific commands where possible, making life a little easier for those who wish to attempt this exercise with a different distribution or even "plain" Kubernetes.

AWS and Spot Instances

I chose AWS as the cloud provider in this experiment for a few reasons:

It supports the concept of "spot instances," where customers can request instances from a pool of spare capacity in the provider. These instances are relatively cheaper than regular instances, with one catch: the cloud provider can take them back after giving your cluster a short notice (typically 30 to 60 seconds).
OpenShift has native support for requesting spot instances from AWS.
It is the most widely used public cloud provider, making this examination relevant to a broader audience.

Note that OpenShift also has native support for allocating nodes on Azure or Google Cloud spot instances, so you should be able to replicate the same setup with minor changes to the OCP MachineSet samples in this article.

Knative

Knative is my favorite auto-scaling component in Kubernetes, not the least because it favors a stateless micro-service pattern that carries very into the various serverless offerings across different cloud providers.

At its core, Knative has the "Serving" component, which processes Knative service definitions. Service definitions map to various Kubernetes services, combining a container image with targets of concurrency and sustained requests per second.

The serving component manages the number of replicas for that container to meet service targets. Depending on the service definition, terminates all containers for that workload when there are no more incoming requests.

Environment configuration

Choose the AWS region for the cluster
Create an OCP 4.11 cluster
Create the OCP MachineSet using spot instances
Create cluster and machine autoscalers
Install Knative on the cluster
Enable Knative feature flags

Note: This section requires you to apply many resources mentioned in the article to the cluster. You can apply them from a terminal using Heredoc constructs (see below) or using the OpenShift Import YAML button.

# make sure you logged into the cluster 
# from this terminal session

cat << EOF | kubectl apply -f -
...yaml resource copy-pasted from the article
EOF

Choose the AWS region for the cluster

Assuming you have the flexibility to choose the AWS region, try and choose one with a higher "placement score" of provisioning spot instances.

You can assess that score using the "Spot placement score" page of the AWS console and play with different requirements for the number of instances, CPUs, and memory size.

The inline documentation on the page is pretty self-explanatory about the odds of acquiring spot instances meeting your requirements.

Scores serve as a guideline, and no score guarantees that your Spot request will be fully or partially fulfilled. A score of 10 means that your Spot capacity request is highly likely to succeed in that Region or Availability Zone at the time of the request. A score of 1 means that your Spot capacity request is not likely to succeed.

I experimented with many size combinations, and the placement scores matched my expectation that smaller instances (e.g., 4x16) would have higher scores than larger instances (e.g., 16x64.)

My empirical results also showed that the placement score was a good indicator of how soon the cloud provider would reclaim the spot instance. Requesting spot instances of size 16x64 was essentially unusable for a Kubernetes cluster, with the cluster spending more time bringing an instance online than actually being able to load pods in the instance. The results may vary with region and time of day, but statistically speaking, working with smaller spot instances tends to yield much better results.

Create an OCP 4.11 cluster

Note: I mentioned earlier that it is possible to replace OCP with an AWS EKS cluster - using "Spot Managed node groups" instead of OCP machine sets. If there is enough interest, I can try and replicate the entire exercise with EKS.

These instructions were tested with OCP 4.11.7 but should work on OCP 4.6 and above.

I recommend that you use at least three availability zones in the target region (e.g., pick us-east-1a, us-east-1b, and us-east-1c availability zones if installing the cluster in the us-east-1 region.) The multiple availability zones help later in the setup when it is time to create the new worker pool using spot instances.

You can install OCP on AWS using different variations of the IPI procedure (Installer Provisioned Infrastructure,) where the installer requests all infrastructure components from the cloud provider and loads the Kubernetes software on the resulting computing instances.

The OCP's installation documentation covers the entire procedure, so instead of repeating those instructions, I will add a few comments about the variations:

Use Red Hat's cloud console. This method also offers a user interface around the IPI-based installation, creating a cluster that is functionally equivalent to the other methods.
Using Red Hat Advanced Cluster Management for Kubernetes. Assuming you have access to this product, it offers a user interface around the IPI-based installation process, which you can use to create the cluster with a few clicks without getting distracted collecting keys and certificates for the installation process.

Create the OCP MachineSet using spot instances

OCP machine sets abstract the underlying cloud provider compute instances for the cluster nodes.

A machine set contains the cloud provider-specific information required to provision new compute instances, such as the region, availability zone, number of CPUs, amount of memory, disk capacity, subnets, and a few others.

OCP requests that "machine" to the cloud provider, and once the resulting cloud instance is ready, OCP loads it with all the software required to run a Kubernetes node and makes it a part of the cluster.

For illustrative purposes, the following listing is the example of a MachineSet resource immediately after requesting the creation of an OCP cluster. I stripped out a few Kubernetes elements to make it a little more readable, but you can see the entire resource here.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  name: cluster-name-worker-us-east-1a
  namespace: openshift-machine-api
spec:
  replicas: 1
  template:
    metadata:
      labels:
        ...
    spec:
      metadata: {}
      providerSpec:
        value:
          apiVersion: machine.openshift.io/v1beta1
          kind: AWSMachineProviderConfig
          instanceType: m5.xlarge
          blockDevices:
            - ebs:
          placement:
            availabilityZone: us-east-1a
            region: us-east-1
          securityGroups:
            ...
          subnet:
            ...

Creating a new machine set requires applying a custom resource to the cluster, like the one in the previous section. Ultimately, the machine set for using AWS spot instances is not that different from a regular machine set, although it requires a few essential modifications:

Adding a spotMarketOptions element to the node template. A node template is an element in the machine set that tells OCP about the extra configuration of the cluster node associated with each instance. The spotMarketOptions element informs OCP that it should request AWS for spot instances instead of on-demand instances. A related feature not explored in the article is the ability to include a maxPrice inside the spotMarketOptions element, indicating the maximum hourly rate you are willing to pay for the instance, which can be pretty helpful in specific scenarios.
Adding a taint to the node template so that only workloads with toleration for that taint will run in the node. This setting prevents essential pods or pods that cannot tolerate frequent interruptions from landing in the nodes created on EC2 spot instances.
Adding a label to the nodes indicating that they run on interruptible instances. Later we can tell Knative to provision service pods with an affinity for those nodes.
(optional) Add a label indicating the purpose of these machines (and respective cluster nodes.) This label is helpful when it is time to delete the machine set.

The definition of specific fields can be quite involved, such as secrets and subnets, so I tend to copy and then modify one of the existing machine sets in the cluster. I wrote an example using shell script and the yq utility to create the machine sets to request spot instances.

#!/bin/sh

infra_id=$(kubectl get Infrastructure cluster -o jsonpath='{.status.infrastructureName}')
echo "Infrastructure id is: ${infra_id}"

# Make sure to pick a valid instance type for the cloud region
instance_size=${1:-c5a.xlarge}

kubectl get MachineSet \
        -n openshift-machine-api \
        --selector hive.openshift.io/machine-pool=worker \
        --selector hive.openshift.io/managed=true \
        -o yaml \
    | yq 'del (.items[].status)' \
    | yq 'del (.items[].metadata.annotations)' \
    | yq 'del (.items[].metadata.uid)' \
    | yq 'del (.items[].metadata.resourceVersion)' \
    | yq 'del (.items[].metadata.generation)' \
    | yq 'del (.items[].metadata.creationTimestamp)' \
    | yq 'del (.items[].metadata.labels."hive.openshift.io/managed")' \
    | yq 'del (.items[].metadata.labels."hive.openshift.io/machine-pool")' \
    | yq '.items[].metadata.labels += { "machine.sourcepatch.com/interruptible": "true" }' \
    | yq '.items[].spec.template.metadata.labels += { "machine.sourcepatch.com/interruptible": "true" }' \
    | yq '.items[].spec.template.spec.providerSpec.value.spotMarketOptions={}' \
    | yq '.items[].spec.template.spec.taints += [{ "effect": "NoSchedule", "key": "workload.sourcepatch.com/interruptible", "value": "true" }]' \
    | yq '.items[].spec.template.spec.metadata.labels += { "sourcepatch.com/node.interruptible": "true"}' \
    | yq ".items[].spec.template.spec.providerSpec.value.instanceType= \"${instance_size:?}\"" \
    | sed "s/name: ${infra_id:?}-worker/name: ${infra_id:?}-worker-spot/" \
    | sed "s/machineset: ${infra_id:?}-worker/machineset: ${infra_id:?}-worker-spot/" \
    | kubectl apply -f -

You can see a graphical visualization of the differences between an original machine set and an interruptible machine set in the illustration below.

Immediately after creating the new machine sets, you may notice that they may not report as ready or available. You can use the following command to list their status, noticing how it selects the results using the label added in the .spec.template.metadata.labels section of the modified machine set.

kubectl get machineset \
    -n openshift-machine-api \
    --selector machine.sourcepatch.com/interruptible="true"

The command should produce an output like this:

NAME                         DESIRED   CURRENT   READY   AVAILABLE   AGE
...-worker-spot-us-east-1a   1         1                             6s
...-worker-spot-us-east-1b   1         1                             6s
...-worker-spot-us-east-1c   1         1                             6s
...-worker-spot-us-east-1d   1         1                             6s

After a couple of minutes, assuming the cloud provider fulfilled the spot instance requests, you should see the machines (OCP's representation of the cloud provider instances) already running with the requested instance type.

Repeat the previous command until it produces an output like this:

NAME                         DESIRED   CURRENT   READY   AVAILABLE   AGE
...-worker-spot-us-east-1a   1         1         1       1           5m1s
...-worker-spot-us-east-1b   1         1         1       1           5m1s
...-worker-spot-us-east-1c   1         1         1       1           5m1s
...-worker-spot-us-east-1d   1         1         1       1           5m1s

After another couple of minutes, with all required software loaded into the instances, you should also see the Kubernetes nodes added to the cluster. You can verify the state of the new nodes with the command below, noticing how it targets the nodes using the node label declared in the .spec.template.spec.metadata.labels section of the modified machine set.

kubectl get node \
    --selector sourcepatch.com/node.interruptible="true"

The command should produce an output similar to this:

NAME                           STATUS   ROLES    AGE     VERSION
ip-10-0-nnn-200.ec2.internal   Ready    worker   6m42s   v1.24.0+...
ip-10-0-nnn-201.ec2.internal   Ready    worker   7m11s   v1.24.0+...
ip-10-0-nnn-202.ec2.internal   Ready    worker   6m31s   v1.24.0+...
ip-10-0-nnn-203.ec2.internal   Ready    worker   6m24s   v1.24.0+...

You should also be able to see the EC2 spot instances allocated in the respective AWS panel. Those instance requests have a status of "fulfilled."

Create cluster and machine autoscalers

As an optional feature, OCP machine sets can have a variable range of machines instead of a fixed number of replicas. OCP uses pod scheduling information in the cluster to decide the optimal number of instances in a machine set.

The first step to leverage that feature is to enable the OCP cluster autoscaler feature, applying the following resource to the cluster (you can also find this resource in the GitHub repository for this article.)

Important: The settings in this ClusterAutoscaler resource are adequate for the relatively small examples in the article. Make sure you read the OCP documentation for detailed information and important considerations for each setting.

---
apiVersion: autoscaling.openshift.io/v1
kind: ClusterAutoscaler
metadata:
  name: default
spec:
  podPriorityThreshold: -10
  resourceLimits:
    maxNodesTotal: 24
    cores:
      min: 8
      max: 128
    memory:
      min: 4
      max: 256
  scaleDown:
    enabled: true
    delayAfterAdd: 10m
    delayAfterDelete: 5m
    delayAfterFailure: 30s
    unneededTime: 5m
    utilizationThreshold: "0.4"

The second step is to create OCP machine autoscalers for the machine sets created in the previous sections. You need one machine autoscaler per machine set, which tells OCP to adjust the number of replicas in the respective machine set to match the capacity requirements of the cluster.

You can create the resources using a code block like the one below, which iterates through the names of the machine sets created in the previous section. It establishes the respective autoscaler for the machine set, with a range of 0 to 3 machines:

kubectl get MachineSet \
    -n openshift-machine-api \
    --selector machine.sourcepatch.com/interruptible="true" \
    -o template='{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' \
| while read -r ms
do
  cat << EOF | kubectl apply -f -
---
apiVersion: autoscaling.openshift.io/v1beta1
kind: MachineAutoscaler
metadata:
  labels:
    machine.sourcepatch.com/interruptible: "true"
  name: ${ms}
  namespace: openshift-machine-api
spec:
  minReplicas: 0
  maxReplicas: 3
  scaleTargetRef:
    apiVersion: machine.openshift.io/v1beta1
    kind: MachineSet
    name: ${ms}
EOF
done

Since the minimum number of replicas was set to zero, and given the taint in the interruptible machine sets preventing other workloads from running in those nodes, the cluster should eventually delete all machines associated with the new machine sets until there are pods scheduled to run on those nodes.

You can observe the node deletion progress by listing the machine resources labeled with the label assigned to their respective machine set in the previous sections.

kubectl get Machine \
    -n openshift-machine-api \
    --selector machine.sourcepatch.com/interruptible="true"

If you timed it right, you should see an output like the one below, where you can still see the cluster deleting the unused nodes.

NAME                            PHASE     TYPE       REGION    ZONE       AGE
...-worker-spot-us-east-1b-...  Deleting  c5a.xlarge us-east-1 us-east-1b 13m
...-worker-spot-us-east-1c-...  Running   c5a.xlarge us-east-1 us-east-1c 13m
...-worker-spot-us-east-1d-...  Running   c5a.xlarge us-east-1 us-east-1d 13m

Install Knative on the cluster

Installing Knative in OCP is simplified through the concept of Kubernetes operators. OCP calls it the Openshift Serverless operator, which is essentially a downstream version of the Knative open-source project.

You can install the OpenShift Serverless operator directly from the OCP console by navigating to the OperatorHub tab, selecting "OpenShift Serverless" and clicking "Install."

You can also install the operator from a terminal, applying the respective operator group and subscription resources, like the one listed below for convenience:

---
apiVersion: v1
kind: Namespace
metadata:
  name: openshift-serverless
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: openshift-serverless-group
  namespace: openshift-serverless
spec:
  upgradeStrategy: Default
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: serverless-operator
  namespace: openshift-serverless
spec:
  channel: stable
  installPlanApproval: Automatic
  name: serverless-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace

The next step is to install the Knative serving instance, which is responsible for the scheduling of containers to meet the targets of Knative service definitions.

You can see this resource in source format, which is listed for reading convenience here:

---
apiVersion: operator.knative.dev/v1alpha1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving
spec:
  cluster-local-gateway: {}
  controller-custom-certs:
    name: ''
    type: ''
  knative-ingress-gateway: {}
  registry: {}

Enable Knative feature flags

As of now, and by default, Knative does not enable pod affinity and taint tolerations for Knative service definitions. If you attempt to create Knative service definitions before enabling those features, you will receive error messages like this:

Error from server (BadRequest): error when creating "STDIN": admission webhook "validation.webhook.serving.knative.dev" denied the request: validation failed: must not set the field(s): spec.template.spec.nodeSelector, spec.template.spec.tolerations

and

Error from server (BadRequest): error when creating "STDIN": admission webhook "validation.webhook.serving.knative.dev" denied the request: validation failed: must not set the field(s): spec.template.spec.tolerations

Before you can use those Knative features, you must enable the corresponding feature flags :

Pod spec affinity: https://knative.dev/docs/serving/configuration/feature-flags/#kubernetes-node-affinity
Enable taint toleration: https://knative.dev/docs/serving/configuration/feature-flags/#kubernetes-toleration

You can modify the Knative configuration object directly from the OpenShift console or run these two commands from a terminal:

kubectl patch ConfigMap config-features \
    -n knative-serving \
    -p '{"data":{"kubernetes.podspec-affinity":"enabled"}}'

kubectl patch ConfigMap config-features \
    -n knative-serving \
    -p '{"data":{"kubernetes.podspec-tolerations":"enabled"}}'

The examples in this article do not use the node selector feature. Still, you can enable it if you want to experiment with different strategies for matching Knative workloads to specific nodes:

kubectl patch ConfigMap config-features \
    -n knative-serving \
    -p '{"data":{"kubernetes.podspec-nodeselector":"disabled"}}'

And now we can start creating Knative service definitions.

Service requests with spot instances

This section shows all components working together. The first step is to create the Knative service definition, which gives us a target service endpoint, then use a utility to exercise that endpoint.

Create service definitions
The service sample
Validating the service readiness
Validating the installation
Driving heavy loads

Create service definitions

A Knative service definition is analogous to pairing a Kubernetes workload definition with a Kubernetes service. You can read the entire service specification for details, which I summarize as follows:

Container settings, such as image source, resource requests, limits, probe definitions
Service settings include the service's endpoint URL, target concurrency, and throughput metrics.
Routing settings, such as how much traffic goes to each service revision.

The Knative serving module listens to incoming requests on an endpoint and continuously decides the ideal number of containers to meet the service targets - concurrency and requests per second.

The service sample

The basis for this section is a heavily modified version of this Knative code sample: https://knative.dev/docs/serving/autoscaling/autoscale-go/.

Create a Knative service instance with an affinity for the nodes created from the machine set in the previous sections (identified with the "node.interruptible" label) and toleration for the interruptible taint (identified with the node.interruptible taint:)

---
apiVersion: v1
kind: Namespace
metadata:
  name: knative-interruptible
---
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: mywork
  namespace: knative-interruptible
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/target-utilization-percentage: "70"
        autoscaling.knative.dev/target: "10"
    spec:
      containerConcurrency: 15
      containers:
        - image: "gcr.io/knative-samples/autoscale-go:0.1"
          name: user-container
          resources:
            requests:
              memory: "256Mi"
              cpu: "200m"
            limits:
              memory: "256Mi"
              cpu: "200m"
      enableServiceLinks: false

      # Schedule pods in nodes labeled 
      # "sourcepatch.com/node.interruptible"
      # and tolerate their respective taints.
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key:  sourcepatch.com/node.interruptible
                    operator: Exists
                    values: []
      tolerations:
        - key: "workload.sourcepatch.com/interruptible"
          operator: "Exists"
          effect: "NoSchedule"

Validating the service readiness

Before we start sending the request, let's first validate that the service is ready:

Install the Knative CLI

Ask for the service details:

kn service describe mywork -n knative-interruptible

At this point, all interruptible nodes may have been deallocated due to inactivity. In that case, you would see the message "Error: Unschedulable` for the service revision.

Name:       mywork
Namespace:  knative-interruptible
Age:        12s
URL:        https://mywork-knative-interruptible.apps.mycluster.domain.io

Revisions:
      !  mywork-00001 (latest created) [1] (12s)
        Error:  Unschedulable
        Image:  gcr.io/knative-samples/autoscale-go:0.1 (at e5e89c)

Conditions:
  OK TYPE                   AGE REASON
  !! Ready                  12s RevisionMissing
  !! ConfigurationsReady    12s RevisionFailed
  !! RoutesReady            12s RevisionMissing

You can confirm that the problem is only temporary due to the machine autoscaler being in the process of allocating a new node by inspecting the service's underlying pods.

kubectl get pods \
    -n knative-interruptible \
    --selector serving.knative.dev/service=mywork -o yaml

If you waited too long to run the previous command, you might get an empty list result, which is potentially a good sign, indicating that Knative was able to create the pod, validate that the service is configured correctly, and terminate the pod.

In that case, your service is configured correctly, and you can skip the following paragraphs about troubleshooting potential (or transitory) scheduling problems.

In case the pods are still pending, they should contain output resembling something like the output below:

{ ... "conditions": [ { "lastProbeTime": null, "lastTransitionTime": "2022-10-17T20:02:38Z", "message": "0/7 nodes are available: 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }, 7 node(s) didn't match Pod's node affinity/selector. preemption: 0/7 nodes are available: 7 Preemption is not helpful for scheduling.", "reason": "Unschedulable", "status": "False", "type": "PodScheduled" } ], "phase": "Pending", "qosClass": "Burstable" }

You can further confirm that problem is transitory by listing the available machines in the machine set created in this article:

kubectl get Machine \
    -n openshift-machine-api \
    --selector machine.sourcepatch.com/interruptible

The command could produce an output like this, indicating that a machine is being provisioned:

NAME                 PHASE          TYPE         REGION      ZONE         AGE
...-worker-spot...   Provisioning   c5a.xlarge   us-east-1   us-east-1b   24s

Under these circumstances, the new interruptible node should be ready in a couple more minutes, in which case the Knative service should report being ready.

You can validate that the service is ready by running the following command:

kn service describe mywork -n knative-interruptible

The output should look something like this:

Name:       mywork
Namespace:  knative-interruptible
Age:        3m
URL:        https://mywork-knative-interruptible.apps.mycluster.domain.io

Revisions:
  100%  @latest (mywork-00001) [1] (3m)
        Image:     gcr.io/knative-samples/autoscale-go:0.1 (at e5e89c)
        Replicas:  1/1

Conditions:
  OK TYPE                   AGE REASON
  ++ Ready                  40s
  ++ ConfigurationsReady    41s
  ++ RoutesReady            40s

Validating the installation

There are many alternatives for validating the installation, such as the ones in the following list:

A simple curl request to the service endpoint
If you plan to take this exploration into production, I strongly encourage you to look into something like the K6 framework, then set up K6 locally and in the cluster to gather detailed metrics for continued analysis during development and post-deployment.
Use the hey load generator (go install github.com/rakyll/hey@latest) referenced in the Knative documentation.

Note: You will need to install the Golang SDK before installing the hey utility since the project does not release binaries for all platforms. The utility, by default, will be downloaded to $HOME/go/bin, and you can move it to a system folder in our PATH environment variable such as /usr/bin or /usr/local/bin

Important: When running a performance load utility such as K6 and hey, be particularly mindful of where you will run the utility, considering the network bandwidth and data transfer costs between the client and the cluster. I strongly suggest you run these utilities from a location as close as possible to the cluster, such as another compute instance in the same private subnet where the cluster nodes are running.

For this article, I am using the hey utility, which provides a good balance of ease of use with the ability to generate controlled bursts of requests toward a service endpoint.

Let's start with a run of 5 sustained requests over 30 seconds:

url=$(kn service describe mywork -n knative-interruptible -o url)

hey -z 10s -c 5 -t 240 "${url:?}?sleep=100&prime=10000&bloat=5" \
    && kubectl get pods -n knative-interruptible

Note the -t 240 parameter, which instructs the utility to wait longer for responses from the service, accounting for the possibility that all autoscaled machine sets created in this article may have scaled down to zero before you got to this step.

This light load does not require many pods but is a good warm-up exercise, and successful responses validate that all components are working correctly.

While the utility is still querying the service, run the following command from a different terminal to determine the service status:

kn service describe mywork -n knative-interruptible

Notice that at first, due to the machine set autoscaling configured in previous sections, the machine set for these pods may not have any machine allocated to them, which may lead to an output like this:

Name:       mywork
Namespace:  knative-interruptible
Age:        16m
URL:        https://mywork-knative-interruptible.apps.mycluster.domain.io

Revisions:  
  100!  @latest (mywork-00001) [1] (16m)
        Error:     Unschedulable
        Image:     gcr.io/knative-samples/autoscale-go:0.1 (at e5e89c)
        Replicas:  0/7

Conditions:  
  OK TYPE                   AGE REASON
  !! Ready                  13s RevisionFailed
  !! ConfigurationsReady    13s RevisionFailed
  ++ RoutesReady            13m

Notice how the field Replicas lists 0/7, indicating that Knative assessed the need for seven replicas to handle the incoming requests and that zero of those pods could be allocated until a new node comes online.

If you wait a few more minutes, repeating the previous command should produce an output like this:

Name:       mywork
Namespace:  knative-interruptible
Age:        3h
URL:        https://mywork-knative-interruptible.apps.mycluster.domain.io

Revisions:
  100%  @latest (mywork-00001) [1] (24m)
        Image:     gcr.io/knative-samples/autoscale-go:0.1 (at e5e89c)
        Replicas:  7/7

Conditions:
  OK TYPE                   AGE REASON
  ++ Ready                  24m
  ++ ConfigurationsReady    24m
  ++ RoutesReady            24m

We can see that the service is reporting being ready and has determined that it allocated all seven container replicas to meet the service requirements.

Driving heavy loads

With the installation validated, it is time to generate a higher load to exercise the total capacity of the pool of interruptible nodes.

Let's request sustained 250 requests per second for 300 seconds. Note the usage of 300-second timeout (using the -t 300 parameter) to account for the time it may take for any eventual scaling up of a machine set.

url=$(kn service describe mywork -n knative-interruptible -o url)

hey -z 300s -c 250 -t 300 "${url:?}?sleep=100&prime=1000&bloat=1" \
    && kubectl get pods -n knative-interruptible

You should be able to see the various signs of activity in the cluster using different methods, such as:

Running the kn service describe mywork -n knative-interruptible command to determine the current allocation of service replicas versus the target number of replicas.
Observing the corresponding Deployment resource in the OCP "Workloads -> Deployments" tab and switching to the knative-interruption project at the top.
Listing the corresponding pods associated with the service with kubectl get pods -n knative-interruptible.

Once the hey utility returns, you should see an output like the one below with likely different numerical results:

Tue Oct 18 20:16:31 UTC 2022

Summary:
  Total:        300.1402 secs
  Slowest:      1.2196 secs
  Fastest:      0.0031 secs
  Average:      0.1113 secs
  Requests/sec: 2246.3932
  
  Total data:    66059521 bytes
  Size/request:    97 bytes

Response time histogram:
  0.003 [1]     |
  0.125 [621740]|■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.246 [50065] |■■■
  0.368 [1104]  |
  0.490 [428]   |
  0.611 [306]   |
  0.733 [201]   |
  0.855 [101]   |
  0.976 [100]   |
  1.098 [114]   |
  1.220 [73]    |


Latency distribution:
  10% in 0.1034 secs
  25% in 0.1039 secs
  50% in 0.1045 secs
  75% in 0.1055 secs
  90% in 0.1128 secs
  95% in 0.1667 secs
  99% in 0.1957 secs

Details (average, fastest, slowest):
  DNS+dialup:    0.0000 secs, 0.0031 secs, 1.2196 secs
  DNS-lookup:    0.0000 secs, 0.0000 secs, 0.0200 secs
  req write:    0.0000 secs, 0.0000 secs, 0.0101 secs
  resp wait:    0.1112 secs, 0.0031 secs, 1.2196 secs
  resp read:    0.0000 secs, 0.0000 secs, 0.0010 secs

Status code distribution:
  [200]    674224 responses
  [502]    9 responses

Tuning

Notice how the client in the previous example reported a few HTTP responses of "502", which means those requests effectively failed. After troubleshooting the problem, I noticed that some pods were exceeding their memory reservation, already set at a rather lofty - for a serverless container - 256Mi of RAM.

This small error rate is not a problem for an exploratory workload like this, especially when using a container image designed to stress the service boundaries through its prime and bloat query parameters. You can try different combinations that increase or improve the failure rates, knowing that higher values for these parameters will require higher container resource requests and possibly result in higher rates of HTTP error responses.

When working with real workloads meant for a production environment, this type of setup with dynamic capacity allocation requires a firm grip on the performance characteristics of your container, chiefly among them:

Maximum number of concurrent requests per instance
Container memory utilization while serving that maximum number of requests
Container CPU utilization while serving that maximum number of requests
Time to readiness, counting the time between the pod being assigned to a node and all readiness probes returning their first successful reply.

In general, you want to optimize these parameters for the best balance between quick response times and resource utilization and then reflect these parameters in the Knative service definition.

Conclusion

This article shows how the cluster auto-scaling and Knative services team up to run over interruptible cloud instances and generate significant cost savings for workloads with more relaxed service-level objectives.

These savings are multiplied over the life of the service, with the cluster autoscaling reducing fixed capacity allocation and the interruptible cloud instances reducing the hourly cost of that remaining capacity.

For the variable cost portion, the hourly savings depend on the spot price of the instance at the time of the request. In the case of AWS, you can see a "savings" summary at any point to give you an idea of how much you saved over the allocation of on-demand instances.

For the fixed allocation part, Knative pairs well with the cluster autoscaler, with the scheduling of services using scale-to-zero settings. In those scenarios, the reduced activity on an endpoint eventually translates to the deallocation of nodes reserved for that service (using the affinity and tainting techniques explained in this article,) producing significant cost savings over fixed allocation strategies for pods and nodes.

References

How to Outperform a 10x Developer

2022-09-12T15:00:00.006-07:00

(I originally posted this article on medium.com)

First awareness. Then productivity.

From a software developer’s perspective, the complexity of a production system makes it abstract and distant, so we tend to shut it out and focus on the immediate work of writing code.

Jeff Foster wrote a great story on the origins of the expression “10x developer,” contrasting the value of outsized code-writing skills against the ability to focus on the product.

From a DevOps perspective, its all-inclusive reach from development to operations demands collaboration. Focusing on the product trumps coding skills every time, not the least because sometimes not writing new code is the best decision for the product.

This story offers concrete examples where a better understanding of a production system can change how we see our roles as software developers and significantly improve how we design systems and write code.

When quality is not enough

Quality code and honed skills still matter, of course. Software developers should observe good practices such as automated linters, code reviews, unit testing, and many others. Still, writing great quality code fast is a small fraction of all the tasks required to bring a system to production and keep it alive.

In that sense, organizations should not spend too many resources finding and nurturing 10x developers. Firstly, they may not be actual 10x developers. Secondly, and most importantly: sustained software development must include supporting material, such as design documentation and operating procedures. Those documents must be constantly reviewed and updated in collaboration with people using them. You can streamline the coordination activities, but the underlying human interactions and workflows will hardly happen 10x faster.

Update on 9/22: Hajime Vukelic added a comment to the story, pointing out that I made it sound like outsized individual performance is impossible or undesirable. I still think it is difficult in a DevOps practice, but his thoughtful reasoning stands on its own and also led me to read one of his personal stories, titled “What’s a high-differential developer”. I think it is an excellent addition to this general topic.

You want the developers who perform at a higher, yet attainable, level (1.2x developer?) so that other team members can understand and assimilate more productive techniques and behaviors.

Growing talent is the better path toward productivity. The characters on the right are too unique and rare to make a significant impact across multiple teams.

Lesson 1 — Greatness takes time: 27 times more.

I don’t mean “time” as in “decades of experience” — that helps too — but rather “time” as the actual time it takes to do great work.

Even the most talented developers sabotage themselves with poor estimations of how long it takes to go from a prototype to a sustainable system in production. I must emphasize the word “sustainable,” where on-call shifts produce few page-outs that even people who did not write the code can resolve.

My favorite technique to estimate the total cost of a feature in production is to create a working prototype and then multiply the time spent on the prototype by 27. That is right; if it takes me two days to develop a prototype, it may take nearly three months (2x27=54 business days) of combined effort across the whole team to have it humming along in production.

Update on 9:22: User Liquid Analytics made an excellent point in the comment section about the 27x multiplier becoming a possible inhibitor to meaningful progress. I should have mentioned that the idea is to deploy the prototype using modern “test in production” approaches, such as trunk-based development behind dark launches or feature flags.

Why 27x?

Frederick Brooks Jr. explained the first “9x” in “The Mythical Man-Month,” his landmark set of essays on software development in book form.

The first 3x multiplier is the cost of going from running code (the book uses the term “program,” characteristic of the time) to a reliable application (the book used the term “program product.”)

The second 3x multiplier comes from the additional testing needed to turn that reliable application into a system (a “system product.”)

Brooks covered the aspects of creating software to be handed to customers for on-premise deployment — characteristic of IBM’s business model at the time. As such, it excludes the costs associated with operational aspects, where I add the final 3x factor — based on my experience in operations engineering, so your multiplier may vary — to turn a system into an operable Cloud-based service.

Brooks explained why something ready to be deployed at a customer site takes nine times the effort of creating a prototype. I added the extra 3x factor for deploying and maintaining the system.

That final multiplier includes development activities such as:

Instrumenting the system for observability
Documenting operational procedures based on the system design
Development and upkeep of the continuous deployment pipeline
Integration testing with other cloud provider services.

Combining all multipliers (3x3x3) gives us the 27x additional cost to take a prototype to production.

Why it matters: While a seasoned developer may not undershoot the sizing of a new feature by the entire 27x multiplier, anything beyond a 2x or 3x factor is guaranteed to create intractable surprises across the team and, more importantly, deprive you of the time needed to add and validate everything you need to avoid handing a nightmare to the operations team.

Lesson 2 — Operations engineering trumps art

By all means, art has its place, and some corners of the system may look clever or exquisite, but ultimately form follows function, and the function of a production system is to run software that meets requirements and cost objectives.

This section covers lessons learned in operations and how they should affect design and coding activities, accounting for a portion of that 27x multiple.

The section does not cover the entire spectrum outside code development, which would make this story too long for now.

More components. More cost. Check everyone’s budget. Taking operational costs into account during design and architecture is the most important lesson I have ever learned in my time in operations. Any new component in the system alters the cost structure of operating that system, so you need to ask yourself whether that component is removing more cost than it is adding and be ready to forego adding something that looks really fun into the system.

After all, the same world where CFOs (rightfully) tell you that bringing in more revenue trumps cutting expenses is inhabited by developers who will abstract a 50ms SQL query into a microservice-wrapped series of paginated RESTful calls (true story.)

I don’t mean to take a reactionary approach to change, but this is an area where one needs to find ways to collaborate or, at a minimum, consult with the operations team.

World-class status and health endpoints. No one can afford to fumble after the system status during an outage, especially in micro-service architectures with hundreds of components.

A production system needs an overall health point aggregating the health from various dependencies. The idea is that an operator can quickly assess (1) which parts of the system are not working and (2) which dependencies are not working. The “why” part comes later.

I listed my suggestions for designing health endpoints in my article on readiness and liveness for Kubernetes containers — most of the recommendations in that article also apply to non-Kubernetes systems.

Log errors from the reader’s perspective. Once an operator realizes that the system is not entirely healthy, the next step is understanding what is causing it and how to fix it.

At an elementary level, something should be happening, and it is not. From the perspective of the person reading a log message, the most helpful log messages follow a template like this:

[ERROR|WARNING|INFO]: [Component X] attempted to [take action Y], which returned [response Z].

That format may seem evident at first — it looks like a regular subject-verb-object construct — but it is the terminology that matters the most:

Is “component X” something that exists in the system documentation?
Is that “action Y” something accessible to the reader?
Is “response Z” mentioned somewhere in a troubleshooting section?

Log messages can help explain the system behavior, but only if the message contents match the contents of the official documentation, even if forums unaffiliated with the product fill some gaps.

I have read my fair share of error messages that made me thankful that someone spent the time adding them to the code but gave me pause about referencing file names and library calls that only meant something to their authors. Those internal references should be in dedicated trace files or marked with a “debug” prefix for easy filtering.

Ultimately, every log message must lead to a clear resolution step that does not involve contacting the message’s author or reading the source code.

Write system documentation: Writing documentation forces you to structure your understanding of the system into a new medium. The activity brings many benefits to the system besides the imperative of telling people how to install, monitor, secure, or troubleshoot the system.

The act of writing requires authors to think through aspects of the system that may be difficult to explain.

Often, that difficulty may indicate potential design flaws, such as struggling to explain the correct order of installation of all components. Other times, the struggle may be due to “underdeveloped” areas of the product requiring long stretches of pseudo-code instructions for the reader (“…then click here, type this, wait a few seconds, then a panel will pop up, find a button named…”)

When done correctly, documentation also acts as a gathering point for collaborators that may have the willingness to donate their knowledge but not the time to figure out how to do it. These types of contributions are a huge productivity boost for the entire ecosystem (developers and users) and tend to get otherwise lost in team channels and private conversations.

Timebox calls to remote components. In distributed systems, we know better than depend on remote agents always to respond quickly and reliably. Still, we often go with default timeout settings in client libraries and utilities without giving it a second thought because we assume their developers already figured out the magic settings that work for everybody.

That is a common oversight in a world of libraries and utilities shipped with sensible defaults and systems adopting the latest design and operational techniques for maximum availability. The misplaced confidence in the resulting system works out until one of those systems fails, leaving your component unconditionally stuck on a 2-hour wait for a TCP timeout.

Always look for remote calls in the code and make sure you know their limits and how your code will handle those limits:

Maximum connection response time
Maximum request response time
Maximum number of retries

Remote systems or the network paths between your system and the remote system may be so reliable that we start treating them as function calls in local memory. Scrutinize all libraries and network stacks to find their tolerance for connectivity problems and error codes in case of problems.

Creating an effective retry policy for remote calls is a welcome improvement to any system. Still, it has the potential of obscuring the visibility into looming problems, such as masking a steady worsening of response times until they finally exceed the maximum limits, leading me to the next point.

Telemetry, observability, and distributed tracing: There is more to handling outages than looking at system status and log entries. There is also more to operations than handling outages, such as proactively looking at the system’s telemetry of internal metrics, traces, and log entries.

Many platforms already generate a wealth of telemetry data with minimum instrumentation in the source code. However, one still needs quite a bit of telemetry-specific code mixed with the source code, especially for metrics.

Each system component may not tell you whether it is meeting its goals. Making the code generate metrics takes extra effort, aggregating them into an actionable dashboard for the operations team.

Also, ensure you (and other developers) have regular access to a local setup of the telemetry framework used in the system’s operations. Many frameworks support local execution in your workstation or a free trial cloud-based account. A uniform setup that allows a seamless transition from local to remote environments takes development, validation, documentation, and upkeep. Creating such environments can be a lot of fun, but it costs time and money too.

I can’t emphasize enough how even the most experienced and self-confident developers always learn something new or surprising when looking at the telemetry of their code.

Taking it to the next level, work with your operations team to ensure a data-sharing arrangement with the operations team, dedicating special attention and effort to aspects such as data anonymization and access.

Beware of queueing systems. Writing queuing systems is fun, and designing the architectural diagrams can be exciting (and enticing), but most people seriously misjudge the cost of adding queuing patterns into a system.

I know there are legitimate use cases for queueing systems, like high-volume transactions — as in millions of messages per day — where the calling component cannot wait for the response and only cares that the transaction eventually gets processed within a reasonable time.

However, if you are not dealing with those use cases, you may want to seriously reconsider the inclusion of asynchronous message processing into the system. That communication pattern adds costs that stretch across the entire length of a business transaction between a message producer “A” and a message consumer “B.” Here are just a few examples of that extra complexity:

Operational procedures for system administrators to deal with queue sizes exceeding certain limits
Extra design and code to deal with expiring messages.
Operational procedures for handling messages sent to dead-letter queues after expiration.
Extending the system to manage dead-letter queues
Handling distributed tracing for business transactions spanning over a message delivery.

Lesson 3 — Thread the quad: code, build, support, operate

While a long career may gradually teach you about different areas of a DevOps practice, you can expedite your growth by intentionally rotating across development, integration, support, and operations. The idea is to learn how things work and how to build software that works well in each significant area of the engineering cycle.

And once you get to know the people and workflows in those areas, it becomes easier to contribute outside your core expertise with less technical and social friction.

Mid-career and senior developers may not be as inclined to change job roles, but their experience allows them to learn faster. A temporary rotation of a few weeks, a shared project, or even access to customer support tickets and incident reports may work just as well.

A DevOps practice encourages more access and insight into different areas. Take a step further and rotate into those areas, even briefly, to understand how various software characteristics affect their work.

Case study 1 (build).

We had a team primarily responsible for evolving and maintaining the build system in a previous project. Building (compiling and packaging) the entire code base in a local workstation took about two minutes. In contrast, the same operation in the build cycle took a seemingly eternal fifteen minutes.

One of the developers outside the build team spent an afternoon adding various log entries to the build scripts to isolate the problem, narrowing the likely cause to the steps where the build wrote compiled binaries to disk. Somehow, disk write operations seemed to take orders of magnitude longer in the build system than in a local workstation (and yes, the build machine had SDD storage :-)

The build team analyzed those findings and attempted different alternatives (some of those outside the Unix skillset from the original developer,) landing on the final solution of increasing the memory allocation on the VMs and moving the temporary directories for each build to an in-memory filesystem (tmpfs.)

Case study 2 (operations).

I once co-managed the PagerDuty escalation policies and alerting rules for our entire organization — a few hundred people — for a few months. Sometimes, depending on vacation schedules and local holidays, that meant receiving urgent direct messages requesting that I enable alerting triggers in portions of the system outside my immediate scope.

I would always ask a few questions before pushing the buttons, such as:

“When will the components generating these new alerts be deployed?”
“I don’t see a playbook link in the description. Does the ops team know where to look for one?”

Without going into the occasional mix of bewildering answers, that (part-time) assignment taught me valuable lessons about the importance of managing the total number of types of alerts in a system, the cost-benefit of adding new components to a production system, and the absolute imperative of involving operations teams in architectural decisions.

Lesson #4 — Intentional learning: Learn with every task.

At this point, you know how to secure the time to write code that is ready for production and the things you need to include in that code.

That is a long list of things; you don’t want to wait decades to acquire those skills organically, the “10000-hour” rule notwithstanding.

We always learn something while performing a task, but intentional learning means going beyond getting the work done and figuring out why something works and how to improve it.

A web search may give you a precise answer to a specific problem, and one can learn a lot that way, but with intentional learning, the idea is to go beyond ready-made solutions:

Browse. If the solution involves a patterned solution for a framework, such as a specific set of resources for Terraform, go back to browse the complete definition for those resources and maybe browse adjacent resources from the same provider.
If the solution involves a utility with specific parameters, go back to the utility’s manual to study those parameters and also skim over the other parameters. The idea is not to memorize but to index them in your mind, especially in the cases such as “awk,” where the manual can be an entire book.
Explain what you learned. For source code, pull out the “rubber duck” and explain the source code to an inanimate object. For concepts, you may even skip finding someone else and use the Feynman technique, pretending you are presenting that concept to a child.
Write about the subject. Writing is a more intentional form of learning, helping you consolidate and expand your knowledge of a concept (maybe writing an article,) interrelated concepts (using something like a technical paper,) or an entire domain (for example, writing a book.) Writing goes far beyond learning so that it may be more helpful in organizing adjacent bits of knowledge in your head. If you do decide to take on this exciting activity, make sure to read “Writing for Engineers”, by Heinrich Hartmann.

“Read the product guides? Who has time for that?”

I realize this is the age of googling “how-to-make-this-error-message-go-away,” which makes a lot of sense during a time crunch. Still, you don’t learn much from those kinds of shortcuts.

Here I must quote Stephen King, one of the most successful fiction authors of all time, who offered this bit of tough-love advice for aspiring writers:

“If you don’t have time to read, you don’t have the time (or the tools) to write. Simple as that.”

If you do not carve out time from the schedule to explore the technology used in your projects and expand upon it, you restrict yourself to learning just enough to complete your immediate tasks. That dynamic becomes self-reinforcing when people around you assume you are only capable of those same tasks. And it doesn’t take long for those assumptions to become justified.

That “carving out” of time may be more or less challenging depending on different situations, but awareness is a starting point. Sometimes you may constantly be doubling down on taking on repetitive (yet valuable) tasks because it is more comfortable. At other times your organization may become comfortable with someone doing a repetitive chore more efficiently than anyone else. Regardless of the situation, recognizing that you are no longer learning something new in every task is the first step.

Conclusion

A DevOps practice covers many distinct and extensive disciplines, which leads people to specialize in a given field, such as software development, continuous delivery, or operations.

Sometimes that narrowing of interests and purpose happens before people get a chance to explore other areas, so resist the urge and pressure to specialize early in your career. As an organization, balance the productivity that comes from people possibly overstaying in a role with the benefits of a Japanese-style approach of yearly rotations — perhaps not on a fixed schedule.

Early in your career, work with your management team to “intern” in coding, delivery, and operations. That kind of rounded experience is precious and exceedingly rare. It will help you multiply your potential in any area you choose to contribute, whether as a programmer, system architect, UX designer, technical account manager, infrastructure engineer, or any other role you may favor.

The willingness to understand what makes a product better from different angles and blend those lessons into your daily routine is a superpower far beyond anything even an actual 10x developer can muster in a niche area.

Ultimately, these lessons come from personal experience, and every path is different. I welcome feedback in the comment section and promise to incorporate them appropriately into the main story.

Are you spending too much on Kubernetes?

2022-06-29T17:00:00.016-07:00

(I originally posted this article on medium.com)

Are you spending too much on Kubernetes?

Probably, yes.

Anyone running a large fleet of Kubernetes clusters knows it takes sustained investment in hardware and operations personnel to keep everything online. That level of investment keeps Kubernetes in a rarefied layer of architecture diagrams reserved for companies managing large environments, often in regulated industries that need to run them on-premises.

When the choice was Kubernetes versus Docker, the decision to embrace Kubernetes was straightforward, but several waves of new and sophisticated managed services invalidated many assumptions supporting those decisions, from cost to security to reliability.

Nowadays, many viable alternatives exist to schedule container-based workloads without running the entire cluster stack from the ground up.

It takes many different skills across different layers of abstractions to run a Kubernetes cluster.

I compiled some common themes on the most cost-effective and frictionless ways of running containers, inside a Kubernetes cluster or not, offering a landscape view of what is available to get there.

How much does a product really cost?

Before discussing whether it still makes sense for you to run entire Kubernetes clusters on your own, it is essential to break down the significant phases of developing a product.

Different people group those phases in different ways, but let’s use a generic arrangement as the backdrop for this story:

Design. End-user experience, specification of system components, and their interactions.
Development. Coding, functional testing, documentation for end-users, and documentation for system integrators. It also includes the continuous integration pipeline that supports development activities.
Deployment. Rollout development releases to production. It includes the continuous delivery pipeline.
Operations. Everything related to running the system, including hardware and personnel.
Support. Deals with product failures at the customer’s hands.

People participating in a product cycle often do not consider all parts of the business when making decisions. Product owners must continually assess the cost of an opportunity (or new feature) across all layers.

Note that different teams and organizations may co-own these stages. For instance, a customer purchasing software as a license owns most of the “deployment” and “operations” stages.

Lesson #1: Money is cheap. Time is not.

When discussing resources, we must acknowledge that money and time are intertwined.

Adding capital to any system takes good credit and a reasonable business plan. Adding more “time” resources to a system, on the other hand, can take many forms. For instance, you can hire a consultancy, hire more people, or procure tools and services that increase productivity. Paradoxically, adding more “time” capacity into a product cycle takes effort and time. In short, you cannot increase your “time” capacity as quickly as you can increase your money supply.

When making decisions that shift costs from one product stage to another, consider the nature of the shift and ask yourself:

“Will it increase capital or time expenditures?”

It is essential to realize that decisions in the early stages of the cycle disproportionately affect the later stages. Adding an off-the-shelf middleware component during the “Design” stage may lower the costs during the “Development” stage but also increase the expenses during the “Operations” stage. I covered that scenario in “Asking the wrong question: Should developers be on call?”.

Takeaway: Keep focusing on opportunities as long as you are still managing time. If you run out of capital, you can get a loan, maybe dilute ownership, and try again. On the other hand, when you run out of people, it is the end of the line.

Lesson #2: 90% of nothing is nothing

People specialized in one stage of the product cycle generally lack awareness of what matters most in other areas and tend to focus almost exclusively on improvements in their field of expertise.

Improvements to the system architecture may sound great, such as shifting from fixed pod allocation in a Kubernetes cluster to dynamic container scheduling using a managed container service. Many of these improvements are indeed great, but we must always pause for a moment to consider the impact on subsequent stages of the product lifecycle.

For instance, if your product is heavy on R&D, such as a medical decision platform, and the IaaS bill represents only 5% of the overall budget, even a 50% reduction in those costs will not make a big difference, especially when you factor in the investment required to achieve those savings.

In that scenario, optimization efforts should start with the “Development” stage, not with the “Operations” stage. While it may be difficult for development infrastructure-minded folk like us to accept it, there are cases where something like an improved spreadsheet template can do more good than revamping parts of the system architecture. The incentives that keep technical people from making that kind of trade-off deserve their own story.

Takeaway: Kubernetes adoption is a means to an end (running workloads,) not the only way, even for heavy adopters. On any given business, many vital processes still run elsewhere, are more critical, and consume more resources than a fleet of Kubernetes clusters.

Lesson #3: Consider a managed Kubernetes service

Kubernetes is great at further virtualizing a data center into a well-established set of portable APIs, which significantly simplifies the ownership of static resources, but you still own everything. When a storage disk fails, the corresponding persistence volumes in the cluster also fail, and you need people who understand both to diagnose and fix the problems in each layer.

To stress the point, I was writing a Terraform template just the other day to create all the IaaS resources required to install a Kubernetes cluster. The final count was nearly one hundred resources, covering DNS records, subnets, security groups, load balancers, network gateways, and many others. The template became so extensive that it now requires a dedicated CI process to validate changes before applying them to the production system.

With a managed Kubernetes service, you no longer own the underlying infrastructure resources, such as VPC (virtual private clouds,) virtual machines, disks, and networking. That transition shifts administrative costs (less time chasing infrastructure resources) to runtime costs (the service provider charges you a premium to operate all the infrastructure.)

With a managed Kubernetes service, the underlying infrastructure and parts of the control plane are locked down, and kept in a read-only state for anyone outside the service provider. Your company still has access to the cluster, including many parts of the control plane, such as being able to add storage classes and daemon sets.

The deciding factor here is the scale of deployment. Setting up CI/CD pipelines to validate new versions of clusters and skilling up an entire shift of on-call engineers (12–18 people) make no financial sense when you only need a handful of clusters. Once you reach ten or more, it may be cost-effective to staff an entire Kubernetes operations team and develop that practice in-house.

Takeaway: Managed clusters offer a different balance between cost and ownership, making even well-established Kubernetes shops consider their options.

Lesson #4 — Stateless first, serverless second

It is tempting to analyze the workloads in a Kubernetes cluster and stare longingly at shifting everything to a container service in the same IaaS.

Using a managed container service may be a perfect fit for stateless and short-lived workloads, such as a micro-service that translates a page of text from one language to another. Still, if you consulted with your operations team, they would probably tell you that stateful workloads are their most significant source of worry and where they spend most of their engineering efforts. These are your database and messaging clusters, with their intricate procedures for storage, backups, archiving, upgrades, and so on.

Running stateful dependencies inside a cluster is significantly more complex than running stateless workloads. Things like backup, archival, disaster recovery, and upgrades require specialized skills, detailed planning, and continuous upkeep.

Recalling lesson #2 (“90% of nothing is nothing”,) dealing with stateless workloads first may not result in a meaningful reduction in the overall budget of the “Operations” cycle of the product, so consider dealing with stateful workloads first.

Those workloads do not fit a serverless model well, as the compute aspect (CPUs) sits inside the pod, coupled with mounted storage volumes. In that case, the starting point is to replace those workloads with a service in the IaaS, such as using managed instances of PostgreSQL or Kafka.

Note that such shifts still leave residual operational costs behind, such as the need to monitor the health of the service and have procedures in place to track eventual problems to failures in the service. Still, those activities require fewer people and do not require in-depth knowledge of the service runtimes.

Takeaway: Assess how much of your overall product cost is spent on each service in your data center, from design to support. Then assess whether it could cost less (again, in terms of total cost) to lease the services from a specialized company.

Lesson 5 — Virtual Kubelets

Although not as pervasive across all cloud providers (see examples here and here,) this type of service is an excellent addition to a managed Kubernetes service. Virtual Kubelets extend the cluster scheduling service to run select containers elsewhere in the IaaS instead of inside a cluster node.

That is an excellent solution for scenarios where you are not ready to rearchitect the entire system into a combination of services and still need to keep Kubernetes clusters around for a while, with the added advantage of keeping the number of fixed worker nodes as low as possible.

Since the approach reuses Kubernetes development and deployment APIs, the costs of transitioning to a serverless model are minimal.

With a virtual kubelet, someone creates a new workload in a managed cluster. The service provider uses additional user configuration to decide whether to schedule the pod inside the cluster or elsewhere in the service provider cloud.

Takeaway: If you already leaped into using managed Kubernetes clusters, isolate workloads that don’t run frequently and don’t need to run inside the cluster. Let the service provider schedule those pods outside the cluster, then deallocate whatever fixed cluster capacity you had set aside for them.

Lesson #6: Auto-scale all things

If you got this far, you have workloads that cannot be sensibly moved out of a Kubernetes cluster. The only thing left is eliminating wasteful capacity overprovisioning within the cluster.

For instance, if your cluster has ten worker nodes and they are all above 90% resource utilization (CPU, memory, or disk), the temporary loss of a worker node means the cluster can no longer schedule all pods.

When you look closer into that utilization, the waste becomes more apparent, as you realize that a sizeable slice is allocated to container resource requests, whether or not a container requesting the resources is using them.

And remember what I mentioned about the total cost of operations? Unschedulable pods will likely cause problems in other systems and prompt several people to deal with the situation, from customers contacting support to automated monitors paging people to look into the issue.

I covered auto-scaling options in this story, but in a nutshell, you want to consider:

HPA (Horizontal Pod Autoscaling): Adapts the number of replicas for a workload based on resource utilization in the workload containers
VPA (vertical pod autoscaling): Modifies container sizes in scenarios where one cannot modify the number of container replicas.
KPA (Knative pod autoscaling:) Schedules containers based on request-per-second targets, with the option to scale a deployment down to zero if there are no pending requests for a while.
Cluster autoscaling: Scales pools of worker nodes according to the resource requests and utilization.

Pod autoscaling technology automatically adjusts capacity to match workloads. When paired with cluster-autoscaling, they allow the operations team to run containers with fewer nodes. The effects are more noticeable on the IaaS bill as a marginal change in the number of cluster nodes has little impact on the effort required to manage the cluster.

Takeaway: Autoscaling is not for everyone; it takes work to get right. It is an option for large-scale deployments where the savings in runtime costs offset the investment in deploying and tuning the autoscalers.

Conclusion

Kubernetes is a noticeable portion of an IT infrastructure, and optimizing resource utilization may be a tempting starting point for cost reductions.

Please resist the temptation to start at the most visible target and take the time to understand the total cost of operations for a product and its distribution across the multiple stages of the product lifecycle.

Teams with design and development responsibilities must grasp how other stages work and make every decision with the overall cost to the project in mind.

Shifting entire system layers to IaaS vendors can be a sensible choice for small operations teams, where it may not be possible to staff 24x7 on-call shifts of people skilled across all layers of the stack. Adopting managed Kubernetes over self-hosted clusters is a good starting point, while stateful workloads make for an excellent second step.

GitOps repos: Five lessons on what to include and what to skip

2022-05-24T14:36:00.007-07:00

(I originally posted this article on medium.com)

This article provides a decision framework to help practitioners choose what to manage with GitOps and what to avoid. It is a companion to my previous article on structuring GitOps repositories.

With the advent of Infrastructure as Code (IaC) frameworks like Terraform and Pulumi and their ability to build entire environments from nothing but a well-funded cloud account, the scope of what you can manage with IaC-based approaches has virtually no limits.

Even companies running on-premise hardware can benefit from an extensive library of plugins listed in the Terraform and Pulumi registries. And if a plugin is not available, people can author new plugins using languages and toolkits like HCL, Pulumi’s SDK, and Terraform’s CDK.

Without technology barriers, it is easy to start bringing everything into GitOps repositories and wonder whether some of that data may be better managed elsewhere.

I learned the lessons in the following sections the harder way while establishing our practices. I hope they can help make your decisions a little easier.

Lesson #1: Configuration formats matter

Part of the decision about which configuration to manage with GitOps is the choice of the IaC framework, which should prioritize alignment with the system abstractions.

As an example, I mentioned Terraform in the introduction. Its data structures and syntax are good at mapping IaaS resources such as VPCs, VMs, and network gateways. As you move up the complexity ladder, they may not be adequate to describe the internal configuration of other components, such as Kubernetes clusters.

Terraform’s Kubernetes provider can do the job, but the extra layer of abstraction used to represent cluster configuration makes it a challenging medium for anything beyond a couple of resources.

Once the number of configuration objects increases, I recommend incorporating domain-specific frameworks like Argo CD and Flux into the IaC mix. Those frameworks allow cluster administrators to work with GitOps repositories containing Kubernetes resource definitions.

It may be impractical to use a single IaC framework in a large environment due to the specialized needs of complex components. For instance, Terraform excels at allocating resources at the IaaS level, while Argo CD excels at managing Kubernetes configuration.

Takeaway: As the system grows and your IaC starts demanding awkward mappings for new configuration formats, consider incorporating new frameworks that match the way people work.

Lesson #2: Favor behavior before state

Modern infrastructure can be mind-bogglingly extensive, making the complete representation of their desired state virtually impossible. We often need to settle for things that act the same instead of obsessing over making them precisely the same.

For instance, assume a VM running Ubuntu. Only a couple of its configuration files may require modification post-installation for the VM to perform its role in the system.

In that scenario, it is more reasonable to represent the differences between the VM’s desired configuration and their default values in the image template. To make the picture worse, and for a bit of real-world GitOps heresy, a few configuration files may be opaque boxes requiring imperative approaches like Terraform’s remote-exec or Ansible to complete the job.

Take that example further and assume your production system depends on a managed Kubernetes offering like ROSA. Your GitOps repository may specify it needs one of such clusters, and a Terraform plugin will be happy to oblige. However, ROSA’s SRE team retains control of many low-level resources underpinning the cluster, such as VMs, disks, and networking.

GitOps principles call for declarative and immutable settings. Still, sometimes you need to bend those principles and store configuration relative to baselines that may be entirely or partially outside your control.

In short, we must grudgingly accept that we are often aiming for similar behavior rather than for the desired state.

Lesson #3: Leave out state co-managed by specialized components

The first type of “state” matching this lesson is application data. It is managed inside specialized servers with their own APIs and tooling. Now, let’s broaden the concept from “application data” to “data that has its dedicated system of record, lifecycle, and workflow.”

The distinction is helpful because sometimes, you may need to share configuration management with other components outside the GitOps framework.

Some of the best examples of co-management are the various fields used to control the number of replicas for pods and worker nodes in an autoscaled environment. In those situations, you want most of the workload definition in the GitOps repository while deferring things like replica counts and resource limits to autoscaling components in the cluster. Note you still want the configuration for the autoscaling components managed with GitOps.

Automated processes and components may be better positioned to configure or tune details of some aspects of the infrastructure.

Kubernetes addresses that overlap with the “server-side apply” pattern, allowing different components to manage (or co-manage) certain portions of a resource. Argo CD supports that pattern through resource annotations, telling it to ignore entire branches of a resource or specific managed fields during drift detection.

Lesson #4: Avoid secrets and certificates

We often see articles and tutorials extolling the virtues of storing encrypted secrets inside Git repositories, a technique commonly known as “sealed secrets.” Since secrets (and certificates) have their own system of record, lifecycle, and workflows, the previous lesson of avoiding that kind of data in a Git repository still applies.

If you need extra convincing, I wrote an extended version of my reasons for not recommending the practice of using sealed secrets with GitOps.

Lesson #5: Treat the GitOps CI/CD pipeline as code too

I earlier wrote about how it is possible to bring the desired state of the entire system into Git repositories. This section extends the definition of “system” to the GitOps repositories and pipelines underpinning those deployments.

I know this may sound overly abstract, but the increased consistency leads to more productivity and better security.

Pipeline as Code technologies such as Argo Workflows, Tekton, and GitHub Actions are good choices for applying GitOps principles to pipelines.

I particularly like this article from Alexandre Couëdelo showing how one can use Terraform to configure the very Git repositories holding the infrastructure definitions. That kind of approach ensures remote repositories remain configured correctly without risking loopholes that could compromise the entire CI/CD pipeline.

GitOps can also manage the artifacts behind the primary GitOps process, supporting versioned and consistent settings for CI/CD pipelines.

Conclusion

Define the boundaries of your entire system early on and decide on a phased approach to bring configuration into one or more GitOps repositories. Some components will be whole systems within the system, requiring multiple GitOps frameworks to cover everything.

The internal state of some components may be too extensive and dynamic for the complete representation in a Git repository, requiring adaptation of GitOps principles. Accept you will often be aiming at system behavior rather than system state.

Do not store any data already managed in a specialized system of record, such as application data and secrets. Watch for resources co-managed using a server-side apply pattern, then configure your GitOps framework to ignore co-managed fields during drift detection.

And lastly, use GitOps to manage GitOps, adopting Pipeline as Code approaches that extend the benefits of GitOps to your GitOps CI/CD pipelines.

Infinite scaling with containers and Kubernetes: The total perspective vortex

2022-05-10T15:01:00.002-07:00

(I originally posted this article on medium.com)

Containerizing the best code in the world means nothing if that container does not have the resources to do its job.

The idea for this story started while writing about effective container probes. It became clear that giving precise answers about readiness and liveness to the kubelet was only the first part of the problem.

The second part is the subject of this article: allocating resources to containers. Containers started with insufficient resources perform poorly and, in extreme cases, risk termination due to successive timeouts in their liveness probe. On the other hand, containers started with unnecessary resources are one of the fastest ways of converting your budget into heat.

This article draws a cloud landscape of resource allocation, from a single container in a Kubernetes pod to near-infinite capacity in the cloud, so it is time to buckle up!

The scaling of container-based workloads evolved rapidly in the past few years, making concerns about resource capacity a thing of the past in system design.

Kubernetes resource management: a recap

“How to ensure containers always have the resources they need?”

Consider this section a recap for those just starting with Kubernetes resource management. The official documentation on Kubernetes resource management for containers and namespace limit ranges is still mandatory reading in the space. If you are already familiar with the subject, skip to the next section.

Kubernetes resource management primarily covers CPU, memory, and local ephemeral storage. A container specification may define resource requests and resource limits for each resource type.

A resource request is the minimum amount of that resource required for the container runtime. The node scheduler only schedules a pod to a worker node if that node can meet the resource requests from all containers in the pod.

A resource limit is the maximum amount of resources the kubelet should allocate to a container during its lifetime. When a container has the resource limit set and lacks the resource request declaration, the latter is presumed to be identical to the resource limit.

Kubernetes allows software developers and system administrators to collaborate on setting minimum reservations and maximum limits for resources like CPU, memory, and temporary storage.

A container may have any combination of resource requests and limits, with the different combinations of CPU and memory reservation defining the quality of service for the parent pod as follows:

Every container in the pod has both resource request and limit set, and for each container, request and limit have the same value: Guaranteed.
Resource requests or limits not set for any container in the pod: BestEffort.
All other pods that are not in the two previous buckets: Burstable.

“Guaranteed” workloads have the same value for limits and requests for both CPU and memory in all containers. Requests are guaranteed to be honored if all “Guaranteed” workloads fit in the cluster. Anything above “requests” and below “limits” depends on the cluster having spare resources. “BestEffort” containers are the first to be evicted if the cluster runs out of resources.

It is helpful to look at what these categories mean when the kube-scheduler decides to schedule a pod on a worker node:

“Guaranteed” means the scheduler only assigns the pod to worker nodes with enough resources to meet the pod request.
“Burstable” QoS means the scheduler looks for a worker node with enough memory to meet the resource requests of the pod. The scheduler does not care whether the worker node can meet the resource limits of containers in the pod or anything above the resource request, for that matter.
“BestEffort” means the scheduler will meet the resource requests of every pod with a Guaranteed or Burstable QoS before scheduling the pod. The scheduler continuously revisits that assessment, so it may still mark a BestEffort pod for eviction if a “Guaranteed” or “Burstable” pod needs those same resources.

Decision time: Who should define container resources?

The decision on requests and limits starts with understanding that resource requests protect the container while resource limits protect the cluster.

The two main parties in that decision are product developers and system administrators. The challenge is spread over time, as product developers spend most of the time establishing resource boundaries in isolation from deployment to production environments.

On the flip side, system administrators often find themselves dealing with Kubernetes workload specifications with insufficient documentation on how changes in preset requests and limits affect the workloads.

Product developers have greater visibility into the needs of the workloads and their behavior under different resource allocations. In general, they should be able to identify at least the following thresholds during the months where they labored over the code base ahead of shipment:

Absolute minimum resource request. This value is the resource utilization required to load the runtime and be able to answer probe requests from the kubelet comfortably. This article shows several PromQL queries to identify troubled containers.
Maximum usable resource limit. This value is the point where the performance of the container starts to hit external limitations, such as disk I/O or bottlenecks in calls to 3rd party services.

Product developers must define the range of absolute minimum and maximum usable resources, then balance runtime safety and resource waste when choosing resource requests and limits.

Ideally, product developers should also establish and document how the container performance varies within those boundaries. That information enables system architects to plan initial capacity before deployment and system administrators to adjust resource requests in production environments.

Quotas and range: Avoiding the tragedy of the commons

System administrators have visibility into cluster capacity and how much of that capacity is still available. As someone who often wears an SRE hat, I appreciate workloads in the “Guaranteed” QoS bucket. It takes a special kind of confidence (and lots of testing) for a development organization to set fixed resource boundaries on all containers in a pod and trust that pod to work reliably within those boundaries.

“Burstable” workloads fall somewhat lower in that spectrum of trust. As an SRE, my best hope is that all containers in a burstable pod have those “absolute minimum” resource requests mentioned in the previous section.

When it comes to resource limits in a burstable workload, my opinion depends on the development philosophy behind those limits:

(good) Autoscaling within the workload (such as a Java Virtual Machine launched with minimum and maximum memory limits.) This kind of arrangement (resource limit set higher than resource request) means the development team spent time identifying the usable operational range of the workload and has a good grip on managing those resources.
(bad) Protection against eventual bugs. This design philosophy reminds me of the “tragedy of the commons,” where everyone takes a bit more of a shared resource to anticipate its eventual exhaustion, accelerating the exhaustion of the shared resource. That philosophy is particularly disastrous when the development team underestimates the resource requests. In that situation, the workload continuously operates in the burstable zone of its operational range, making it susceptible to CPU throttling, memory exhaustion, and termination by the kubelet in the worker node.

And that finally leaves us with “BestEffort” workloads, a perennial source of worries due to their unfettered ability to exhaust cluster resources and their tenuous scheduling status in the face of other pods requesting those same resources.

When dealing with these types of workloads, a system administrator must take a moment to consider their options:

Use limit ranges to set default resource requests and limits for containers and pods in a namespace. I think the blanket assignment of identical resource limits for all workloads combined with the lack of input from the product developers makes it an unrealistic proposition.
Use resource quotas to cap the combined resources for all pods in a namespace. This approach is a more promising solution. It reduces the blast radius of an eventual container going haywire without the waste of multiplying the safety buffer by the number of pods in the namespace.
Take a deep breath and hope for relief from the pod autoscaling technologies covered in the next section.

System administrators can use resource quotas and limit ranges to keep workloads from interfering but often lack information to balance runtime safety and waste.

Pod autoscaling: Beyond static sizing

Many readers are probably ready to bring up the usual workarounds for dealing with unruly workloads:

These popular autoscaling techniques can dynamically allocate capacity within a cluster based on resource metrics; HPA can alter the number of pod replicas in a deployment, while VPA can change the resource requests and limits for a pod.

While capable of automatically extending capacity, these components are not a free pass to forego the precise definition of the operational ranges for their workloads. For instance, HPA will not take action on a metric where containers in the pod do not have the resource request for that metric. HPA is also ill-suited to scenarios where a deployment relies on a specific number of replicas for the pod in the cluster.

HPA balances deployment capacity by increasing or decreasing the number of pod replicas when the resource utilization strays from the desired target value.

Turning to VPA is better for workloads that require a fixed number of replicas or where the workload does not ship with preset limits. There is, however, a relatively long list of limitations. Additionally, if using limit ranges, the considerations may get quite involved.

VPA balances deployment capacity by observing resource utilization inside the pod and recreating the pod with more adequate resource requests and limits for the containers inside the pod (resizing pods without a restart is currently an experimental feature.)

Ultimately, the effectiveness of these tools in autoscaling workloads depends on the profiling effort spent on defining resource requests and limits during development. Here, I must give VPA extra credit for its ability to “learn” the operational resource ranges for a deployment.

Auto Pilot operators: SRE in a box?

Kubernetes operators introduce the notion of custom resources to manage applications and workloads. Conceptually, an operator continuously watches the custom resource and maps it to workload resources.

The Operator Maturity Model calls for five different levels of autonomy, with the highest being level 5, or “Auto Pilot.” An operator at that stage of maturity should, amongst other kinds of autonomous behavior, apply horizontal or vertical scaling to the workloads it manages.

Operator Maturity Model calls for “Level 5” operators capable of self-tuning and autoscaling workloads.

There is an extra natural cost in developing an operator, including designing and documenting a custom resource definition. There is also the additional runtime cost of the operator pod. That is why I think the value of operators starts with level 4, or “Deep Insights,” which entails creation metrics, alerts, log processing, and workload analysis.

Below level 4, operators do little to alleviate the continuous cycle of product developers attempting to predict the future and system administrators trying to bend those predictions into reality.

Knative: Elastic capacity with zero waste

According to its website, Knative “is an Open-Source Enterprise-level solution to build Serverless and Event-Driven Applications.”

From the perspective of this article, we are interested in Knative’s Serving module. That module contains the Knative Pod Autoscaler (KPA,) which mimics HPA’s ability to vary the number of pod replicas to meet demand, with two key distinctions:

Scale replicas based on the volume of external requests
Scale replicas down to zero when there are no external requests

Knative Pod Autoscaler monitors requests to a URL and decides how many pods are required to handle the pending queue of requests. With the proper configuration and no outstanding requests, KPA removes all replicas of the pods for maximum resource savings.

Unlike HPA, which reacts to resource utilization when it crosses preset boundaries, KPA makes its decisions based on concurrency and requests-per-second targets.

As a rule-of-thumb, HPA is better suited for long-running, slow-starting processes where resource demands do not change abruptly. In contrast, KPA favors transient processes that can boot quickly to meet immediate demand.

And last but not least, serverless or not, KPA works with containers, so resource limits still matter.

Cluster autoscaling: Pushing the limits

Once you have appropriately tuned all workload resources and pod autoscalers, it is time to look deeper into the toolbox and reach for the Kubernetes cluster autoscaler.

The cluster autoscaler can add (or remove) worker nodes based on resource demands. The autoscaler respects configurable lower and upper boundaries to ensure minimum capacity and some level of cost controls to avoid unruly workloads overstretching the cluster.

Slow stretching. Remember that response times for a cluster autoscaler are decidedly sluggish compared to pod autoscalers, with a worker node taking several minutes to become available to the kube-scheduler.

Suppose you need a cluster autoscaler to work with peak demands. In that case, you must resort to deploying a sacrificial buffer of “pause pods” (described in the cluster autoscaler documentation.) When new workloads need those resources, the kube-scheduler immediately evicts the pause pods to free up resources. The cluster later scales up the number of workers to reallocate the evicted pods.

Cluster autoscaling enters the picture once pod autoscalers can no longer find enough room on cluster worker nodes.

Also, note that Kubernetes taints and tolerations also play a role in how autoscalers do their work, making tainted worker nodes off-limits for the placement of certain pods.

Serverless: Limitless capacity…with limits

For the ultimate level of autoscaling, it is time to abandon the boundaries of a cluster — even Kubernetes has limits — and look at serverless capabilities at the cloud provider level.

Major cloud providers can spin up container images on demand based on multiple events, such as web requests, database operations, and timers. Their capacity is virtually limitless, with almost negligible cost compared to running the same workload inside a cluster.

Ironically, despite an ultra-low-cost per invocation, serverless approaches make you switch from worrying about resource limits to worrying about cost overruns:

Container limits still matter: Cloud providers require you to indicate the number of CPUs and the amount of memory for each container invocation.
Budget limits: Where a bug in a test suite may exhaust cluster capacity, that same bug in a serverless deployment may erase your entire department’s budget overnight.

You may find several variants of serverless offerings, such as handing a small function to the cloud engine rather than an entire container image or letting the cloud provider use one of your managed clusters to run the containers (trading scale for security.)

For added convenience, many of these offerings can start from a Git repo containing a Dockerfile, then build and cache the container image for when the first invocations arrive.

There are no concerns about cluster capacity with serverless allocation at the cloud provider. System administrators still need to worry about individual containers having enough resources while also focusing on near-infinite runaway resource utilization.

Conclusion

We covered the entire spectrum of resource allocation for a workload, from tuning a single container to hyperscaling a workload outside Kubernetes.

The article started with a recap of Kubernetes resource management and its core concepts of resource requests and limits. We then built upon those static limits into the realm of pod autoscalers, such as HPA, VPA, and KPA.

From pod autoscalers’ ability to efficiently utilize a cluster’s maximum capacity, we moved on to increase that capacity with cluster autoscaling. We then completed the total perspective vortex with virtually unlimited resources (and costs) in the form of serverless offerings across cloud providers.

Except for VPA, these techniques still build upon reasonable effort to determine a container’s resource consumption during development activities. For example, HPA does not take action based on resources where the limits are not specified. Cloud providers outright ask for the CPU and memory resources assigned to each container run.

With the landscape of resource scaling out of the way, my next article will cover concrete techniques to identify resource requests and limits during development and keep a watchful eye on them in production.

References

Six critical blindspots while securing Argo CD

2022-04-21T09:48:00.002-07:00

(I originally posted this article on medium.com)

Lately, I have been addressing security risks in our local GitOps deployments, specifically around Argo CD* control planes.

GitOps frameworks like Argo CD run with elevated privileges relative to target deployments, making them prime targets for supply chain attacks.

Keeping systems secure is a continuous effort over time. This article shows the core strategies for securing an Argo CD deployment and keeping you ahead of potential exposures. If you are new to Argo CD, you may want to read it side-by-side with the Argo CD security manual.

GitOps control planes have elevated privileges on entire deployment environments, making them a prime target for supply chain attacks.

Applications and Projects

Before diving into risk analysis, it is helpful to briefly mention Argo CD’s basic deployment unit: Applications.

An Argo CD Application combines a source location (Git URL, revision, and folder containing resources) and a target destination (a namespace in a Kubernetes cluster.)

Authorized users add applications to an Argo CD Project. An administrator configures a project to restrict the range of sources and destinations in applications and creates access control lists of people authorized to work with the applications in the project.

In addition to these capabilities, I also like that one can issue scoped tokens from a project and use them on external CIs, much like GitHub access tokens.

An Argo CD project combines user permissions, lists of allowed resources, source repositories, and destination clusters.

Lesson #1: Use a dedicated project for the control plane

This lesson assumes that you manage the Argo CD control plane through GitOps — how else, right?

Why it matters: Without a dedicated project for the control plane, a project shared with other applications allows those applications to manage workloads inside the Argo CD namespace.

The corresponding application resource(s) for the control plane must target the location of the control plane squarely with the following fields:

spec.destination.server refers to the local cluster: https://kubernetes.default.svc
spec.destination.namespace refers to the namespace where the Argo CD instance is running.

The dedicated project should be configured with the following settings:

spec.sourceRepos[] refers to the Git repositories managed by the instance owners and only those repositories containing the control plane’s configuration.
spec.destinations[0].server refers to the local cluster: https://kubernetes.default.svc
spec.destinations[0].namespace refers to the namespace running the Argo CD instance

A dedicated project for managing the Argo CD instance allows administrators to separate applications that require elevated privileges from other applications meant to manage other resources.

No other project in the instance should have that same combination of server and namespace in its destinations field.

Note that the attack surface of the control plane extends to the Git repository hosting the control-plane resources. In that sense, Argo CD administrators must also be well-versed in managing and securing Git repositories.

Securing a GitOps repository is a topic that deserves its own future article. For now, I suggest a minimum of:

Making the repository private.
Enabling branch protection on the main development branch of the CI pipeline.
Requiring signed commits.
Requiring code reviews before merging branches.
Restricting who can push to the main development branch.
Adding automated processing to lint pull requests, labeling the pull request as sensitive if it contains changes that affect access control, such as roles and role bindings.

Lesson #2: Argo resources are for Argo admins only

Only GitOps repositories owned by Argo CD administrators should contain resources belonging to the argoproj.io API group.

Why it matters: Popular Argo tutorials contain resources like ApplicationSet.argoproj.io and Application.argoproj.io. These resources require their projects to allow the creation of other resources in the same namespace as the Argo CD instance.

Multiple users consider this lesson quite restrictive, but that is the current security model. The chain of explanations goes as follows:

Argo CD only recognizes Application resources in the namespace hosting the Argo CD instance (see discussion in this issue.)
From the previous point, a Project containing Application resources must include the namespace of the Argo CD instance in its destinations field (see this other discussion.)
Since that destinations field applies to the contents of all repositories in the project, only Argo CD admins or delegates can merge pull requests in those repositories.

You can read the detailed discussion in this issue, which led to proposals calling for further attention on both documentation and the user interface.

Beyond calling attention to this common security blind spot, there are proposals to tighten the security privileges of an Application within an instance, such as issue 7689.

Lesson #3: Delete the “default” project

Argo CD’s initial configuration has a “default” project. This project allows authorized users to add applications from any source repository and manage all resources on all destinations.

Why it matters: A malicious or poorly designed Application may contain resources that alter the configuration of the entire cluster. The service accounts used to run Argo CD are often privileged enough to oblige.

That configuration is ideal for a non-production Argo CD instance tuned for fast progress during a learning session. It is also the stuff of nightmares for an unsuspecting system administrator.

The “default” application project in Argo CD allows applications from all sources to manage resources on all destinations. It is meant solely for learning purposes in a disposable environment.

A cluster may still reject requests due to its own RBAC permissions, but removing broad permissions is an essential first line of defense for the control plane.

Note: The Argo CD UI prevents the deletion of the project, so delete the project directly using kubectl delete AppProject default -n ${argocd_namespace:?}. Alternatively, you may edit out all of the * permissions from the project.

Lesson #4: Block ClusterRoleBindings in (most) projects

The previous lessons prevent unauthorized repository owners from commandeering an Argo CD instance. This lesson is about protecting the cluster itself.

Why it matters: Applications managing ClusterRoleBindings can attach privileged cluster roles to a service account, later using that account in pods running resource hooks to execute privileged actions.

The only potential exception to this rule is when a dedicated Argo CD project is assigned to the cluster administrators.

While we all have dealt with our share of products requiring admin-level privileges during their installation, blocking these resources at the project level forces meaningful conversations between product and cluster admins.

These conversations can lead to safer arrangements, with narrower roles tied to separate service accounts and managed in git repositories owned by cluster administrators.

Lesson #5: Narrow roles on remote clusters

Argo CD offers distinct deployment topologies where an instance can control anything from the local cluster to a fleet of remote clusters.

Why it matters: If the Argo CD control plane is compromised, narrower permissions assigned to a dedicated service account in the remote cluster will limit the exposure and speed up recovery efforts.

When adding remote clusters to the list of destinations in Argo CD, I suggest spending a few minutes discussing the following points with the cluster administrators:

Determine the list of namespaces in the cluster that the Kubernetes administrators want to manage with Argo CD applications. Is it the whole cluster, or is it specific namespaces?
Ask the Kubernetes administrator to create a dedicated service account in each cluster, such as “argocd-control-plane-account.”
Ask the Kubernetes administrator to bind that service account to a role or roles) that match the specific namespaces discussed in the previous steps. Full permissions in the entire namespace may be acceptable. Still, one may consider narrower permissions depending on the scenario (for instance, not granting permissions to read secret resources.)
Ask the Kubernetes administrator to generate the “kubeconfig” file from that service account, then use that file when adding the cluster to Argo CD.
Agree on the rotation interval for the service account tokens and the process to replace them in Argo.

Cluster administrators should create dedicated service accounts with the narrowest possible roles before handing “kubeconfig” files to Argo CD administrators.

Lesson #6: Have a CVE response plan ready

Like any software component, GitOps frameworks may suffer from vulnerability and exposures, and mitigating those risks requires awareness and speed.

Why it matters: Having a response plan in place expedites the assessment of the situation and eventual recovery efforts.

ArgoCD publishes its CVEs as GitHub security advisories, making it easier to stay on top of new developments. It is also good to familiarize yourself with Argo CD issues labeled with the “security” tag to stay ahead of new developments and help shape their progress with comments and suggestions.

As a consumer of these frameworks, there is no fixed recipe for responding to such advisories. The best strategy is to read through sections related to workarounds and mitigations and implement those recommendations as quickly as possible.

Once a CVE is announced, administrators should be ready to react quickly with patches and mitigations.

Patching a CVE is not sufficient. The next step is to determine whether someone may have exploited the CVE. The assessment, at a minimum, should involve these steps:

Inspect Argo-generated audit events and logs of calls related to applications used to manage the control plane.
Revisit all roles and role bindings in the cluster hosting the control plane.

Summary

Argo CD’s out-of-the-box permissions are tuned for a quick on-ramp into learning exercises. They are unsuited for production environments and compounded by many early adopters not realizing the security implications of using the popular “App of Apps” pattern.

Use separate Argo instances for individual teams, create a dedicated project for managing Argo, and remember to delete the “default” project.

When separate teams manage remote clusters, discuss service accounts’ roles and token rotation policies with the Kubernetes administrators before registering the clusters as destinations in Argo.

And finally, be prepared to monitor and react to Argo vulnerabilities with validated plans for mitigating the risks outlined in a CVE and post-incident assessments about eventual exploits in your systems.

References

Kustomize and GitOps: The Good, the Bad, and the Ugly

2022-03-31T20:30:00.013-07:00

(I originally posted this article on medium.com)

Teams adopting GitOps invariably face a handful of pivotal decisions in their path, such as the choice of GitOps framework and how to design repositories to match their deployment workflows.

In a Kubernetes-based shop, that framework is probably either ArgoCD or FluxCD, which are adept at detecting and correcting configuration drift between the repositories and the target clusters.

A GitOps framework’s primary job is to detect “drift” between the desired configuration in a Git repo and the actual settings in a deployment.

What you see is almost what you get.

While these frameworks can apply the literal contents of a Git repository to a target Kubernetes cluster, those contents are likely to need adjustments to fit most common scenarios.

For example, assume a system with five production environments where the only difference in the configuration for each cluster is the number of worker nodes. It would be challenging to maintain one complete copy of the folder per cluster over time, with authors mindlessly replicating changes and pull request reviewers droning over duplicate content.

Copying-pasting folder contents may be the fastest way to bootstrap a new environment but will make maintenance more challenging over time.

These Kubernetes-based GitOps frameworks offer internal configuration management pipelines as a common strategy to avoid these tedious and error-prone repetitions. The pipelines transform the repository’s contents before applying the results to the destination — while Flux CD uses the term “pipelines,” Argo CD refers to these transformations as “build environments.”

The primary choices for transformation pipelines are Kustomize and Helm. ArgoCD also supports the less popular JSONNet framework and the concept of custom configuration management plugins (CMP.) JSONNet, with its object-oriented approach, is peculiar enough to deserve a separate closer look, while I have a word or two about CMPs towards the end.

How does Kustomize work?

Kustomize uses a configuration file named kustomization.yaml to transform the contents of a directory or GitHub URL. In its simplest form, a kustomization.yaml file has the list of files from the source directory. Running the kustomize command-line interface with that file against a local folder or remote Git URL yields the contents of the files in that list.

Kustomize uses a kustomization.yaml file and a base folder as sources. The kustomize CLI uses the kustomization.yaml file to decide which files to process and how to modify them.

Kustomize’s capabilities grow from there, with features such as adding a set of annotations to all resources, generating new ConfigMaps and Secrets based on a seed file, modifying select portions of resources, and a few others.

Kustomize meets most imaginable configuration scenarios, as long as you are willing to put effort into specific areas. The question addressed in later sections is whether that effort pays off during regular interactions with the repository, such as authoring and reviewing pull requests.

Helm charts, the basics

Helm charts offer a higher-level abstraction than Kustomize, centered around the concept of “Charts.” At the core, a chart is a combination of files in a folder named “templates” and a “values.yaml” file. The “templates” folder contains Kubernetes resources, just like in Kustomize, but Helm can substitute optional templating statements inside each file with contents from various sources.

The templating statements can be as simple as a reference to a variable declared in the “values.yaml” file but can grow in complexity with flow controls, built-in functions, and named templates.

Helm transformations revolve around a folder named “templates” with resources and a file named values.yaml. Helm looks for templating statements and replaces them with values such as variables and contents from other resources.

After covering the basics of Kustomize and Helm, it is time to explore the Good, the Bad, and the Ugly of each approach in the context of a GitOps practice.

The Good #1: Kustomize as a library cart.

If your organization has a curated library of configuration resources, Kustomize is an ideal companion to select elements from that library. Add only the filenames of the resources you need into a kustomization.yaml file, indicate the Git URL (and “ref” !) of the folders containing the resources, and the Kustomize pipeline inside Argo CD or Flux CD will apply those resources to the target system.

Although Helm can achieve the same result of applying select resources from an extensive repository, both Argo and Flux require the creation of a wrapper resource for each Chart.

The Good #2: What you see (in the files) is what you get.

One of the core premises of a GitOps practice is to manage system configuration using a Git repository. While it is virtually impossible to represent all configuration levers for a Kubernetes cluster, let alone for the entire infrastructure, Kustomize’s design relies solely on configuration files as the input to manage configuration.

As evidence of Kustomize’s opinionated approach to data sources, the only way to specify the value for a variable mentioned in a file is to provide another file with that value.

No ambiguity. To take its opinionated approach a step further, Kustomize requires that repository owners be precise about the “address” of the change, specifying the type of resource and the respective location of patches and replacements. That specificity virtually eliminates ambiguity and the potential for a change accidentally causing unexpected side-effects elsewhere in the results of a kustomize transformation.

As a result, one can be confident about the resulting configuration after applying Kustomize to the contents of a GitOps repository by looking solely at the repository and without consulting any other source.

Kustomize’s design calls for a deterministic selection of resources based on a precise selection of resource types, names, and field locations.

Contrast that certainty with Helm’s approach to templating, where the invoker of the Helm command-line interface can override any variable in the values.yaml file with a command-line parameter. Helm propagates that value to all occurrences of the variable name anywhere in the Chart, with the semantics of simple text replacement and not a hint of type awareness.

While convenient, that flexibility means a pull request reviewer may feel inclined to reject a change if they cannot tell whether a variable value in the Chart may compromise the system, such as a variable name representing the role reference in a ClusterRoleBinding resource.

The Bad: Mixing filesystems with Kubernetes concepts

While I appreciate the simplicity of a file-based approach to select resources, there is some conceptual back-and-forth between filesystems and Kubernetes CRDs when reading a Kustomize-based repository.

Surely, we can expect a GitOps practitioner to know their way around both concepts, but that kind of context-switching reduces productivity for pull request reviewers.

Is it a file or a resource? For example, the entry-level construct “resources” locates resources by filename. At the same time, an element such as “patches” follows the conceptual model of Kubernetes patches, with meta-references to fields in a custom resource definition. Sitting in between filesystems and patches, we find elements such as “replacements,” combining some addressability concepts from patching and what looks like JSON selectors.

A kustomization.yaml file contains a mixture of concepts, with some constructs based on filenames (“resources”) and other constructs based on Go-templating (“replacements”) and a few based on Kubernetes custom resources (some forms of “patches.”)

I know Kustomize advocates may become proficient at the context-switching and consider it just a matter of course when working with configuration management in Kubernetes. Naturally, I would not expect Kustomize to use a constrained file-based approach like the “sed” syntax to modify a file. Still, it is hard to gloss over how reading a Kustomize file calls for the somewhat challenging ability to assemble these different concepts in one’s head, especially when I look at the most common workarounds:

“You could run the kustomize CLI to see the results instead of trying to read the source.”

At that point, I am already going outside the GitOps tooling to work around the problem, cloning the branch for pull requests and figuring out the correct parameters to invoke kustomize.

“A CI/CD action could run the kustomize CLI and show the differences in the context of the pull request review.”

A bot showing the outcome of the kustomize output between the source and target branches, even when integrated into something like the “Checks” tab of a pull request in GitHub, does not tell me the source files and lines where the changes originated. At that point, that is a review of a deployment plan instead of a pull request review.

The Ugly #1: Dealing with variable replacements

The same opinionated philosophy that makes Kustomize-based repositories self-contained and prescriptive during operations makes Kustomize brittle when dealing with everyday situations.

Replace with “replacements.” With “vars” being deprecated, one must pause to take in the complexity of its functional replacement, which is coincidentally and unironically named “replacements.”

I appreciate how a block of code like the one below leaves no margin of doubt about what it will do, but it is hard to overlook the many keywords and metadata references spent on addressing the one field to be replaced.

Kustomize sometimes makes simple tasks, like replacing a variable inside a resource, labor-intensive and difficult to maintain.

The Ugly #2: File names are (almost) forever

The precision with which Kustomize addresses file locations in a repository is its undoing when the time comes for refactoring files and folders.

If you have an overlay folder containing a kustomization.yaml file, which includes a “resources” section referencing a few files in a base folder, any change to a filename in that base folder may break the overlay folder.

Overlay 3rd-party bases at your peril. Addressing filename changes is simpler when you own both the overlay and base folders. You presumably have complete control over making changes to both folders simultaneously during a refactoring cycle.

Refactoring resources across folders and files is a brittle process, where developers must exercise extreme care in matching every name change with the contents of other files.

A less common and dangerous scenario is when the refactoring of a base folder splits the contents of a file into multiple files while preserving the original file’s name. For instance, someone may have placed an extraneous “ClusterRoleBinding” in a file named “roles.yaml”, then versioned the repository, and then refactored that role binding resource into a dedicated “bindings.yaml” file on the next release.

That is the kind of change that can easily go unnoticed in a “kustomize” invocation (the “resources” section has the name of a file that still exists,) requiring exquisite levels of attention when updating the reference to a newer version of the “base” folder.

The Ugly #3: The magic hybrid

Argo CD’s custom configuration management plugin” allows an admin to provide Argo custom code to run the transformation pipeline. One of the samples offered in the guides is a blend of Kustomize and Helm.

At first glance, the idea makes sense: If Kustomize and Helms have strengths and weaknesses, combine both, right? Not really.

Combining both means practitioners have to design (and observe) the boundary where Kustomize ends, and Helm starts. For instance, the design may call for the kustomization.yaml file only allowing the usage of the “resources” clause, deferring everything else to Helm. But what stops a new team member, unaware of that boundary, from using a “replacements” or “patch” annotation to avoid creating another Helm chart?

A custom configuration management plugin can run different commands in a sequence. With great flexibility comes the great need to design, document, and enforce bespoke workflows.

Between the overhead of working with two overlapping technologies, constantly toeing the imaginary line between the two, and having to correct eventual boundary crosses, the total cost of ownership for the hybrid approach offsets any productivity gain in leveraging Kustomize in small corners of the repository.

There may still be good cases for using those plugins, but making Kustomize and Helm work together isn’t one of them.

Conclusion

Kustomize offers advantages when dealing with file selection scenarios, but resource contents invariably need alterations. With Kustomize, those alterations mean reviewers of pull requests staring at stretches of “patches” and “replacements” blocks, ultimately trying to run small Kustomize pipelines in their heads.

Pull request reviewers may often ponder their career choices while deciphering the meaning of “JSON 6902”, and authors may find themselves wondering whether a custom resource definition tolerates the patch they would like to apply to a resource.

For a discipline like GitOps, which requires frequent adjustments to large folder structures before an agent of some sort can deploy the results to a target cluster, I think Kustomize requires disproportionately greater attention and effort than working with Helm charts.

While I can enjoy the occasional mental gymnastic involved in reconstructing a Helm chart as a Kustomize-based file structure, I remain hard-pressed to pick Kustomize over Helm for all but the most limited use cases in a busy GitOps practice.

References

Effective CI/CD strategies for GitOps

2022-03-16T20:30:00.003-07:00

(I originally posted this article on medium.com)

With Git at the heart of GitOps, it is natural to consider adopting the core CI/CD concepts of continuous integration, delivery, and deployment to validate changes to a GitOps repository.

CI/CD for GitOps still needs a strong focus on automation for validation and rollout steps, but dealing with infrastructure significantly affects the dynamics of these steps to the point where a few revisions are in order. The following sections build upon the lessons from my previous article on GitOps repository design and highlight the central adaptations for GitOps CI/CD pipelines.

Efficient CI/CD processes requests independently; developers only worry about submitting pull requests.

Lesson #1: The product is a blueprint

This lesson affects the continuous integration portion of a CI/CD pipeline. Continuous integration aims at merging frequent code iterations to the main development branch. The foundation of continuous integration is the concept of small, independent code changes validated by comprehensive test automation.

If pull requests accumulate behind sluggish validation sequences, feature branches begin to overlap in a cacophony of merge conflicts, requiring extra validation cycles.

The challenge with CI/CD for a GitOps repository is that there is no self-contained binary or package to be transferred over to a runtime environment, unlike applications. The repository has configuration files and instructions for a third-party agent to effect changes in the final medium: the infrastructure.

If CI/CD for applications produces (virtual) bricks, CI/CD for GitOps makes blueprints.

A pull request is more than its contents: it is a set of instructions for a remote agent responsible for acting on the infrastructure.

Applications may be sophisticated bricks that react differently to the environment where they sit — so do real bricks — but a downloaded binary is structurally identical to any other downloaded binary.

Now, contrast the certainty about the composition of that virtual brick with the confidence you may have in what you get from a folder in a GitOps repository. For simplicity, let’s assume a GitOps repository containing a few folders representing a fleet of Kubernetes clusters, with a “common” folder containing shared settings across all clusters and a folder for each cluster containing only the settings that differ from the shared settings.

When someone modifies a shared setting in that “common” folder, that also changes the effective desired state for all clusters. It is not like one can validate the full extent of that change short of actually effecting the change on all these clusters, so we must settle for validating the new blueprint in resources as similar as possible to those in a production environment, which is covered in lesson #4.

Lesson#2: Small, independent changes

Small, independent changes are an often overlooked aspect of continuous integration. It is easier to break this rule while doing application development, especially when the test automation achieves a certain level of functional coverage.

With GitOps, extensive changes to the repository mean multiple parts of the infrastructure changing at once, possibly asynchronously, which tends to surprise people with avoidable outages. No one likes the words “surprise” or “outage” next to a production deployment, so it is a good idea to formalize the practice of making small changes as part of the CI/CD pipeline and avoid accidentally terraforming a deployment into a whole new world.

Flag extensive changes. All major Git providers support APIs to add various “status” messages to a Git commit, so this is the perfect place to validate whether a pull request is “small.” For example, if the repository design calls for a parent folder for each class of devices, the automation may flag pull requests containing changes to more than one class of devices.

Large pull requests can exceed the unwritten pipeline specifications, resulting in “terraforming”-like reconciliation with the target infrastructure.

Lesson #3: When small is big

Changing a few lines on a blueprint may have an outsized impact on the final product in the real world. Likewise, a few characters in a GitOps repository may trigger massive shifts in the infrastructure, so we need to extend the concept of “small, independent changes” to the impact of a change.

Continuing the previous example — a change to a folder containing shared settings for individual resources — imagine you have a compatibility-threatening configuration change request, such as upgrading Kubernetes clusters to the recent 1.22 release. This release removed support for several long-standing beta APIs, and adopters received extensive warnings and education for several months before the release about the potential disruptions to existing workloads.

Gradual rollouts. While the pull request with the version change to the “common” folder is technically “small,” it probably deserves a more deliberate rollout approach, updating only a few clusters in the earlier stages of the pipeline.

The exact strategy for “deliberate” may vary. A simple approach may entail an initial commit to a couple of individual folders mapped to clusters in the “development” stage. It is best to look into tooling specialized in rollout strategies for larger deployments and increased productivity, such as blue-green, canaries, and feature flags.

Pull requests small in size may contain subtle changes that have an outsized impact on the infrastructure.

Lesson #4: Unit testing is integration testing

One of the cornerstones of continuous delivery is an extensive battery of automated tests. Good test automation covers the range of unit, integration, and system tests. With GitOps, unit testing becomes a little murkier because the resource contents are often proxies to infrastructure components. The resources also lean towards declarative approaches using “mini languages,” such as Terraform files and even the occasional DDL file for relational databases.

Testing “mini languages” in pull requests is a common dilemma in unit testing because the statements are entangled with specialized backend servers. Think relational databases, messaging servers, and other types of stateful layers. Attempting to simulate the work of these dependencies is often a losing and expensive proposition, so application developers tend to rely on the usage of mock objects and defer the vestigial gap in test coverage to later stages of the CI/CD pipeline.

When it comes to GitOps, that gap coverage gap is much broader, even if you are cleverly mimicking the technique of using mock objects with something like cdk8s.

Closing that gap in the earliest pipeline stage is a challenging balancing act between speed and coverage. Leave too much testing out of the pull request validation, and problems escape long enough to require rework to most issues; try and validate all scenarios, and the pull requests start to pile up and collide.

Reconciliation is for speed. The effects of merging a pull request on the infrastructure tend to fall along the CRUD axis of creation, updates, and deletions.

In order of speed, from faster to slower, we typically see deletion, update, and creation.

When creating new resources starts to take disproportionally longer than the other operations, it makes sense to favor reconciliation tests against a standing environment, at least in the earlier stages of the pipeline.

Standing environments may also bring a financial cost to the balance between speed and coverage, a common gold-plated blindspot for many technical people. I am not saying you should be counting the dollars before issuing a pull request, but whoever designs the CI/CD pipeline for a GitOps practice absolutely should be making those calculations side-by-side with the technical decisions.

Applying pull requests to existing environments is often faster than creating entire environments from scratch.

The safety of no reconciliation. In this approach, any change to the repository means deleting and recreating all resources corresponding to that change.

The approach has some merit, as it cuts down test scenarios considerably and ensures a more consistent overall system state due to components effectively resetting their internal state every time their configuration changes.

The drawback is that as the system scale increases to thousands of resources, one must have unwavering trust in the entire subsystem responsible for recreating thousands of resources quickly. One must also be willing to become intimately familiar with the underbelly of large-scale provisioning in most IaaS providers: API rate limits, network traffic jams due to terabytes of images floating around the private network, bootstrap bugs that only rear their heads when the network is congested, and many others.

Lesson #5: There is no rolling back

This lesson assumes the GitOps repository is versioned with branches or tags, as outlined in the previous article in this series.

In a multi-staged CI/CD pipeline, you have parallel environments such as “development,” “test,” and “production.” A pull request may pass validation in the early stages of the pipeline and still fail in production.

When a production environment has a problem, it is natural to expect people to call for a rollback, especially from outside the DevOps team. Rolling back makes a lot of sense in the abstract: if the changes broke things, then bring back the old configuration.

In the real world, however, a GitOps repository contains a mixture of imperative (scripted) and declarative statements, which means a previous commit does not ensure the system state will revert to what it was before the attempt to move forward. Still, even if referencing that old commit somehow brings the system back to life, you now have a disrupted pipeline flow and two unsavory choices:

Create a new patch. Submit a change request for whatever afflicted the production system and roll it through the pipeline. All earlier stages will go from commit “N” to commit “N+1,” and the production stage will eventually make a long jump from commit “N-1” to commit “N+1”. That means the production system will have gone through different internal states than other stages, potentially increasing the differences between the stages.
Reverse the pipeline. Rollback the changes on all other pipeline stages to match the rolled back stated of the production environment. I consider this approach unthinkable as a standing practice because it is always possible that a completely unrelated pull request has already started its way through the pipeline. Even if one successfully reverses the entire pipeline back to a known baseline, there is still the unresolved matter of dealing with previously merged requests after ejecting them from the pipeline.

When developers and operations managers need to step in to deal with a troubled pull request, the flow of activities comes to a halt, and there is uncertainty on how to progress.

Given enough brainpower in studying the failing pull request and wizard-like ability with the Git command line, a rollback is often possible but rarely cost-effective, so it is crucial to keep a couple of design principles in mind to minimize those occurrences:

Small and independent. I mentioned it before, but worth repeating it. Keeping pull requests small in size and impact on the infrastructure means it is easier to entertain a new pull request to undo the breaking change than considering a rollback across pipeline stages.
One pull request at a time. Assuming the volume of changes allows for it, implement a policy of allowing a single pull request to roll across the pipeline. The pipeline can queue the processing of another pull request until it promotes the current pull request to production.
Treat rollbacks as exceptions. Unless the infrastructure supports reliable downgrades to the previous version, do not entertain rollbacks as a regular, automated process. If the infrastructure has complex internal states or, worse, databases, rollbacks rarely bring the system back to health.

Ultimately, it may be impossible to completely rule out the chance of a rollback in a production environment, but following these tips can minimize their occurrence and impact.

Conclusion

CI/CD practices for application development are well-suited for GitOps repositories, but the pipelines produce very different artifacts from applications.

GitOps repositories contain blueprints that remote agents can follow while building or reconfiguring infrastructure. Changes have ramified and widespread implications even for small changes, so it is essential for pipeline designers to carefully consider the goals for developer productivity and costs to the organization.

For developers, GitOps requires renewed attention to the CI/CD principles of keeping changes small in size, realizing that “small” also extends to anticipating the impact of changes in the underlying infrastructure and adjusting pull requests to reduce that impact where possible.

These lessons helped me minimize the number of manual interventions to our GitOps pipeline and, more importantly, smooth out the flow of activities and the amount of rework around them. While some of the lessons may be more of a journey than concrete recipes, I think they are universal enough to help practitioners at various stages of adoption.

References

Design strategies for GitOps repositories

2022-02-28T20:30:00.004-08:00

(I originally posted this article on Medium.com)

(This story is the foundation of my other story, designing CI/CD pipelines for GitOps.)

Spend a few hours in your newly established GitOps practice, and you soon realize you need to decide how to represent your infrastructure as folders.

On the one hand, infrastructure is rich in concepts and relationships, spread over physical and logical entities. On the other hand, Git repositories have only a few building blocks to represent that knowledge: branches, tags, a folder tree, and files inside the folders.

Mapping a semantic network to a folder structure is not always straightforward.

Now you have a classic mapping problem. Should folders be organized by device class? By pipeline stage? Maybe dedicate a folder to each specific device? How about relationships?

This article lists the core lessons I learned while establishing a GitOps practice for deploying and managing Kubernetes (OpenShift) clusters running software stacks with thousands of containers.

Lesson #1: Beware the topology

Do not spend too much time developing an elaborate folder structure resembling the structure of the deployment environment. Anything over a certain depth makes navigation cumbersome, documentation more complicated, and explanations longer.

My magic number is three levels. Your number may vary, but the challenge past that number is that folder trees are relatively narrow and rigid structures without the means to add attributes to folders or relationships. While one may assign semantics to levels in the tree, those semantics are not visible in a file explorer and need to be written down somewhere else.

As a result, “tall” trees not only result in dissonant changes of themes across folder levels, these external semantics do not hold up well as the system evolves. For instance, take the somewhat long navigation path below as a concrete representation of how folders are organized inside a Git repository:

/production/us-east/kubernetes/omnicorp-cluster/operators/elastic

“Tall” repository trees can make repository navigation a constant chore.

Such implicit folder structure may feel like a sensible containment tree, but what if you need a new Kubernetes cluster deployed with redundant workers across multiple regions? Now you can no longer respect the semantics for the second-level folders because they represent a single region containing the cluster workers.

At that point, you have two options:

1. Adding a “multi-region” folder under “production.” The second level of folders in the tree would have a conceptual mixture of specific regions in the infrastructure provider and, disconcertingly, a logical representation that would not match a particular region.

2. You struggle with the cognitive dissonance in option “1” for a few minutes and consider removing the concept of regions from the folder structure, first ensuring that there are no region-specific settings for the clusters. Once you finally remove the concept of regions from the tree structure, you start wondering why you added it in the first place.

The insidious aspect of leaning towards topology mappings is that they initially feel right, even when not strictly necessary or valuable. Therefore, it is important to question every new extra layer proposed to the folder structure, asking yourself whether the new layer will inform considerations and decisions in the workflows around the repository.

Lesson #2: Versioning with branches and tags

All changes must eventually converge to the main branch, but iterations should be versioned using branches and tags in the repository.

Branches and tags are the only extra structural dimension for the tree-based organization of the folders in a Git repository. I use the word “structural” because one can create new dimensions through external conventions, which incurs many of the pitfalls listed in the previous section.

Git branches and tags are essential to support parallel deployments of the same repository across different deployment zones. That ability to support parallel deployments is needed when validating repository changes across different stages of a deployment pipeline.

As you work with branches and tags, it is often beneficial to adopt semantic versioning (https://semver.org/) as the naming strategy. Semantic versioning rules are sensible and well-documented, making it easier for users of a repository to identify more recent versions and quantify the differences between branches as major, minor, or simple patches.

I have seen teams implement simplified versions of semantic versioning without the “major” field, reasoning that “major” entails backward-incompatible changes and that there should never be such a thing in a continuous deployment pipeline.

A new candidate patch branch is being rolled across the deployment pipeline.

Feature branches are ok. I am excepting the usage of short-lived “feature” branches from this lesson. The use of these branches is undisputed since these are the branches created temporarily to support reviews and validation of pull requests. Merged branches add up quickly and start to become a source of clutter and confusion, so I recommend turning on the respective Git provider setting to delete branches immediately after being merged to the target branch.

Not all differences are created equal. A core design principle for a continuous deployment pipeline is minimizing differences between stages because fewer differences increase the chances that a change works in the next pipeline.

Before discussing solutions, let’s qualify those differences:

Intrinsic differences. These are long-term structural differences between different environments, such as a VLAN identifier, different numbers of VMs, or different cloud regions.
Versioning differences. These are the expected transient differences while changes are validated in the previous pipeline stage, such as image URLs for the containers used in the product or versions of micro-services comprising the whole product.

Intrinsic differences should reside in files named as specific to a pipeline stage, such as “values-production.yaml” or “values-us-south.yaml”.

Versioning differences should not go in files named or otherwise marked as specific to an environment. There may be rare exceptions to that rule, but as Ford Prefect once said: “Even if you prove it to me, I won’t believe it.”

With that combination of folder structure, file contents, and branching, let’s walk through the two typical workflows: one that should affect all deployments and one that should only affect a single environment:

Modifying all deployments

The genesis. Developer clones the repository and checkout the “main” branch. Usually, this should be the default branch for the repository.
The developer creates a new branch named “feature-1”, makes local changes to the cloned repository, commits the changes, and pushes the new “feature-1” branch to the remote Git repository at the Git Provider.
The developer submits a pull request for “feature-1” into the “main” branch (For GitLab users, that would be a merge request.)
Trying times ahead. The Git provider issues an event about the pull request to an automated process — owned by the repository owners — responsible for applying the change to the “dev” environment.
The automated process applies the changes in “feature-1” to the development environment and starts the validation tests.
A new branch is born. Assuming the validation tests pass, the new “feature-1” is considered good enough to be merged back into “main,” and that new version of “main” with the changes becomes the basis of a new versioned branched. Note: An automated process cannot guess the correct name for the new branch in a semantic versioning scheme, so it is helpful to allow the author to add that type of clue in the text of the pull request.
Failure is an option. It is still possible that validation in the next pipeline stages uncovers a problem with the new branch resulting in it being sidelined as failed, stopping its progression across the pipeline.
Retreating is not an option. While there are proponents of a rollback strategy for the stages containing a failed branch, accept that rolling forward is the better approach (A topic I covered in a separate article about CI/CD pipelines for GitOps.) In the “roll forward” strategy, a new issue is automatically created, and the team works on a new patch that addresses the issue. Fixing this situation takes precedence over any other modification, and the pipeline is closed for further changes.
If it doesn’t fail, keep it going. With the pull request merged into “main,” that request is closed automatically, and an automated process starts to deploy the changes to the next stage of the pipeline.

Modifying a single environment

If environment-specific folders and files are isolated to small corners of the repository, this scenario is virtually identical to the modification of all environments.

There is room for some optimizations, like skipping slower or expensive tests in environments other than the one being modified. However, one must carefully manage the costs of developing these unique paths and weigh them against the volume of changes to single environments.

No branch per environment. I must put a strong emphasis on the word “versioning” in the title of this lesson. “Beware the topology” still applies, and trying to create a branch per deployment environment is riddled with pitfalls. This article covers the main problems in great detail, so I will not revisit them here.

Lesson #3: Beware the Monorepo

“Monorepo” is a strategy where the entire infrastructure is represented in a single Git repository, such as applications, services, and infrastructure.

I consider this approach useful for demonstrations and class material, where the content is managed by a few people and applied to non-production, short-lived environments.

Monorepo GitOps repo with multiple disciplines, from applications to infrastructure

For production environments, a GitOps practice is likely using some form of managed or self-hosted Git provider, such as GitHub, GitLab, or BitBucket (Note that GitLab is the only provider to open-source its code and allow self-hosting.) These Git providers have a few architectural constraints limiting the usage of monorepos,

Managed Git providers attach extra configuration, eventing, workflows, and access control to Git repositories. As your organization grows, disciplines like application development and infrastructure tend to become more specialized, with that specialization manifesting itself in at least two relevant dimensions:

Workflows. Git providers support automated handlers for events in the repository, such as creating new branches, creating new commits, or requesting a code review.

Handling the creation of a new commit on an “infrastructure” folder is likely to require significantly different validation strategies when compared to handling a change to the “apps” folder. When all folders are in the same repository, every change to any of the folders generates a new event, leaving it up to the automated handler to reroute the event.

Writing and maintaining routing event mechanisms can be fun in their own right but can also become avoidable distractions in your GitOps practice.

Work management and access control. With different workflows come specialized sets of people. Someone who understands the releases of the shopping cart application may not necessarily be the same person who can vet a merge request to assign virtual machines to a load balancer.

Git providers do not offer access control per folder, file, label, or branch, so organization members end up having access to everything in that repository. Update on 4/16: Joe Bowbeer noted in the comments section that GitHub’s “code owners” feature could ensure a minimum set of reviewers based on file names in the pull request.

Skipping past eventual security concerns around Sealed Secrets, repository members may still be distracted by extraneous churn and notifications. Over time, as the volume of uninteresting notifications increases, you start to notice some undesirable outcomes: Either people muscle through the noise or people turn off the notifications. Muscling through the noise incurs individual productivity penalties while turning off notifications slows down the flow of activities.

Lesson #4: The workflow rule. Workflows rule.

This lesson extends the previous one: if monorepos have limited application, it follows that one should always have multiple repos, but which ones? While we examine this lesson, it is also important to note that an excessive proliferation of repositories also introduces new costs in managing all these repositories, keeping tabs on access control lists, webhooks, access tokens, and many others.

Even as Git providers support some consolidation of those settings in a parent “organization” containing multiple projects, that means sidelining people and resources to own the shared responsibility for those settings.

Lesson #1 shows how topology or containment mappings may not be the best organizing principles, so it is time to let folder structures take a back seat and remember that Git providers organize workflows around repositories.

People and processes shape the Git repository structure, not the deployment topology.

The rule of thumb to define boundaries between repositories is to look at the Git provider settings and find the overlaps between people and automated processes. If people and processes overlap entirely, then a single repository is likely sufficient for that team.

If people or processes differ, it is time to consider another repository, possibly under the same Git provider “organization.” An “organization” works as a shared container for multiple repositories. It is a good place to share settings across repositories, such as a well-designed continuous integration pipeline for building containers, configuration linters, etc.

Conclusion

Working with deployment environments and Git repositories is a constant exercise in mapping complex topologies to simpler structures. While a deployment structure may seem like a tempting starting point due to the team’s familiarity, topologies tend to be a misleading organizing principle.

Many mapping rules may be costly to be established, only to fail a moment later when you try to represent new topology nuances. Organizing repositories around workflows and people keeps the mapping exercises focused and reduces friction between day-to-day activities and folders.

As the practice grows, use branches and tags to keep your deployment pipelines moving forward. As processes mature, promote common automated workflows to the organization level while keeping repositories isolated to minimize distractions in manual flows (e.g., reviews and approvals.)

I hope these lessons are helpful in new and established deployments, and I would like to read your comments (and links) on the topic. Now, let’s deploy some systems.

References

Why you should avoid Sealed Secrets in your GitOps deployment

2022-02-11T10:46:00.004-08:00

(I originally posted this article on medium.com)

GitOps is the practice of representing system configuration in a Git repository and then using Git workflows to manage changes to that configuration and updates to the system.

That process of representing system configuration in a repository is initially straightforward, seeding configuration files, copying declarative requests from product documentation, and maybe even the occasional scripted sequence.

At first, you feel progress is inevitable, and success awaits a few commits around the corner, then you hit the ultimate GitOps foil: secrets.

Why plain secrets are bad

While it may be evident that placing plain credentials on a Git repository is not the best idea, it is still worth spelling out why it is problematic, lest someone feels it may be an acceptable compromise when using a private repository:

The people who work with the GitOps repository may not be the same people authorized to manage the target environments.

Let’s say you are the approver of a pull request and need the resident network expert to review changes related to the firewall. Unaware that credentials are stored in plain text in the repository, you ask the repository manager to add that expert to the list of users. Suddenly, the network expert has access to a customer database full of private data. If that person is not cleared for that level of access, you are looking at all sorts of paperwork and remediation procedures to rotate and deploy new credentials.

Now, let’s assume a better scenario, where you are aware of the credentials in the repository, thus avoiding the accidental disclosure: now you need to go outside the pull request workflow to involve that person in the review process.

When the better scenario is inefficiency and the worst scenario is akin to juggling knives blindfolded, you know it is time to move on to better practices.

A sealed solution

The idea of a sealed secret in GitOps is to encrypt secrets before adding them to the repository, sharing the private encryption key with those who need to use those secrets. Typically, these encryption keys are placed on the target system and used by a local agent to decrypt the credentials and place them wherever they need to be in the target environment.

This technique allows people to work with the repository without the risk of accidentally disclosing credentials, and that is an improvement over storing credentials in plain sight.

That approach is clever, and I was quite fond of it at the beginning of my journey into GitOps. The initial setup is somewhat easy, the repositories are not widely used on a daily basis yet, and it is easy to find others at that same stage of adoption vouching for the approach, so you can also find support in the community.

Getting past the initial stage and going into larger and more permanent deployments, that simplicity gave way to limitations and risks, and I abandoned the practice altogether. In the next sections, I highlight the main reasons you probably should abandon it too.

Reason #1: The keys to which environment?

Let’s say you have a deployment pipeline with a progression of “dev,” “test,” “stage,” and “production” environments. Each environment will be running the same software, but if you use good SecOps practices, they will use different sets of credentials.

Some GitOps repositories are designed to have one subtree per target environment, while others are designed with a single parameterized tree deployed across different environments.

In the case of a folder tree per target environment, the git repository must have separate locations for the keys for each environment. When you work with the ubiquitous presence of Kubernetes clusters in the enterprise, you will be dealing with individual “Secret” resources spread across multiple namespaces, making the sprawl of folders and files unavoidable.

If we look at parameterized trees, then the situation becomes a little better, with all secrets for each environment getting concentrated into a single file per environment.

Regardless of how you design the GitOps repository, using sealed secrets generates more folders and files, which increases the amount of information people need to absorb, and the amount of artifacts the delivery pipelines need to handle.

Reason #2: The secrets are … right there.

Yes, they are encrypted, and it takes an encryption key to get to the actual credentials, but they are still in the hands of potentially bad actors. Imagine telling someone how their database credentials are all visible to the world and then proceeding to dissuade their anxiety by explaining how the bad actors still don’t have the key.

You may chime in the comments section and explain the technical reason why this is an unfounded fear, but anyone who works in security will tell you that the psychological aspect is also part of making customers feel safe with their choices.

Sealed secrets follow an “Excalibur” approach, where anyone can access the (encoded) secrets but only privileged users can decode them.

On the more technical side, I had people argue that having sealed secrets in the open is no different than using public key pairs to encrypt traffic, but those are asymmetric key pairs where you never have the private key out in public, encrypted, or otherwise.

Lastly, while the secret itself is encrypted, the metadata around them isn’t. Bad actors can exploit committer information to seed social engineering exploits, infer the rotation policies for the infrastructure components, determine whether secrets are reused across environments, and gather many other clues that can aid attacks against the target environment.

Retreating from all lines of defense against cyber attacks and pinning all hopes on defending a single point of failure is a terrible starting point for a secure system.

Reason #3: The key to secure all keys is still a key.

The lifecycle of secrets in the repository may differ depending on what they are securing. A database credential may expire every 30 days, while a cluster credential may expire every 60 days.

What about the master encryption key for the sealed secrets themselves? Security policies will eventually force you to rotate that master key, which will require a separate process to redistribute the new key securely to all target environments.

But wait! Not having a separate process to distribute keys to target environments was the reason you chose to seal secrets in the Git repository in the first place. One may argue handling one master secret is better than handling multiple secrets, but the cost of managing one key or multiple keys is virtually the same, with the added challenge of managing the sealed secrets yourself and still needing a password manager of some sort to secure and distribute the master keys.

Reason #4: There are better solutions.

Git repositories were not designed with key management in mind. They have no support for key rotation, no support for serving secrets as symbolic references, no way to perform usage audits, no support for different levels of access to administrators, and so on.

A key management solution is designed to address all these requirements, reduce the surface area for potential leaks, and offer mitigation paths in case a key is ever compromised.

That reduction of surface area is especially important, because you cannot accidentally disclose or lose a key if it never leaves the system. As one example, in the world of Kubernetes, clusters are invariably colocated in a service plane that contains multiple key management solutions. It is common for IaaS providers to offer backend integration between services where key values never have to leave the environment.

Sticking with the example of Kubernetes, where you are likely to be using ArgoCD or Flux for your GitOps practices, they currently lack native integration with key management services, but they expose extension points to achieve that integration. For instance, the ArgoCD documentation says, “Argo CD is un-opinionated about how secrets are managed”, but then proceeds to offer a long list of solutions to integrate with dedicated services.

“Late binding” of secret names to secret values at deployment time.

A more promising approach is getting started through “External Secrets Operator” project, which synchronizes secrets from various key management services into local secrets in a Kubernetes cluster (credit to my colleague Carlos Santana for that reference.)

Thomas Boerger chimed in the comments section about another alternative to sealed secrets: the usage of SOPS with Flux and Argo.

Conclusion

There may be valid reasons to use sealed secrets. Still, I have yet to see one framed in a positive light beyond sealed secrets being “good enough”, which implies they are cheaper to deploy than a proper key management solution. I rarely see the discussion get into the considerations of everything else involved in handling those secrets.

I don’t doubt the engineering prowess of those setting out to mimic a key management service with text files in a code repo, but I question the cost-effectiveness of those approaches. The DIY crowd must resort to a combination of placing files in the git repo, mapping out key rotation cycles to git pull requests, and instrumenting continuous deployment pipelines with decryption keys to parse the repo’s contents. And if you are asking how to manage those master keys, you may find yourself in a constant cycle of coming up with creative ways of making Git act as a key management service.

One can always argue in favor of a build-as-we-grow approach, but that would position sealed secrets as a stepping stone towards using a key management service, and that is not the case. Trying to “grow out” of using sealed secrets means changes to the GitOps backend of choice and re-training operations people to completely change how they handle credentials. There is no natural progression, just paying twice for the same results.

Eschew sealed secrets, start your GitOps practice right, and use a managed key service.

The art and science of probing a Kubernetes container, part 3: Metrics and PromQL queries.

2022-01-05T18:38:00.023-08:00

I wrote about using the kubectl command-line interface to troubleshooting problems with container probes as part of the series "The art and science of probing a Kubernetes container."

That command-line interface approach produces quick assessments, but those assessments are constrained to the present state of the containers. If you want more insight into how that state evolved and how it relates to other events in the cluster, it is time to enter the observability realm and start querying Kubernetes metrics.

A Prometheus server is the most common platform for storing and querying these metrics, so this blog assumes that you have Prometheus installed in the cluster and that you can execute PromQL queries against the server using its built-in expression browser*.

Metrics

These are the essential Kubernetes metrics to study the behavior of a container probe:

prober_probe_total: A counter metric representing the cumulative number of a liveness, readiness, or startup probe for a container by result.
kube_pod_container_status_ready: A gauge metric describing whether the pod is ready to serve requests.
kube_pod_container_status_restarts_total: A counter metric representing the number of container restarts per container.

Queries

Query #1: Table with all probe metrics:

prober_probe_total

prober_probe_total{container="acm-agent", endpoint="https-metrics", instance="10.17....", job="kubelet", metrics_path="/metrics/probes", namespace="...", node="worker0....", pod="klusterlet-addon-workmgr-...", pod_uid="5635c...", probe_type="Liveness", result="failed", service="kubelet"}

kube_pod_container_status_ready

kube_pod_container_status_ready{container="acm-agent", endpoint="https-main", job="kube-state-metrics", namespace="...", pod="klusterlet-addon-workmgr-...", service="kube-state-metrics"}

kube_pod_container_status_restarts_total

kube_pod_container_status_restarts_total{container="acm-agent", endpoint="https-main", job="kube-state-metrics", namespace="...", pod="klusterlet-addon-workmgr-...", service="kube-state-metrics"}

Why: In general, all container metrics have the namespace, pod, and container time series, and this is a good opportunity to see the other time series available for the metrics and plan the next queries. For instance, it is easy to see how the probe_type and result time series in prober_probe_total are relevant for identifying probes that may be failing too often.

Query #2: All failed probes

prober_probe_total{result="failed"}

Why: Using this query in a graph helps identify containers with problems and characterizes these problems over time. Take the screenshot in Figure 1 as an example, where each line represents a container probe (startup, readiness, or liveness.) Lines with an upward slope indicate probes that keep failing, whereas a simultaneous increase for all lines after a skipped collection cycle likely indicates a system-wide problem.

Figure 1 - Prometheus query evaluator showing a graph view of failed probes

Query #3: Rate of failure for all liveness probes

rate(prober_probe_total{probe_type="Liveness",result="failed"}[5m])

Why: A rate function is better at highlighting the upward slopes seen in the previous query while also "smoothing" out the occurrences over time ranges (5 minutes in the example.) Note how it becomes harder to pick apart individual probes but easier to see the mass spike in failures around the 12:40 mark. Also note that from the perspective of finding problems with container probes, the generalized spikes are not all that interesting since they probably indicate a problem with the cluster.

Figure 2 - Graph with the rate of failures for liveness probe in the entire cluster.

Query #4: Rate of container restarts

rate(kube_pod_container_status_restarts_total[5m])

Why: Container restarts are often triggered due to successive failures of a liveness probe, but there may be other reasons, such as outright execution failures that cause the container to exit with an error code. This query offers a valuable backdrop for the next query.

Query #5: Rate of container restarts correlated to failures in liveness probes

rate(prober_probe_total{probe_type="Liveness",result="failed"}[5m]) *

on(namespace,pod,container) group_left

rate(kube_pod_container_status_restarts_total[5m])

Why: This query shows non-zero values when successive failures of a liveness probe happen within 5 minutes of a container restart. This query is handy to account for the "failureThreshold" setting of a liveness probe, where the container probe may fail a few times and still not trigger a container restart. As noted in the previous query, a container may restart for other reasons, which is why you want to have the results of the preceeding query on hand before drawing conclusions.

How: Unlike the previous queries, this one may be a little harder to read because it uses more advanced PromQL syntax, such as the "on" and "group_left " operators. An easier way to read the query is to start with the "*" operator. That operation tells Prometheus to multiply the corresponding values from the two series at each timestamp, with the "on" operator specifying how to match the values between the two series (by namespace, pod name, and container name, in this case.) The "group_left" operator specifies that the left side of the multiplication (the prober_probe_total metric) may not have matching elements on the right side (the kube_pod_container_status_restarts_total metric).

Figure 3 - Graph with container restarts correlated to failures in liveness probes

Query #6: Rate of container unreadiness correlated to failures in readiness probes

rate(prober_probe_total{probe_type="Readiness",result="failed"}[5m]) * on(namespace,pod,container) group_left rate(kube_pod_container_status_ready[5m])

Why: This query shows non-zero values when there are successive failures of a readiness probe within 5 minutes of the kubelet presuming a container is not ready. This query is helpful to account for the "failureThreshold" setting of a readiness probe, where the container probe may fail a few times and still not cross the threshold.

Figure 4 - Graph of container not ready correlated to failures in readiness probes

Query #7: Rate of liveness probe failures correlated to CPU throttling

rate(prober_probe_total{probe_type="Liveness",result="failed"}[5m]) * on(namespace,pod,container) group_right rate(container_cpu_cfs_throttled_periods_total[5m])

Why: Using good practices when designing liveness probes minimizes the chances of the probe failing prematurely, but it is still possible that applying severe CPU limits to the container may starve the probe from much-needed cycles to return prompt responses to the kubelet. A graph of this query can help you isolate liveness probe failures correlated with CPU throttling. In those cases, relaxing the probe settings or increasing the CPU allocation for the entire container may alleviate the problem.

Figure 5 - Chart showing correlation of liveness probe failures with CPU throttling

Grafana

Grafana is a common companion to Prometheus and a good way of bringing multiple queries into a cohesive view. I spent a few minutes** adding some of the queries to a dashboard to exemplify the potential results, with a minor twist to the queries to make the charts look a little less cluttered.

That twist is to only consider metrics with non-zero values by enclosing each query between parenthesis and then adding ">0" in the end, such as in this query:

( rate(prober_probe_total{result="failed"}[5m] )>0

The modification removes from the graph all containers where the probes did not fail in the selected time interval.

Figure 6 shows a snapshot of such a dashboard. You can find the source code for that dashboard in the companion GitHub repository for this blog.

Figure 6 - Grafana dashboard with charts for probe metrics

Conclusion

The first post in the series outlined patterns and anti-patterns for designing effective probes for Kubernetes containers, with the second posting showing simple command-line interface invocations that quickly identify troubled containers.

In this third posting, we looked at how Kubernetes generates essential metrics about the behavior of container probes and about the general health of containers. PromQL queries combine these metrics to paint a more complete picture of how probe failures are distributed over time and their impact on container readiness and restarts.

Applying all these techniques can guide your decisions when designing container probes and help you expedite troubleshooting activities when things don't go well in operations.

I chose the smallest set of metrics that could illustrate these steps while still showcasing the usage of mildly complex PromQL constructs that can become the basis for more elaborate queries. I intentionally left out the lengthier explanations for each query to keep the posting somewhat readable for broader audiences and to keep the conversation going. You can reach me @dnastacio on Twitter.

Happy profiling and good hunting!

* I used Red Hat OpenShift as the basis for this posting, in which case it is also possible to issue PromQL queries directly from the OpenShift console, via Administrator -> Monitoring -> Metrics.

** Credit to Kevin Chung for a precise blog entry on deploying Grafana to Red Hat OpenShift, the Kubernetes distribution used as the basis for all examples throughout this series.

The art and science of probing a Kubernetes container, part 2: kubectl queries

2021-12-28T08:56:00.037-08:00

I covered design and programming tips for creating effective container probes in "The art and science of probing a Kubernetes container". Now, it is time to look at some useful commands to inspect the behavior and effectiveness of the probes in a cluster.

The Pod resource specification combines configuration and runtime information, making the kubectl command-line interface a good starting point to query underperforming resources.

Note: All examples use the -A option, which means all namespaces. Replace -A with -n $target_namespace to narrow the results to a namespace.

(Query #1) All containers that are not ready:


kubectl get pod -A -o go-template='
{{- range .items -}}{{- $pod := . -}}
  {{- range .status.containerStatuses -}}
    {{- if eq .ready false -}}
      {{- $pod.metadata.name -}}{{- "," -}}
      {{- $pod.metadata.namespace -}}{{- "," -}}
      {{- .name -}}{{- "," -}}
      {{- .state.waiting.reason -}}{{- "," -}}
      {{- .state.waiting.message -}}
      {{- "\n" -}}
    {{- end -}}
  {{- end -}}
{{- end -}}'

Why: Getting a complete view of all unready containers helps identify more significant problems with the cluster, reducing the chances of misdiagnosing a system-wide problem as a problem with the tuning of probes in individual containers.

(Query #2) All containers that have restarted more than five times:


kubectl get pod -A -o go-template='
{{- range .items -}}{{- $pod := . -}}
  {{- range .status.containerStatuses -}}
    {{- if gt .restartCount 5 -}}
      {{- $pod.metadata.name -}}{{- "," -}}
      {{- $pod.metadata.namespace -}}{{- "," -}}
      {{- .name -}}{{- "," -}}
      {{- .restartCount -}}{{- "," -}}
      {{- .lastState.terminated.reason -}}{{- "," -}}
      {{- .lastState.terminated.finishedAt -}}
      {{- "\n" -}}
    {{- end -}}
  {{- end -}}
{{- end -}}'

Why: A container terminated repeatedly is a sign of a container with fundamental problems in its processes or, more often than not, a container with a low tolerance in its liveness probe.

(Query #3) All containers that are not ready and have restarted more than five times:


kubectl get pod -A -o go-template='
{{- range .items -}}{{- $pod := . -}}
  {{- range .status.containerStatuses -}}
    {{- if and (eq .ready false) (gt .restartCount 5) -}}
      {{- $pod.metadata.name -}}{{- "," -}}
      {{- $pod.metadata.namespace -}}{{- "," -}}
      {{- .name -}}{{- "," -}}
      {{- .state.waiting.reason -}}{{- "," -}}
      {{- .state.waiting.message -}}
      {{- "\n" -}}
    {{- end -}}
  {{- end -}}
{{- end -}}'

Why: A container that keeps getting terminated and is currently unready may be a sign of fundamental stability problems in its code or, more often than not, a sign of low tolerance in its readiness and liveness probes.

(Query #4) All containers with the same endpoint for the container liveness and readiness probes:


kubectl get pod -A -o go-template='
{{- range .items -}}{{- $pod := . -}}
  {{- range .spec.containers -}}
    {{- if and .readinessProbe.tcpSocket.port .livenessProbe.tcpSocket.port -}}
        {{- if eq .readinessProbe.tcpSocket.port .livenessProbe.tcpSocket.port -}}
            {{- $pod.metadata.name -}}{{- "," -}}
            {{- $pod.metadata.namespace -}}{{- "," -}}
            {{- .name -}}{{- "," -}}
            {{- .readinessProbe -}}
            {{- "\n" -}}
        {{- end -}}
    {{- end -}}
    {{- if and .readinessProbe.httpGet.path .livenessProbe.httpGet.path -}}
        {{- if eq .readinessProbe.httpGet.path .livenessProbe.httpGet.path -}}
        {{- if eq .readinessProbe.httpGet.port .livenessProbe.httpGet.port -}}
        {{- if eq .readinessProbe.httpGet.scheme .livenessProbe.httpGet.scheme -}}
            {{- $pod.metadata.name -}}{{- "," -}}
            {{- $pod.metadata.namespace -}}{{- "," -}}
            {{- .name -}}{{- "," -}}
            {{- .readinessProbe -}}
            {{- "\n" -}}
        {{- end -}}
        {{- end -}}
        {{- end -}}
    {{- end -}}
    {{- if and .readinessProbe.exec .livenessProbe.exec -}}
        {{- if eq (print .readinessProbe.exec) (print .livenessProbe.exec) -}}
            {{- $pod.metadata.name -}}{{- "," -}}
            {{- $pod.metadata.namespace -}}{{- "," -}}
            {{- .name -}}{{- "," -}}
            {{- .readinessProbe -}}
            {{- "\n" -}}
        {{- end -}}
    {{- end -}}
  {{- end -}}
{{- end -}}'

Why: Even when the settings of a container liveness probe allow it to wait longer than the readiness probe, the primary goal of a liveness probe is to safeguard for bugs and unrecoverable errors, whereas the purpose of a readiness probe is to ensure the container can accept new requests. The checks may overlap, but the non-overlapping cases may trick the kubelet into terminating containers that are simply busy.

(Query #5) All containers with no startup probe and a high value for the initialDelaySeconds setting of the readiness probe:


kubectl get pod -A -o go-template='
{{- range .items -}}{{- $pod := . -}}
  {{- range .spec.containers -}}
    {{- if and (not .startupProbe) (.readinessProbe.initialDelaySeconds) -}}
      {{- if gt .readinessProbe.initialDelaySeconds 60 -}}
        {{- $pod.metadata.name -}}{{- "," -}}
        {{- $pod.metadata.namespace -}}{{- "," -}}
        {{- .name -}}{{- "," -}}
        {{- "readiness," -}}
        {{- .readinessProbe.initialDelaySeconds -}}
        {{- "\n" -}}
      {{- end -}}
    {{- end -}}
  {{- end -}}
{{- end -}}'

Why: Use startup probes instead of pausing readiness probes, then break down that initial delay into multiple checks, allowing the container to become ready faster if the initialization work completes faster. For instance, use 10 checks every 6 seconds in the startup probe instead of a fixed 60-second delay in the initial delay of the readiness probe.

(Query #6) Estimated startup duration for all containers

The Go templating in kubectl is powerful but has limits, so you eventually need to step up to using other tools to process the command output or write dedicated applications using the Kubernetes API.

In a nod to the "How to draw a horse" meme, here is a combination of kubectl and shell scripting I have been working on, returning all containers, probes, restart counts, and an estimation of how long each container took to become ready. You can see an output example here.

I must emphasize that the startup duration (measured in seconds) in the above example is only a rough estimate because the pod specification does not break down all the timestamps for the container lifecycle.

Also note that if a container is terminated and then restarted within the pod, there are situations where the last termination timestamp is not recorded, making it impossible to determine the duration of the last startup.

References

"Customizing OC Output With Go Templates", though you need to remember to replace "oc" with "kubectl" in the examples if you are not using a Red Hat OpenShift cluster ("oc" is OpenShift's enhanced, yet compatible, version of "kubectl").
Go-template functions
kubectl cheat sheet

The Art and Science of Probing a Kubernetes Container

2021-12-17T15:32:00.032-08:00

Keeping containers alive in a Kubernetes cluster can feel more like art than science.

In this article, I dive into the sea of madness awaiting those responsible for authoring container probes, with particular attention to the relatively new addition of startup probes to the mix. Along the way, I leave a breadcrumb trail of curated links you can use to take the next step in implementing the various suggestions in the article.

Starting, nay, requesting the start of a new container in a Kubernetes cluster is relatively simple: provide a PodSpec to the cluster, possibly as a PodTemplateSpec wrapped inside one of the various flavors of workload resources. After receiving the PodSpec, the kube-scheduler assigns the pod to a node, then the kubelet in that node starts the containers in the pod.

Pods follow a clear lifecycle, and part of that lifecycle requires the kubelet to keep tabs on the containers in the pod, to ensure they are responsive throughout their existence, with remedial actions ranging from suspending traffic to the pod to terminating the pod.

As background, there are three types of probes a developer can include in a container specification:

Readiness probes: This probe tells the kubelet when the container is ready to process requests. It is the most prevalent of container probes.

Liveness probes: This is the "break in case of emergency" probe. Ideally, containers should exit after realizing they can no longer do their job, but bugs are rarely graceful in their symptoms. A kubelet terminates containers that do not respond successfully in a specified interval.

Startup probes: We could also call them the "I just got here, leave me alone" probes. It tells the kubelet when to start evaluating readiness and liveness probes.

Figure 1 - Relationship between container probes and a pod lifecycle

Readiness probes

If a container fails to respond to its readiness probe, the kubelet removes the container from the service load balancer, effectively diverting traffic away from the container. Here, the developer hopes that a replica elsewhere can handle that traffic.

The design of a readiness probe is somewhat straightforward. You want to take into account the state of dependencies and the resource usage in the container:

Dependencies. If your container depends on something like a database server or another remote server, those types of dependencies tend to offer an endpoint or command-line interface to assess their readiness. If the dependencies are in the critical path of serving requests to clients, then factor in the dependency status in the calculation of the readiness status of the probe.

Maximum allowed connections. Many frameworks have a maximum configured limit for accepting new connections, so factor that in as well.

System resources. This one is not as obvious, but running close to memory ceilings and running out of space in a filesystem are known for destabilizing processes. You want the cluster to stop sending traffic to the pod before running out of system resources, so consider failing probe calls when these limits reach a certain threshold of the maximum. One of my least favorite forms of resource exhaustion is running out of file handles, which is harder to detect than a simple lack of disk space and can block even the most basic troubleshooting tasks.

Do's:

Have a probe. Always define a readiness probe for your runtime containers. You may feel like your container clients can deal with the container becoming unresponsive. Still, there is never a good reason to let the cluster route requests to a container that may not be ready to handle the requests.

Have the status ready. Consider assessing the status of remote dependencies and resource utilization outside the thread servicing the readiness probe. Readiness probes have a timeout value, and it is best to return a clear failure code immediately than risk timing out a response while gathering input from all dependencies.

Don'ts:

Do not exceed the limits of the liveness probe. If the container also has a liveness probe, do not make the maximum time tolerance (failureThreshold * periodSeconds) longer than the maximum tolerance for the liveness probe. That is a recipe for letting the cluster route requests to a potentially hopeless container about to crash.

Do not mix slowness with readiness. A slow response is still a response. You may feel like your container must let callers know that it is not ready because a dependency is taking much longer than usual to process requests, but that is a concern best left for the observability discipline.

Liveness probes

If a container fails to respond to this probe consecutively, the kubelet will terminate the container. Emphasis on "terminate" versus "restart," which depends on the restart policy of the pod.

These probes are notoriously difficult to code correctly because the goal of the probe's developer is to anticipate the unexpected, such as a bug in a process that could put the whole container in an unrecoverable state.

If the probe is too lenient, the container may become unresponsive without being terminated, effectively reducing the number of replicas of that pod with a chance of serving traffic.

If the probe is too strict, the container may keep getting terminated unnecessarily, a condition that is insidiously hard to detect when happening intermittently, risking the pod looking healthy right when you are looking for problems.

Do's:

Define a liveness probe. Not having a liveness probe means leaving your container exposed to becoming permanently unresponsive due to a bug, so always give it a try.

Monitor resources. Cover resources like filesystem space, file handles, and memory. These resources are notorious for locking up containers when exhausted. It is wiser to return a failure when memory utilization passes the 90% mark than risking becoming completely unresponsive after hitting 100%. The probe has timeout values, but it is always best to return a clear failure code than let the kubelet infer a crash from a timeout.

Use commands. Favor invoking commands instead of tcp or http requests. This advice may be controversial, but invoking a shell command uses a lower-level interface, which in turn has a higher chance of allowing your code to assess the internal state of the container. I have met my share of web servers that are happy to respond to simple ping requests even while half-crashed due to lack of memory.

Micro control-planes. If possible, use a different connection pool than the one used to serve customer traffic, or set a connection aside specifically for the probes. Think of the distinction between control planes and regular workloads in the cluster but on a much smaller scale. Unless your liveness probe has a dedicated connection for answering the liveness probe, there is a chance that the container is busily servicing customer traffic (and appropriately reporting not being ready through its readiness probe.) Terminating the container under those conditions is harmful since the cluster ends up (temporarily) having even fewer containers to service actual requests.

Have the status ready. Similar to the suggestion in the section about liveness probes, consider assessing the liveness of the container outside the thread servicing the liveness probe. Liveness probes have a timeout value, and it is best to return a clear failure code immediately than risk timing out a response while gathering input from all dependencies.

Don'ts:

Readiness is not liveness: Do not check for the availability of dependencies. If they are failing, it is the responsibility of the readiness probe, and terminating the pod is very unlikely to help matters.

Do not reuse the readiness criteria: I often see container probes hitting the same container endpoint but using different thresholds in failureThreshold and periodSeconds. The concerns of a liveness probe are distinct from those of a readiness probe. A container may be unable to handle traffic due to external factors, and a liveness probe using the same endpoint as a readiness probe risks compounding the problem by telling the kubelet to terminate the container.

Tolerate more than readiness probe: Whether you are looking at the failureThreshold, periodSeconds, or considering the availability of system resources in the container, make sure that the tolerances of the liveness probe exceed the patience of the readiness probe. For instance, making the maximum time tolerance (initialDelaySeconds + failureThreshold * periodSeconds) shorter than the maximum tolerance for the readiness probe is a recipe for terminating the container prematurely while still servicing remote requests.

Do not try to be conservative: In doubt, err on the side of leniency, and set higher values for failureThreshold, periodSeconds, and initialDelaySeconds, giving your container plenty of latitude to report being alive. Unexpected hung processes are an edge case and far more rare than slow responding processes, and a liveness probe should favor the most common causes. A good rule of thumb is using a total tolerance that is twice as long or longer than the tolerance of a readiness probe.

Figure 2 - A liveness probe should allow more time to completely fail than the readiness probe

Startup probes

Startup probes are a relatively new addition to the stable of container probes, achieving GA status in late 2020, in Kubernetes 1.20. Note: Credit to my colleague, the ever knowledgeable Nathan Brophy, for pointing out the feature was already available by default, albeit in beta stage, as early as of Kubernetes 1.18.

A startup probe creates a "buffer" in the lifecycle of containers that need an inordinate amount of time to become ready.

In the past, in the absence of startup probes, developers resorted to a mix of using initialization containers and setting long initialDelaySeconds values for readiness and liveness probes, each with its own set of compromises:

It can wait but shouldn't. Initialization containers free up developers to isolate specific tools and security privileges from the runtime container, but are a heavy-handed way of waiting for external conditions. They run serially until completion, and because they are separate containers, it may be cumbersome to transfer the result of their work to other containers in the pod.

Slow starts. Long pauses set via the initialDelaySeconds field on a readiness probe can waste time during a container's startup because the kubelet always waits that much time before considering sending traffic to the pod.

Consider whether your existing liveness and readiness probes take relatively long to start and replace high values of initialDelaySeconds with an equivalent startup probe.

For instance, this container spec below:

spec:

containers:

- name: myslowstarter

image: ...

...

readinessProbe:

tcpSocket:

port: 8080

initialDelaySeconds: 600

periodSeconds: 10

livenessProbe:

tcpSocket:

port: 8080

initialDelaySeconds: 600

periodSeconds: 20

can be significantly improved by moving that delay to be observed in a startup probe, like this:

spec:

containers:

- name: myslowstarter

image: ...

...

readinessProbe:

tcpSocket:

port: 8080

periodSeconds: 10

~~initialDelaySeconds: 600~~

livenessProbe:

tcpSocket:

port: 8080

periodSeconds: 20

~~initialDelaySeconds: 600~~

startupProbe:

tcpSocket:

port: 8080

failureThreshold: 60

periodSeconds: 10

In the first example, the kubelet always waits 600 seconds before evaluating the readiness and liveness probes. In contrast, in the second example, the kubelet checks up to 60 times in 10-second intervals, thus enabling the container to start as soon as it meets its startup conditions.

A hidden benefit of the frequent checks in a startup probe is that it enables a developer to set a high value of failureThreshold and periodSeconds without worrying about slowing down the container startup. In contrast, the unwieldy observance of initialDelaySeconds puts pressure on developers to ignore edge cases and set lower values that allow the entire application to start faster. In my experience, "edge cases" are synonymous with "things we have not seen during development," which translates to unstable containers in some production environments.

As a rule of thumb, use startup probes if the initialDelaySeconds field in your readiness and liveness probes exceeds the total time specified through failureThreshold * periodSeconds fields. As a companion rule of thumb, anything over 60 seconds for initialDelaySeconds in a readiness or liveness probe is a good sign that your application would benefit from using a startup probe instead.

Timing is everything

After observing the suggestions in this article, you are ready to ask the inevitable question:

"So, what should I use for the probe settings?"

For timeoutSeconds, I would recommend keeping the value at its default (one second.) This suggestion builds atop my other advice for assessing the responses of a probe outside the thread answering the kubelet request. A quick answer about a problem is always better than letting the kubelet infer the result from a timeout.

For the combination of periodSeconds and failureThreshold, more checks in the same interval tend to be more accurate than fewer checks. Assuming you followed the suggestion of assessing the container status separately from the thread responding to the request, more frequent checks will not add significant overhead to the container.

Mind your CPU limits

Different clusters, different speeds.

A common problem with probes, specially liveness probes, is to assume a cluster will always give your container as much CPU as you request. The other common mistake is to assume that clusters will always observe fractional requests with surgical precision.

Starting with the hypervisor and the VMs hosting worker nodes, all the way to the CPU limits in a pod specification, there are myriad reasons why a container can run the same stretch of code at different speeds.

Here are the top factors that can blind-side a developer:

Overcommitted CPUs in the hypervisor: Shared VMs in the IaaS provider. Even identical hardware and networking speeds can be affected by the occasional "noisy neighbor" hitting a burst of CPU usage. Modern hypervisors are pretty good at compensating for such bursts and even throttling processes, but an IaaS may overcommit CPUs under the assumption processes will not all burst at the same time.
Infinitesimal CPU requests: Setting a CPU limit of 20ms for CPU allocation to a container may seem like a responsible, conscious decision for a container that rarely does any processing. However, in the real world, a worker node does not have a tiny vCPU 2% the size of a full vCPU. The worker node attempts to simulate that tiny vCPU by giving your container an entire vCPU during a small slice of time, resulting in a "lumpy" allocation of CPU to the container. As a result, the container may find itself receiving a lot more CPU than requested for a brief interval and then entirely paused for longer than expected.

Keep those two factors in mind when setting the values for probes, especially liveness probes. Learning a bit about the hardware characteristics and overcommitment settings of your IaaS provider can go a long way in deciding the safety multipliers to add to settings like timeoutSeconds, failureThreshold, and periodSeconds. Depending on what you learn, you may also reconsider the settings for CPU requests and limits so that your probes have enough processing capacity to respond to requests promptly.

Figure 3 - Healthy container getting terminated due to a restrictive CPU limit

Conclusion

This article offers a range of suggestions to improve the precision and performance of container probes, allowing containers to start faster and stay running longer.

The next step comes from careful analysis of what runs inside the containers and studying their behavior in actual runtime across a diverse set of clusters and conditions, going as far as simulating failures of dependencies and reduced availability of system resources. Using the kubectl utility and its ability to format and filter contents is a great way to find containers with a high number of restarts and inadequate probe limits, which is a more technical subject covered here. Using PromQL with Kubernetes metrics can further expand that technique with charts for various time series, a topic covered in this other posting.

In summary, keep the goal of the probes in mind when writing them, ensure they run quickly and assuredly, providing clear information with minimal (if any) false positives in their response to the kubelet. Then trust the cluster to do what it does best with the data, ensuring maximum availability of your containers to their clients.

Should (only) developers be on-call?

2021-09-06T12:32:00.014-07:00

The question “should developers be in the on-call rotation?” is a frequent debate in technical forums, with the narrow nature of the question demanding polarization of the community around the answer.

I don’t like that question as a starting point because it preempts all other discussions on how to make (or keep) the system manageable without requiring operators with intimate knowledge of the source code.

This story examines the negative effects of on-call rotation on people and how addressing their causes also results in systems where non-developers can join the on-call rotation in a healthy and effective manner.

Why do I even need an on-call rotation?

The order of the discussion goes:

Anything done on a sustained basis needs a system.
Even in a well-designed system, things don’t always go according to plan.
That gap in the plan may happen at any time because online systems never stop.
Without a scripted procedure, the best chance of resolution lies with brains wired by millennia of adaptations to life-and-death circumstances.
The unpredictability of that exceptional event requires an on-call schedule.
Should a developer be the person answering that pager? And to go further, should developers be the first people answering that pager?
If not, then who?

Ah, clarity! Or at least more of it.

Before diving into answers, we must understand that people are more than actors in the system: they are also a part of it. This shift in perspective augments the scope of sustainable delivery, especially when we look at the science behind the psychological effects on people.

Psychology of on-call

Without getting too deep into the excellent resources listed at the end of this posting, there is a scientific basis demonstrating how on-call shifts increase anxiety and stress on people supporting the system.

Lack of control is at the core of the findings:

The inherent lack of control in being called out of regular business hours.
Lack of control over the causes of the problem.
Uncertainty about having the skills, tools, and support to handle the situation.

The studies show that the mere expectation of being paged produces similar anxiety and stress levels to being paged. The fear of missing an alert while on-call further magnifies the degradation of people’s sleep patterns.

Taken to extremes, where the system routinely wakes people in the middle of the night, an on-call schedule may even reduce life expectancy for responders.

Without going further into this rabbit hole of doom, it is clear that any sustainable system must consider its contributors from the onset. The design must include clear service level objectives for calling out people, systematic training on troubleshooting techniques, and a robust support system for people once they start handling the incident.

Predictability as the basis of sanity

Based on the science of psychological effects on responders, it is evident that we need to pay close attention to the frequency and intensity of on-call shifts.

Addressing the frequency portion requires a continuum of approaches, ranging from preemptive to adaptive.

Shift-left testing is an excellent example of a preemptive approach focused on earlier system validation across design, development, and testing.

Testing in production is the most comprehensive example of an adaptive approach, aiming at rollouts gradually gated and validated by actual runtime results.

Addressing the intensity portion comes later, in section “Asking permission, not forgiveness.”

Limits. We all have them.

Every system looks much better when unconstrained by limits, but the average system always has limits.

The above-average system has documented limits. A great one has written procedures for dealing with each exceeded limit.

A superb system has people continuously working towards removing the limits or automating the handling of blown ceilings.

We are at the point where you remember I wrote that people are part of the system. People also have limits. The limits may vary with personal circumstances and skills, but they exist, and, unlike machine limits, they cannot be quickly removed or automated.

As an operations manager will tell you, dealing with systems at the limit is a volatile business. When it comes to employees experiencing their limits, it can be borderline dangerous to colleagues and, ultimately, the company.

An operations manager has a direct vantage point into non-human limits and the mechanisms to address those failures by adding more capacity and coordinating the deployment of software fixes. On the other hand, a people manager may have an obstructed view of the state of personal stress in his team and how people cope with the stress of being part of the larger system.

For a people manager, the solution to an unclear picture of employees’ stress and anxiety passes through the same gates as the overall system: preempt and adapt.

Preempting anxiety and stress means ensuring the people on-call have an outsized voice in the system’s architecture, design, and rollout.

Asking permission, not forgiveness

If someone is on the hook to handle a system outage off-hours, that person must have a voice in what will be in that system then. Without a voice, that person may not know what they are dealing with while interacting with the system.

The best form of participation happens early in the cycle, starting with architectural reviews. A documented system architecture is the only way to involve the operations people in that process. I think we all heard our share of “the system is the code” statements, but strings of code are rarely an efficient way of conveying abstractions. The thinking is also downright unhelpful during an outage of a system with hundreds of thousands of lines of code written in 5 different programming languages (“what, your ops people don’t know Ruby, Go, Java and Node? Sheesh!”)

An architectural review is a time for on-call people to probe every arrow of the system diagram for limits and error conditions.

You want to ask:

What are the system limits? Probe for network bandwidth usage, system resource consumption, and maximum horizontal and vertical scales.
What are the error conditions? Probe for the possibility of network outages and individual component failures in the system diagram.
How are limits and errors perceived? Will the system administrators notice rejected requests, slow responses, reroutes to static CDN, etc.?
Can error conditions cause the system to exceed limits more quickly?
Will these exceeded limits cause system errors?
How should one handle exceeded limits? Scale out horizontally, scale out vertically, or both? This question is also an opportunity to ask about components that do not tolerate horizontal or vertical scaling.
And ultimately, for one of the most crucial questions: Which parts of the system automatically handle these conditions and which ones require human judgment and subsequent manual intervention?

Under no circumstance should a system leave the architecture stage without an answer or scheduled follow-ups to each of these questions. And if the answer to the last question leans too much towards “people,” you have my permission to quote this scene from the movie Soylent Green.

Beyond design: total cost of ownership

TCO, or total cost of ownership, is the most under-appreciated factor in maintaining a system and one of the most glaring blind spots for system architects, designers, and developers.

As a general rule, people without experience in operations engineering will overestimate the capacity of operations teams and underestimate the financial costs of keeping the system running. Everything, in systems or life, has a cost, whether it is a direct economic cost of paying for goods or services or the indirect lost-opportunity cost of investing time in an activity.

I once worked in an organization where development and operations reported into different silos with separate budgets (you can see where this is going.) This organization and the product no longer exist, so I can share a more relatable example of how a lack of TCO awareness caused severe and long-lasting damage.

In a 6-month interval, the development team introduced two new types of backend servers into a SaaS offering without consulting with the operations team other than informing them of the date they should deploy the new servers. The operations manager accepted the imposition under the condition that the development team was available to support the new servers “until the operations team acquired the skills required to maintain them,” without committing to specific transition dates.

Without a plan or cost controls, the operations team took an entire year to find enough training budget to close the skill gap, let alone code the monitoring bridges from the monitoring platform to the new systems. In the meantime, the development team started spending most of its time on routine system upgrades, coding those bridges from the monitoring systems, and assisting in troubleshooting system outages. To make matters worse, some of the outages did not trace back to the new backend systems but required paging developers to rule out the new backend servers as the cause of the incident.

Suddenly, the product manager started to receive pushback from a hamstrung development team, affecting committed features and prompting customers to wait before signing new contracts until the committed features were available.

The most insidious aspect of authorizing a deployment without knowing its total cost of ownership is that the organization is effectively signing a blank check against its operations and development budgets, and blank checks are not recipes for sustainable businesses.

Also, beware of the “feature I wrote on the weekend” since a prototype will typically cost 30–50 times more to become a stable, managed feature in production, whether or not the development organization is mature enough to estimate the actual TCO of the feature.

Why developers, though?

That is the first point of debate. Are developers uniquely suited to plug those gaps in the system?

In a small organization with a few people and 2/3 of the head-count budget locked in software developers, it may feel natural to sidestep the question with another question: “Who else?”. As a manager, no need for further explanation. As a leader, decisions by elimination are rarely part of a sustainable strategy.

Let’s examine what part “being on call” plays in that system. Your organization promised the product (not the system) would meet specific service levels. Things falling off the rails in the middle of the night or on weekends cannot stay down until business hours resume the next day or on Monday, so “something” needs to be ready to restore the service levels.

Ideally, that “something” would be an automated process that runs immediately, but there are always problems that cannot be identified or resolved in the same ways.

“They are the ones most experienced with our technology stack.”

That is a partially true statement. During the development of a system, software developers spend considerable time working directly with some system components, but not necessarily with all of them.

Suppose someone experienced with the React framework is on-call, and a system outage ultimately originates from a sharding problem in a Mongo database. That personal experience with UI development will not translate into a faster solution unless one argues the next point.

“They are the ones most experienced with technology in general.”

Another partially true statement. The thinking here is that while system components may differ, a software developer has experience in developing systems, and that experience gives them insights into other systems.

This line of thinking does lead to a less common argument: anyone with previous experience in developing systems would have similar insights, which opens the door to people who may still contribute to the system in a different role, such as system designers, architects, and managers.

“Being on-call aligns consequences with [questionable] product-making decisions.”

This argument is flawed in multiple ways:

The previous sections show how simply being on-call adversely affects people’s health, whether or not the system experiences an outage. In other words, people on-call who consistently make good decisions will still be negatively affected, contrary to the intention of aligning actions and consequences.
Eventual outages may not trace directly to a specific design decision, good or bad.
There may be a significant time lag between a decision and its consequence in an outage, so bad decisions may go unnoticed for a long time and maybe never manifest themselves as system outages (such as poor user experience.)
Other non-developer team members can also make decisions that have ramified effects leading to a system outage. Some of the worst, most pervasive system problems causing outages relate to organizational issues rather than design decisions.

There is also ample evidence to suggest that people are not the best at changing current behaviors in light of future events. Even after tracing an outage to a questionable decision, there is no guarantee that the people experiencing that connection will make different decisions in the future.

“Developers learn more about the system by handling system outages.”

This thinking is akin to saying you can learn about boxing by going two rounds against Mike Tyson or Floyd Mayweather. Beyond a certain level of challenge, the personal focus quickly turns from self-improvement to survival.

During a system outage, the focus is on the system’s survival by restoring it to health. I have seen mixed responses to error-budget advocates on Twitter arguing for using that budget as an opportunity to learn more about the system, keeping the system down after finding a solution, and collecting more information about the system while it is in a compromised state.

Nevertheless, in a healthy organization, you can learn more by reading well-written incident reports than by troubleshooting random disks getting filled up in the middle of the night. Hopefully, incident reports will contain:

Timelines.
Initial symptoms.
Troubleshooting steps.
Links to supporting results in web searches.
A list of commands used during the resolution phase.
Links to downstream issues found and reported during the incident to request more permanent solutions in the codebase.

The corollary quip here is that organizations must be disciplined in handling post-incident analysis, systematically studying problems in ways that resemble nothing like this.

“Let’s add all people in the org to the on-call rotation to spread the burden.”

If something is complicated and unstable to the point where it needs 30 people to share the load of handling its outages, that often means outage resolutions require great skills and insights into the system, which only a few people in the organization will possess.

For that reason, I once quipped that a 30-person rotation schedule is a 27-people buffer with a 1-hour delay before calling the only three people who can resolve the outage.

Fix the system, get a grip on its TCO, then manage the frequency and intensity of page-outs. Throwing more people at a wobbling system extends the adverse physical and psychological effects across more people, and there is never a good business case for stressing additional parts of the system beyond their limits.

Conclusion

Placing developers in an on-call rotation is common and expected. However, it should not be the starting point of any conversation on operating profitable and sustainable systems.

While designing any system, the discussions must include the availability and costs of human intervention and aim at reducing or eliminating that type of intervention while also developing the system’s “service points” with high-visibility tracing from symptoms to causes to resolutions. A healthy organization must handle human limits more carefully and rigorously than system limits.

Systemic failures should be consistently studied and documented to learn and contextualize new system features to keep page-outs rare and manageable.

An operations team (or the operational discipline in organizations where the entire staff co-shares system administration) should have a firm grip on the total cost of ownership and be a prominent final voice in the deployment of system upgrades or new systems.

Once an infrequent and well-scoped problem resolution workflow comes out of the entire system design phase, the on-call rotation should include people equipped to recognize limits and error conditions and a clear supporting structure to bring the system back to health.

Building a system that prioritizes the well-being of people on-call also allows the participation of a wider pool of people who can handle the occasional system failure without knowing the system at the source code level. Enabling that wider pool of participants in the on-call rotation is particularly useful as the system grows and a larger organization requires a dedicated operations team.

References

“How the chance of missing the alarm during an on-call shift affects pre-bed anxiety, sleep and next-day cognitive performance”: https://www.sciencedirect.com/science/article/abs/pii/S0301051118300929

Highlights:

Pre-bed anxiety is higher on on-call nights compared with control.
Poorer sleep on on-call nights where the perceived chance of missing the call alarm was high.
Faster reaction times on days after on-call nights with a low chance of missing the alarm.

“Effects of On-Call Work on Well-Being: Results of a Daily Survey”: https://iaap-journals.onlinelibrary.wiley.com/doi/abs/10.1111/j.1758-0854.2012.01075.x

Highlight: “Flexible work schedules like on-call work have effects on well-being. The mere possibility of being disturbed by calls shows negative consequences, regardless of whether the employees are called in or not.”

“On-call work and physicians’ turnover intention: the moderating effect of job strain”: https://www.tandfonline.com/doi/abs/10.1080/13548506.2015.1051061

Highlight: “The highest levels of turnover intention were among those who had on-call duties and high level of job strain characterized by high demands and low control opportunities.”

“On-Call: The definitive guide to running productive and happy on-call teams”: https://www.atlassian.com/incident-management/on-call/on-call-schedules

AI Ops Series - Part 2: dealing with state machines and time

2021-05-21T06:00:00.018-07:00

In the first part of this series, I wrote about the importance of observability and continuous deployment practices for AI Ops. An observability practice must be pervasive across the organization to support all use cases for training AIs. A continuous deployment practice must be responsive and reliable to support frequent deployment and validation of changes recommended by the AI.

In this second part, I explore the topic of state machines, their adverse impact on AI training efforts, and mitigation strategies.

State of the union of various systems

State machines are combinations of system states and rules directing the transition from one state to another. In an IT shop, the more relevant state machines are of the event-driven kind, responding to occurrences in the system.

Some state machines are purpose-built, finite, and well-understood, such as online shopping carts, with all the steps involved in keeping track of the customer selections until they hit that "Buy" button.

However, not all state machines are designed by or visible to the team operating an IT shop. Some of them lurk inside low-level components, known and understood only by the component vendors. Others are formed inside the data center as you plug various systems together. These ad-hoc state machines introduce novel states and transition paths that are often unnoticeable during regular system operation.

The insidious aspect of these "spontaneous" state machines is that their complexity tends to isolate cause and effect behind a seemingly random combination of state transitions. Stretch that barrier over a sufficiently long time, and no amount of mathematical genius, human or otherwise, will be able to infer any pattern between cause and effect in that system.

You can tell where this posting is going: Without (recognizable) patterns, there is no AI Ops.

Butterfly effects: the delayed kind

Let's recall the scenario in part 1, where maintenance of firewall rules inadvertently blocked new connections between two endpoints. Changes to firewall rules affect new session requests but not established links, so pre-existing traffic still flows across the wire. That list of established links is part of the internal state maintained by the firewall appliance.

Suppose you asked the network administrators to find unintended side-effects of those changes to the firewall rules by monitoring the traffic between systems. Without knowledge of that internal state, they may arrive at the incorrect conclusion that there is no correlation because they can still see some traffic flowing.

Now stretch that scenario over an extended period. Let's assume each of those two endpoints hosts one of the pairs in a high-availability database cluster. In this scenario, these two database nodes established their reciprocal link before the network admins accidentally prevented new connections from forming. Imagine that one of these database nodes fails and, upon restart, that database node cannot establish a link to the other node.

Once teams start dealing with that incident, they may be wondering why the failed database node cannot restart since there is no record of recent changes to it. After hours of troubleshooting, both network and database teams finally discover that the action from weeks ago had a deferred impact on the system's reliability.

This type of confusion is just one example of the effects of a state machine (hosted inside the firewall appliance) obscuring cause and effect events across system boundaries and time.

Secrets of state

With state machines (intended or not) being an integral part of the modern data center, it is crucial to apply strategies that can:

Reduce their size.
Restrict them to smaller portions of the system.
Compress their lifecycle into smaller time windows.

That is not to mean you should go about rearchitecting your entire system to eliminate all your queue servers and distributed caches. Still, you should take a hard look at those types of indirections and reconsider whether more direct flows could achieve the same results.

The following are the lessons I learned over time about managing state, which forms the advice I would give anyone trying to curb system state and make it more manageable for an AI Ops practice. As a bonus, anything that makes your system signals more strongly correlated can also make your system easier to operate. Win, win.

Lesson #1: Reach out for the serverless parts-bin

To state the obvious: if a system does not exist, it cannot hold an internal state.

The first approach in reducing the number and complexity of state machines in the system is to consolidate persistent data into fewer components. With the state contained in fewer places, gradually migrate the new state-less systems to a serverless model, which only runs processes long enough to process individual requests.

With fewer systems holding internal states and other systems not existing long enough to accumulate any state, you will reduce the size of your system-wide state machines as well as their duration over time.

As one example from an actual project, a team had dozens of servers capable of processing requests that took several minutes each. In some situations, customers could simultaneously send a few hundred requests and then never send anything else for a week. In the meantime, these servers accumulated state while processing the requests and also while sitting idle. Some of it was useful for performance reasons, such as caching the results of large queries to a scientific literature database. Other cached states were less valuable, centered around the virtual machines processing the requests, such as operating system patches, local user tables, log files exhausting disk space, etc.

Now imagine the same system with a couple of stateful servers handling the database interactions and all the customer requests processed in a serverless fashion, with containers being spun up only for the duration of each request,

The stateful servers can deal with the lengthy database queries and the regrouping of table rows in the responses, while the other permanent servers do not exist anymore.

Lesson #2: Join the crowds

Look for products with good documentation and large user bases to increase your understanding of how it behaves. A product or component with an extensive user base tends to have its edge cases explored more often, leading to fixes (better quality) or better documentation (better visibility into those edge cases.)

If that sounds a lot like open source, you may be right. Commoditization of components and design patterns means a more predictable system, and predictability is always good whether you are learning something or using the resulting data to train an AI.

As usual, be mindful of tradeoffs: any new component introduced in a data center brings along its complexity. Ensure you are not replacing a small and obscure corner of the system with something transparent but unwieldy.

After all, every team has their stories of people bringing in entire queue management systems to handle a handful of messages a day. These teams may also have stories about those queue servers requiring all sorts of unexpected upkeep and monitoring of queue sizes, dead-letter queues, etc.

Lesson #3: State your intentions, declarative over imperative

An imperative approach to system configuration describes all steps required to bring the system to the desired state. In contrast, a declarative model expresses the expected state of the system without prescribing the control flows needed to get there.

In simpler words, with a declarative approach, you tell the system what you want. In an imperative process, you have to build what you need step-by-step.

The examples of imperative approaches in a data center are scripts written in languages ranging from shell scripting to higher-level abstractions such as cdk8s. GitOps is probably the more visible umbrella for declarative approaches, built atop configuration frameworks like helm, kustomize, and jsonnet.

Q: What does any of this have to do with state machines and AI Ops?

Declarative approaches require system states to be handled by components specialized in bringing the system from an initial state to the desired state, such as ArgoCD, Flux, and Jenkins X. These components have narrower mandates than custom-built code (which pretty much can touch everything) and strive to process the system changes as quickly as possible. They ensure that a request for the system to achieve a given state leads to outputs directly tied to the request both in scope and time.

Q: Don't these components bring their own internal state, thus offsetting the benefits and just moving the original problem elsewhere?

Yes, they have their state, but it is narrower in scope and in time. They require you to decouple functions that handle state from your other control loops and consolidate those functions into fewer system components. This consolidation reduces the number of architectural and logical)connections between systems and the number and size of system-wide state machines.

As an extra tip, minimize the use of "escape hatches" in declarative approaches, where you can ask the component processing the requests to execute imperative code blocks before and after each request.

Lesson #4: Always test configuration changes immediately

It is challenging to detect patterns between even the most perfectly correlated events when they happen far apart in time. Unless the time windows separating the occurrences is somewhat constant and well-known, or you are pretty good at time series forecasting (see next section,) the chances of finding those patterns become rather slim.

One common source of delayed reactions in a system is deferred restarts of affected services after configuration changes. As a general rule, whenever the configuration for a component changes, restart all instances of that component, observing high-availability requirements and capabilities of the system. The restart ensures that eventual side effects of those changes manifest themselves sooner instead of the next time the component restarts, like in the earlier example about firewall rules in a network appliance.

I felt that this lesson might not be as broad as the others in this article. However, the combination of their frequency and chaotic impact on machine learning efforts helped it make it into the list.

Lesson #5: Transforming time

After applying all previous lessons, you are still likely to be facing several state machines in the system, which will be in their never-ending march through time. In that time, the system will be producing metrics continuously, generating various time series.

Finding correlations between the measurements in a time series may be difficult if some metrics reflect the consequences of past events. Even a short time may throw off your correlation attempts. For example, a surge in user requests may take a few minutes to spool up new serverless workloads before generating additional heat and consume extra energy to cool those CPUs.

You are now solidly in the territory of data science and machine learning, more specifically, time series forecasting. This discipline attempts to transform the data to minimize the impact of time variances. The topic is somewhat dense for this series's depth, and I plan on covering it with a practical example in a future posting. For now, I recommend a look at this great overview article and, if the subject further interests you, consider a deeper dive into this fantastic series of articles on the topic.

Conclusion

This part of the series covered the effects of state machines in training efforts for AIs and a few strategies to address those effects. The lessons covered different phases of a system lifecycle:

Architecture: favor serverless workloads over permanent servers.
Implementation: prefer open source over closed source.
Deployment: use declarative over imperative approaches for changing the system configuration.
Operations: restart all components adjacent to a reconfigured system.
Optimization: Employ techniques of time series forecasting to eliminate or reduce the effects of time on the training data.

The theme of the lessons is to reduce the number, complexity, and duration of state machines across the system. The ultimate goal is to have more substantial and more immediate correlations between events and outcomes in the data center before using their data to train AIs.

Coming up next

In the third part of this series, I will cover the exciting topic of using chaos engineering and unsupervised learning to train machine learning models.

AI Ops Series - Part 1: observability and continuous deployment

2021-04-28T11:07:00.027-07:00

AI Ops is a recurring theme across the industry, with vendors vying for dominance in the application of artificial intelligence across IT operations.

While there is a valid criticism of "AI Ops," often rooted in unrealistic assessments about the potential of anomaly detection, that type of criticism misses the mark on the usage of AI in every other IT scenario that does not involve finding problems.

This posting is the first of a series of candid assessments of what is possible and what is not realistic with the current state of the art. The series reflects my experiences designing machine learning pipelines for natural language processing of technical literature and designing the operational practices for a global SaaS platform based on thousands of systems.

Note: I use "Artificial Intelligence" as the umbrella for Natural Language Processing and Machine Learning algorithms throughout this post.

Business intelligence precedes artificial intelligence: observability.

Before your organization can "hire" an AI to process and act on data, it must ensure it can collect and feed information to that system.

In the operations world, it means taking a hard look at your observability practices and where your team fits in a spectrum ranging from acting on incomplete data to situational awareness.

Imperfect observability means your AI can see fewer data types and fewer examples of those data types. The problems can range from technical difficulty in collecting the data to organizational challenges in getting buy-in and approval to transfer the data.

Let's assume a scenario where an application becomes unresponsive for a few seconds every couple of hours. Your experienced developers ask: "I see traffic to the backend getting interrupted, was there any change to the internal network?"

To answer that question, you need data about that internal network covering the periods before and after the problem started. The data may even narrow down the time window when the trouble started. You would like to see information such as records of change requests, traffic volumes, network connections, and packet errors.

An organization with an established observability culture could answer this and many other exploratory questions through queries on a console or watching live animations of the traffic changes. Without observability telemetry in place, that data-driven process would have to be replaced with experience and (slower) problem-solving techniques.

A time for action: continuous deployment

In that same example of the unresponsive application, let's say the team narrowed down the problem's cause to a patch applied to a network firewall rule on the day before the problems started. In an organization with effective continuous deployment practices, it should be possible to test the theory with little effort (minutes) and high precision (closest possible configuration to the production environment).

Without an established continuous deployment practice, the network team may not be allowed to wait for all your attempts to debug the problem before they need to execute other unrelated change requests to the firewall rules. A new change may either correct or worsen the original problem, triggering another round of troubleshooting under the changed conditions. Additionally, the lack of opportunity to validate a theory can deprive both development and network teams of valuable insight into how the firewall rules affect the system and how to make the overall system more resilient.

Human oversight and manual approval of deployments requested by the AI will be the norm at first but plan to move up in the scale of autonomous systems and increasingly delegate independent action to the AI. At that point, your organization should be starting a closer look at developing internal ML Ops practices and treating the AI as yet another portion of the running system, with its own CI/CD pipeline and operational processes.

Whereas a "test in production" strategy is not mandatory, it tends to yield more realistic results when testing out theories about the system and, more importantly, faster rollout and rollback times.

Supervised, unsupervised, or reinforcement learning

This article has an excellent description of the various training techniques used in machine learning, which I summarize here:

Supervised learning is when you have input and outcome data for a problem. In the previous example about intermittent application crashes, we would have all the data about firewall upgrades, like timestamps, steps performed, and firewall status after the upgrade was complete.

Unsupervised learning is when you only have input data and want the machine to try and cluster the data, detect anomalies, or find associations between the data points. Back to the firewall example, an AI could tell us that the CPU on the network router is 20% over its average within 20 minutes after each upgrade or that the incidence of application crashes is higher within 30 minutes of making changes to firewall rules.

Reinforcement learning is when the system can generate new situations and observe whether it causes good results. Imagine a data center that can vary the output of its cooling units, watch the resulting variation in inlet temperature sensors, and then predict which racks are more prone to failure due to overheating.

A few words about words: NLP (Natural Language Processing)

One of the areas where I spent the most time researching solutions was training an AI to read free-formed text authored by researchers, which tends to be sprinkled with deep contextual references and written without uniform editorial standards.

Time for a pseudo-dialog, where you ask the questions, and I answer them candidly:

Q: "Can I use AI to read incident reports and tell me what caused them?"

A: No. If you are not an R&D shop, do yourself a favor and move on to the next section.

Q: "But we do DevOps, so we are an R&D shop!"

A: That is more of a rebuttal, though I can see you are determined. You have the "D" in R&D, but research in this space is a discipline of its own. I would still move on to the next section.

Q: "Before I move to the next question, can I at least have an AI cluster the words in incident reports to give me some insight?"

A: It will be fun for a couple of hours. Read this article for some ideas on using a TF-IDF clustering algorithm and be sure to make a quick stop at WordArt to order a mug with the results.

If you have not moved on to the next section, time for the long answer to this question, which demands that I switch sides with you at the interview table and ask a few questions of my own:

Q: "Do you have editorial policies for incident report write-ups, comprehensive modeling of attributes for outages, as well as a strong team discipline in filing out these reports?"

Take a moment to read through a handful of incident reports written by your team and figure where they fall in the spectrum between shorthand notes and something you would find in a magazine article. If yours is a typical shop, you will find writing akin to a telegram, written when people are tired and not sure anyone will read or act on them.

Q: "Do you have at least a few hundred examples of these incident reports?"

Machine learning algorithms need to process many examples before creating a trained model that can handle a random sample. The number of starting examples may need to be even higher if the examples do not follow any editorial guidelines.

Q: "Do you have NLP specialists on hand to create a semantic network? Follow-up question, can you give them months to clean up the input text, deploy a machine learning pipeline, iterate over revised versions of that semantic network, and reshape the input data as they progress?"

If, for some miracle of probability, your answers to all previous questions are still positive, then the final question:

Q: "Are you willing to retrain a portion of your operations team in NLP techniques, develop a machine learning pipeline, operate said machine learning pipeline, then turn them loose with a non-refundable check with six zeroes in it?"

At that last "yes," congratulations, you not only have herculean levels of organizational fortitude, but you have got yourself an R&D project in a low-margin OPEX-heavy operation. You are probably part of a large-cap cloud vendor where you can amortize the investment over deployments with tens of thousands of servers. I'll stand corrected and have fun while reading the whitepapers.

**All right, what can I do with it?**

Unsupervised learning is undoubtedly the place to start.

The simple exercise of tallying up all the data sources in the system will already deliver results and increase your business intelligence. Be prepared for gratifying surprises where even your most seasoned system operators learn new relationships between system components and activities.

There are many vendor solutions already doing that sort of analysis with data observed from the system and that is the lowest cost of entry.

There is a long line of products in the space, starting with (in my recollection) Splunk log analysis in the early 2000s and growing into an almost weekly list of new entrants.

Your first candidates should be areas where your system generates a consistent stream of data. The better candidate will form a "Closed Set," otherwise your AI will oblige by telling you there is no strong correlation between any of the values in the data set.

In layman's terms, that closed set means your variables are a function of one another and not completely disjointed. In IT terms and going back to the example where the operations team modified a firewall rule, a "Closed Set" would contain data about firewall change requests rather than about something that is not affected by the network at all, such as the hourly temperature in the data center.

Q: "So, I need to know the correlation beforehand? I wanted the machine to tell me about the correlations."

Yes, you need to know the correlations beforehand. Until artificial general intelligence has evolved to human levels of knowledge acquisition and classification, you need to be the one who knows the causal relation between events, or at least the one who has some idea about them.

Do not be disappointed. Your brain is capable of cognitive equilibrium; AI "brains" are not. Your brain continuously forms a mental model of entities and relationships, classifying new knowledge on top of that mental model and reshaping that model if the classification is not possible. The brain is quite a marvel, indeed, and I wholly recommend a read of "The Development of Thought: Equilibration of Cognitive Structures" by Jean Piaget to appreciate its complexity.

I digress, the learning and reasoning abilities of machines are nowhere close to ours, but there is still the "Knowledge Discovery" discipline of machine learning, which allows them to categorize and classify data. Oh, beware of bias.

Q: "So, you are saying the machine will at least help me find the correlation?"

Yes, it will.

Let's go back to our example of an outage caused by a change to network firewall settings. Suppose you were to feed a machine learning algorithm with a reduced data set, containing metrics about resource utilization and outage information, but without any data about the change requests.

Whereas a machine learning algorithm processing that incomplete data set could not point at the cause of the outages, the statistical analysis could highlight that resource usage in a group of devices decreases by 90% while the system is down. It could also point out that resource usage goes up by 50% in another set of devices. Studying those types of findings increases your understanding of how the system works, which is valuable in itself.

Additionally, the faster you can learn about the lack of strong correlations, the quicker you can rule out invalid assumptions and move on to formulating new theories.

Q: "But this is only helping people better understand the system. When does the AI start doing its job and fixing problems?"

Once again, business intelligence precedes artificial intelligence. The more people understand how the system works and how it evolves, the better your chance of identifying internal processes where the benefit of the AI outweighs the cost of deploying and continuously retraining it.

Initial conclusions

We covered the importance of ensuring good practices in IT operations before spending time and energy trying to deploy an AI that can make those operations better.

Your organization will need an observability discipline covering all systems supporting your initial scenarios and some level of data science literacy amongst the staff in recognizing and plugging holes in the data coverage.

You will also need effective continuous deployment practices to apply the AI's conclusions to the system and support future autonomy powered by ML Ops.

Lastly, AI Ops initially augments your engineering discipline; it does not replace it. Be prepared to develop basic data science data engineering practices in-house, adapt your processes to combine the strengths of humans and AI, and draft a business plan that shows you can offset the investment with savings in operational costs.

Coming up next

In the second part of this series, I cover the challenging aspect of dealing with the lag between cause and effect in IT events and their impact on the deployment of AI Ops.

Configure Docker Desktop on Windows to run with Azure Active Directory

2019-11-14T15:20:00.005-08:00

This article is a generalized version of the FAQ page I wrote for Appsody. I should note that my initial searches for a solution invariably led to this page, which did not work for me.

Docker Desktop on Windows requires access to the computer's filesystems to mount host volumes contained in those filesystems. I explain how to grant that access in the "Shared Drives" section of the "Get started with Docker for Windows" page.

If your container relies on mounted hosted volumes, Docker Desktop must be configured to access the shared drive containing those directories. Once Docker has that access, the next requirement is for that user to have the "Full Control" permission to directories mounted to containers. This last requirement is essential if the directory to be mounted resides within the home directory for the Windows user.

It is sufficient to configure Docker with the same user as the current Windows user in most cases.

However, for users of Windows 10 Enterprise secured with Azure Active Directory (AAD), the AAD user does not reside in the local host. That user may not be accepted in the "Shared Drives" tab of the Docker Desktop "Settings" page, especially if the organization has configured AAD to use authentication mechanisms that do not include user passwords, such as PIN codes.

Workaround for Windows 10 Enterprise secured with Azure Active Directory

If Docker Desktop does not accept your AAD user and password in the "Shared Drives" tab of the Docker "Settings" panel, or you do not have the password for your user, the only known workaround (I know) is to use a separate, local, Windows account to handle the drive sharing and file permissions.

Figure 1 - Workaround for sharing local folders as mounted volumes inside docker containers when working as an AAD user

Assuming the creation of a new user does not violate your organization policies, the workaround is described in Figure 1 and is achieved with the following steps:

Create a new local user account on Windows and use that account and the respective password in the "Shared Drives" tab of the Docker Desktop's "Settings" panel.

Most systems will have only a single shared drive, labeled "C." If your "Shared Drives" list shows additional drive letters, make sure you select the label matching the drive letter for the AAD user's home directory.
Grant the new user from step 1 the "Full Control" permission to the folders you may want to mount into docker containers. You could use the "Security" tab for each folder properties in the File Explorer application, but the quickest and simplest mechanism is to open a "Command Prompt" and issue the following commands to achieve the same results:

REM This has to be the same user applied to the Shared Drive in
REM Docker Desktop
set DOCKER_SHARED_DRIVE_USER=Developer
icacls "c:\directory\to\be\mounted" /grant %DOCKER_SHARED_DRIVE_USER%:(OI)(CI)F

Note that the user in the DOCKER_SHARED_DRIVE_USER environment variable must match the user specified in the Docker Desktop "Shared Drives" tab in the first step. Since the parameters for icacls specify a recursive authorization, granting the permission to a directory will propagate to all subdirectories of that directory.
Validating the correct permissions

The command below is a single line, but may look like multiple lines depending on your browser window size. It will run a docker container that creates (and subsequently removes) a directory in each of the directories modified by the icacls command. The command will indicate the successful outcome of the validation if the shared drive access and the filesystem permissions are set correctly:

docker run --rm -it -v "c:\directory\to\be\mounted":/data alpine /bin/sh -c "mkdir /data/test-write-permission && echo Success; rmdir /data/test-write-permission"

Removing the workaround

If you want to revert the Windows filesystem permissions later, open another "Command Prompt" and execute a "icacls ... /remove" command for each target directory. The operation is recursive and will remove the permissions from subdirectories of those directories as well.

As an example, for the specific instructions on this page, you would execute the following commands:

set DOCKER_SHARED_DRIVE_USER=Developer
icacls "c:\directory\to\be\mounted" /remove %DOCKER_SHARED_DRIVE_USER%

Bioinformatics was awesome, now back to the cloud, which is also awesome

2019-10-17T15:02:00.002-07:00

September marked the end of my longest tenure working on the same project.

Watson for Genomics received my professional attention for nearly 4 years, and the team was part of my extended remote family during that same time.

I wrote about that transition at the time, and looking back, I summarize it as a whirlwind of learning and personal growth.

This was a job where you could be discussing, within the same week, AI bias, how human brains processed text, how the human brain could challenge and reevaluate years of learning in minutes, then switch to a discussion on the business and scientific impact of designing new gene panels for oncology tests.

On a few occasions, we would close the day with a debate on cultural and professional differences that made us laugh and sometimes made us cringe at past gaffes.

A "unique experience" does not come close to making justice to this period in my life. Each colleague earned a cherished and permanent spot in my memories. Some became permanent friends.

Learning how we learn

My first year was spent as the technical leader for a machine learning project training an artificial intelligence to read clinical trial descriptions and scientific research papers.

Unlike my other previous teams, this was not a typical software development shop. Our language processing specialist was a former surgeon, a couple of the people training the system had a degree in biology, and we had several scientists with a diverse range of advanced degrees in the oncology field.

By Thomas

My portion of that work was completed within the year: a pipeline that translated raw text into a neural network of relations between medical concepts. This pipeline could also interpret a mini neural query language that would mine facts from the neural network.

Studying human cognition and inference for an entire year (and realizing some people study it for a lifetime), I developed a decidedly non-technical sense of awe for the brilliance of the human brain. Being in awe may be good to shake your preconceptions, but we had to break it down into progressive levels of cognition:

Level 0 - Humans reading natural language and re-encoding knowledge into tables of facts.

Level 1 - Low-level machine assistance, in the form of keyword searches.

Level 2 - Mid-level machine assistance: identify concepts in written text.

Level 3 - Higher level assistance: given a model of concept types connected by relation types amongst concept types, the machine can identify relations between concepts in written text.

Level 4 - Supervised autonomy, the machine is capable of mimicking humans at level 0, with human experts reviewing small samples of the knowledge to ensure safety.

Level 5 - Full autonomy, machines operate at the same level as humans at level 0, at scales that are impossible for humans.

Some of the takeaways: Level 1 is trivial. Level 2 is not. Level 2 may be a commodity when you have billions of training examples, such as classifying images of cats after millions of people posted captioned photos of cats on social media, but medical research papers are nothing like that. They have no editorial standards across publishers, and there is nothing natural about the language, which is more akin to programming code than anything resembling the sanitized 3rd-grade language commonly found in news articles.

Level 3 is where the field gets magical. Think Piaget's Theory of Cognitive Development. From thoughts to structures, from structures to thoughts. Assimilation and accommodation.

This was the stage where I left the work and moved to the next stage of my participation in the project, while others continued to take the AI forward.

Understand (the data) before you can teach (the machine)

The work in the machine learning pipeline progressed fast, which allowed extra time to be spent on business intelligence outside the sole realm of processing research papers. I recorded some of that work across several entries in this blog, but the entire activity covered much more than clinical trials.

Towards the end, we had a set of dashboards that could show the breakdown of genetic alterations per cancer type, producing valuable insights for different audiences, from researchers trying to understand the system performance against different types of cancers to customers getting to know more about the makeup of cases being treated at their institutions.

Regular interactions with oncology experts often showed that their training gave them unsurpassed knowledge and intuition about their field, but also showed how interactive visualization of data could still help them augment their knowledge. As one of many examples, we had situations where an expert would look at a list of the 20 most common gene alterations found in a set of patients, immediately recognize 19 of them, and walk away wondering about the one they did not expect to see.

By Ars Electronica

The most important lesson (for me) was that developing a system for collecting, visualizing, and statistically processing data is the most valuable investment when creating a system to train an AI... or even humans.

If anyone is about to start an AI project without having spent several months studying the data, that someone is about to spend a lot of time and resources reverse engineering stumbling blocks to the basic math of strictly convex spaces.

In simpler terms, BI precedes AI.

When all is said, and everything is done

Four years is a lot of time, and all of my initial goals had been met, including some of the goals I didn't think I had.

I acquired a basic understanding of genomics, bioinformatics, and the health domain. I also became moderately knowledgeable about machine learning concepts and techniques, developed a new interest for artificial general intelligence, and acquired a fairly advanced applied knowledge of business intelligence techniques, visualization, and statistical processing.

The next stage of expansion of that knowledge was beyond the project scope and market needs. My interest in artificial general intelligence was one or two decades worth of new degrees away, an investment that could not be pursued without close alignment with business needs.

After a few months of talks, we collectively agreed on me going back to my strengths in core technology, where I could help the company the most, in its strategic push for hybrid clouds. To sum up the situation by anti-paraphrasing Dr. McCoy: (Currently) I am an engineer, not a Doctor (though now I want to eventually become one).

By Anthony Starks

Starting in September, I began my new role as a content developer and developer advocate for the open-source Kabanero project and the IBM Cloud Pak for Applications, spending less time writing product code and more time helping developers understand and adopt the technologies in the real world.

On a daily basis, it sometimes means presenting the technology as-is to various audiences, sometimes it means writing focused guidance for a critical use case, sometimes spending a lot of time developing (and testing) workshops for conference attendants, and everything in between. There are some interesting skunkworks projects that break from traditional molds of advocacy, and I hope to be bringing them to reality very soon.

One of my new colleagues has written a nice blueprint for success in this role, which also hints at the kind of awesome people who will be part of this work. I also had the privilege of watching a presentation on developer advocacy by Cami Willians at ATO, which further elaborated on that blueprint with a clear breakdown of areas of activity.

If developers are the new kingmakers, I don't know what that makes us developer advocates, but it may have something to do with helping kings become better kings, so I may order some virtual cards with the title of "Adviser to the Royal Court" or similar.

Back to work.

Correlation between completion of clinical trials and conditions in the trials

2019-04-22T07:58:00.003-07:00

Business Intelligence precedes Artificial Intelligence.

After posting "Applying machine learning to the study of clinical trial data," I thought about additional features that could help train a prediction engine for the final status of a clinical trial.

At the time, enrollment numbers looked like a promising field. Still, later investigation revealed that the strong correlation between those numbers and clinical trial status was only reliably established as the trial approached completion. That correlation is something I covered in "Using Watson Analytics as part of a Machine Learning process." The analysis also surfaced "number of facilities" as another study attribute correlated to overall status (see Figure 1). However, it did not visibly increase the rate of success for the prediction engine.

Figure 1 - Studies with a single facility, on the left, have higher termination rates (in green) than studies performed across multiple sites.

Conditions under study

After experimentation with attributes directly available in the initial data set, it was time to attempt to derive new data to help train machine learning algorithms, starting with the conditions under study.

Some of the characteristics of a condition, such as incidence and mortality, are directly related to the availability of patients meeting enrollment targets. There is also a correlation between the mortality rate for a patient's condition and the grim reality of those patients not surviving the entire duration of a study. There are less obvious attributes, such as the exclusionary effects of complex eligibility criteria, which may make it difficult to qualify patients even when the patient's conditions are not rare in themselves.

Before attempting to dive into each of these possibilities, an easier preliminary step was to demonstrate correlation at an aggregate level, using a new attribute that I straightforwardly named "average completion rate for conditions." The underlying hypothesis is that some conditions may be harder to study than others and that the studies' final status is somewhat correlated to the conditions associated with the study.

The calculation of this new metric for a clinical trial involves two steps: (1) for each condition, calculate the average completion rate for trials where the condition is mentioned, and then (2) for each trial, calculate the average of the metric obtained in the first step across all conditions mentioned in that specific trial (see Figure 2).

Figure 2 - Calculation of "average condition completion ratio for a clinical trial study

For those interested in the details, I uploaded the calculation of the metrics to GitHub (https://github.com/nastacio/clinical-bi/blob/master/docker/aact/docker-entrypoint-initdb.d/init-user-db.sh), but here are the results:

Figure 3 - Completed studies have higher average completion rates for conditions than terminated studies

The columns of charts in Figure 3 correspond to each of the outcomes under analysis: completed, terminated, and withdrawn.

The first row of charts confirmed my assumption about a correlation between clinical trial completion rates (the green dotted line) and specific conditions. Note how the line is consistently higher on completed trials than on terminated trials.

I also included a second row of charts indicating the number of studies involved in each calculation so that it was easier to keep a perspective of the statistical significance of the calculated values. The low number of trials with the final status of "Withdrawn" made me discard conclusions about those studies.

The next visualization (Figure 4) is a more detailed view of the bottom values for average completion, where I wanted to find a pattern between the conditions and their relatively low performance in terms of clinical trial final status.

A quick read of the values on the bar chart on the right surfaced a common pattern, which I later used to colorize the data points to highlight the pattern: the lowest average completion rates in the whole set are associated with cancers. Cancer conditions are identified with an "Is Oncology" value of "1" in that figure.

Figure 4 - Bottom-40 average completion rates for conditions in interventional studies, considering only conditions appearing in 50 or more studies. The bubble chart on the left has a bubble per condition, with the size of the bubble representing the number of studies referencing the condition.

The charts included only conditions appearing in 50 or more studies to reduce the margins of error, but experimentation with higher or lower values did not produce significant changes to the basic shape and contents of the charts.

On the opposite side of those numbers, Figure 5 has the equivalent charts for the top-100 highest values for that same average completion rate metric. Note how only two cancers (mesothelioma and gynecologic cancer) make that list, which will require detailed analysis of their own.

Figure 5 - Top-100 average completion rates for conditions in interventional studies, considering only conditions appearing in 50 or more studies. The bubble chart on the left has a bubble per condition, with the size of the bubble representing the number of studies referencing the condition.

In conclusion, this preliminary data confirms the initial intuition that characteristics of the conditions correlate with the outcome of studies. These initial findings warrant more investigation before drawing conclusions, especially in terms of normalizing condition names and drawing further attributes from those conditions, which often included qualifying terms such as "advanced," "stage III," "mild," and others.

Clinical Trials visualization with Cognos Dashboard Embedded in the IBM Cloud

2018-09-11T16:06:00.002-07:00

Disclaimer: I am a software engineer at IBM with access to multiple IBM products, and wrote the content mostly as personal notes to share with a larger team. That said, *as of writing of this entry*, the content mentioned here can be hosted and accessed for free in the IBM Cloud, provided that you do not start more than 50 dashboard sessions per month.

Cognos Analytics is IBM's flagship data visualization tool and has been my companion in charting business intelligence and data insights for the Watson for Genomics cloud offering for the past 15 months.

Recently, IBM has launched Cognos Dashboard Embedded, a lightweight alternative to the hosted Cognos Analytics on Cloud product, and I wondered whether that cloud-native approach could be useful for a practical scenario involving a large database.

The domain

I have been doing a lot of side work with clinical trial visualizations this year, which I consider a data-rich field that can benefit greatly from even the mildest efforts to represent the data in aggregate form.

I had written before about how to load the excellent AACT database into the IBM Cloud, as a Docker container running the Postgresql dump of AACT. Later I wrote about using that database to power a predictive engine for clinical trial completion. I also wrote about how to use Watson Analytics to obtain insights from a subset of clinical trial data, and then quickly create useful visualizations from that data.

The technical bits

If you want to replicate this dashboard on your own, you will need a few things:

An account in the IBM Cloud (https://console.bluemix.net)
Create an database warehouse containing the data, a more involved process described in detail in this other blog entry.
Create a Cognos Dashboard Embedded service instance. It is a simple step described in this posting.
Attach the two services to the application sample created for this entry, which is posted on github: https://github.com/nastacio/cognos-de

Assuming you took care of steps #1 and #2, the following sequence of commands will give you a running application that can render a dashboard of clinical trials studies.

    # clone and build the application, you should have git, a JDK,
    # and Maven installed
    git clone https://github.com/nastacio/cognos-de.git
    cd cognos-de
    mvn package

    # Push the application to the IBM cloud
    # Note the --random-route parameter, which means the IBM Cloud
    # will pick the URL for the application automatically and display
    # it once the application push is complete
    # Also note the --no-start parameter, as we don't want the

    # application to start just yet.
    ibmcloud app push cognos-de -p target/cognos-de.war -m 256M -b liberty-for-java --random-route --no-start

    # Create the Cognos service instance.
    ibmcloud resource service-instance-create ctgov-cognos-de dynamic-dashboard-embedded lite us-south

    ibmcloud resource service-alias-create ctgov-cognos-de --instance-name ctgov-cognos-de -s dev

    # Bind the 2 service instances to the application
    # Note that the DB2 Warehouse instance will have been created according to
    # http://sourcepatch.blogspot.com/2018/09/transferring-data-from-the-aact.html
    ibmcloud service bind cognos-de ctgov-cognos-de
    ibmcloud service bind cognos-de ctgov-db

    # Start the application
    ibmcloud app start cognos-de

Once the application is started, you should see the following output, with the "routes" parameter indicating the URL for your own application:

    name:              cognos-de
    requested state:   started
    instances:         1/1
    usage:             256M x 1 instances
    routes:            cognos-de-active-grysbok.mybluemix.net
    last uploaded:     Tue 11 Sep 17:38:22 EDT 2018
    stack:             cflinuxfs2
    buildpack:         liberty-for-java
    start command:     .liberty/initial_startup.rb

         state     since                  cpu      memory           disk         details
    #0   running   2018-09-11T21:42:04Z   159.9%   132.9M of 256M   208M of 1G

The building of the dashboard

I relied on the main documentation and tried to stick to it for the most part. It does a fair job of guiding users from the beginning, though it understandably focuses on getting a dashboard up and running versus the dashboard you would probably like to build.

I like the idea of their interactive web page, from where you can see the Javascript required for building an application containing the dashboard, and then bring up the editor on the right side of the screen. Non-technical users will be quickly discouraged between the tutorial and the expectation of a few trips to web consoles, dealing with JSON payloads, table models and others. For those users, the best approach for a user-friendly experience with Cognos is still Cognos Analytics on Cloud or the Watson Studio product.

If you are building a dashboard that relies on a hosted database, then you will inevitably have to muscle through the creation of a long JSON file containing the metadata for that database. After you create a new dashboard in the interactive page and then add the data source to it, you can finally use the graphical editor to pull widgets into the page and associate table columns to those widgets.

Once you are satisfied with the results, it is time to issue the "Get the dashboard spec" Javascript call from the web console, and then get a copy of the dashboard spec for later use with the "Open dashboard call".

Overall, the service works as advertised and offers a powerful set of visualization at a low overall cost of ownership, but the lack of the "Properties" tab in the editor is a glaring omission in what should be a complete graphical package. That tab, present in the final product, is what allows the dashboard designer to customize most of the visual aspects of the widgets on the page, such as the display of axis labels, titles, overlay values, color palettes, and background colors.

My workaround came in the form of having access to a similar dashboard spec from a hosted version of another dashboard, which I used to reverse-engineer the JSON values required to set borders and titles on the widgets. Most system integrators would not have access to such a "donor" file or would be somewhat reluctant to make changes that are not in the official documentation for Cognos Dashboard Embedded.

You can see the final specification for the dashboard at https://github.com/nastacio/cognos-de/blob/master/src/main/resources/dashboard-spec.json

Update on 9/14: My colleague, David Murrels, pointed me at this other project for a React component for Cognos Dashboard Embedded.

The dashboard itself

I focused primarily on representing the progression of studies over the years (by starting date), with breakdowns by study type, overall status, phase, and conditions. I also added a number of dashboard filters at the top, so that it was possible to apply global filtering criteria, such as oncology-related trials or trials from a specific sponsor.

I am thoroughly impressed with the overall performance for the Cognos and DB2 Warehouse combo. Even complex combinations of selections on the screen, involving 4 dimensions and multiple selections within those dimensions, rendered smoothly without any kind of delay. Granted, there are only 900K rows in that database, but it is remarkable for a freemium entry plan running on a shared server instance.

Transferring data from the AACT database to a DB2 Warehouse cloud instance

2018-09-11T10:46:00.002-07:00

This post was written as a technical deep dive into the analytics database referenced in the entry titled "Clinical Trials visualization with Cognos Dashboard Embedded services in the IBM Cloud".

The specific choice of database was not central to that posting, but the performance characteristics of the database were an important consideration to achieve good performance. Whereas a transactions-oriented database may be adequate for a dashboard containing a handful of widgets displaying a small data set, heavier utilization of larger data sets often require a standalone warehouse tuned for analytic workloads.

Cognos Dashboard Embedded's architecture delegates data storage to embedding applications, which allows the choice of using connections to relational databases or specifying CSV files. The limit for CSV file sizes is a generous 1 GB, but that also means you need to host the file in a URL-accessible location, while also potentially facing warm up costs for the Cognos backend infrastructure to download, process, and cache the file. In my previous experiences with Cognos and CSV files, the performance for such approach was not that great once the files got past 100 MB in size.

With AACT being available as a Postgresql database hosted on the cloud, one would think that option would be an immediate fit, but I quickly realized two major impediments that forced me to abandon that idea very early in the prototype:

Cognos Dashboard Embedded does not offer the Data Modeling view available in the full Cognos Analytics product, which prevented any possibility of joining the various tables by study identifier. Since I wanted to be able to navigate from studies to conditions and interventions, I could not use the AACT multi-table database structure in its standard topology.
Postgresql is never tuned for data analytics right out of the box and the AACT cloud database is no exception. You will not notice it for most workloads where you are running moderately complex SQL queries, but dashboard demands are unforgiving between their initial swarm of queries to populate an initial view, the subsequent wince-inducing WHERE clauses created by ad-hoc screen selections, and the end-user expectations that these queries will be serviced instantaneously to result in a fluid experience.

At this point, the solution looked a bit daunting: use the DB2 Warehouse service in the IBM Cloud, which meant converting the relevant portions of the AACT psql dump to a DB2 table.

As it turns out, a bit of research showed that the DB2 Warehouse offered a complete (and outrageously large) Docker-based client that I could augment with a postgresql client, then use a long string of shell scripting to load the warehouse with the results of a query against the AACT database.

That augmented client is posted to github.com, as well as the data transfer script. The data is loaded from 4 different tables: studies, conditions, calculated_values and interventions, then merged into a single table in the warehouse.

Assuming you have the IBM Cloud CLI, git and docker installed, these would be the setup steps for the warehouse instance and the final activation for the data transfer:

git clone https://github.com/nastacio/clinical-bi.git
cd clinical-bi/docker/build

mkdir -p ~/etc

# Update on 12/13/2020: The DB2 Warehouse cloud instances no longer have a 
# free plan, so you can use these instructions to create a simple DB2 instance
# which still has a free plan
cloud_region=us-south
ibmcloud resource service-instance-create ctgov-db  dashdb-for-transactions free ${cloud_region} -g default
ibmcloud resource service-key-create ctgov-key --instance-name ctgov-db

# The next instructions are common for either DB2 offering
ibmcloud resource service-key ctgov-key -g default --output json  > ~/etc/db2wh.aact.credentials.json


...

# This may take a few minutes, as there are roughly 5GB worth of data 
# and libraries to be downloaded

docker-compose up -d aact db2wh
docker-compose logs -f & 

# wait for the aact database to report being ready to receive connections, 
# it may take a few minutes;
...
db2wh_1          | #######################################################################
db2wh_1          | ###  IBM Db2 Warehouse client container was deployed successfully   ###
db2wh_1          | #######################################################################
...
aact_1           | LOG:  database system was shut down at 2018-09-10 22:21:05 UTC
aact_1           | LOG:  MultiXact member wraparound protections are now enabled
aact_1           | LOG:  database system is ready to accept connections
aact_1           | LOG:  autovacuum launcher started
...

# finally, execute the transfer command

docker-compose exec db2wh /usr/local/bin/db2wh-etl.sh

If everything goes well, you should see an output like this:

INFO: Validating DB2 connectivity parameters

===============================================================================
db2cli writecfg completed successfully.
===============================================================================


===============================================================================
db2cli writecfg completed successfully.
===============================================================================


===============================================================================
db2cli writecfg completed successfully.
===============================================================================


===============================================================================
Client information for the current copy:
===============================================================================

Client Package Type       : IBM Data Server Client
Client Version (level/bit): DB2 v11.1.3.3 (s1803021700/64-bit)
Client Platform           : Linux/X8664
Install/Instance Path     : /opt/ibm/db2/V11.1
DB2DSDRIVER_CFG_PATH value:
db2dsdriver.cfg Path      : /mnt/clientdir/clienthome/db2inst1/sqllib/cfg/db2dsdriver.cfg
DB2CLIINIPATH value       :
db2cli.ini Path           : /mnt/clientdir/clienthome/db2inst1/sqllib/cfg/db2cli.ini
db2diag.log Path          : /mnt/clientdir/clienthome/db2inst1/sqllib/db2dump/db2diag.log

===============================================================================
db2dsdriver.cfg schema validation for the entire file:
===============================================================================

Success: The schema validation completed successfully without any errors.

===============================================================================
db2cli.ini validation for data source name "aact":
===============================================================================

Note: The validation utility could not find the configuration file db2cli.ini. 
The file is searched at 
"/mnt/clientdir/clienthome/db2inst1/sqllib/cfg/db2cli.ini".

===============================================================================
db2dsdriver.cfg validation for data source name "aact":
===============================================================================

[ Parameters used for the connection ]

Keywords                  Valid For     Value
---------------------------------------------------------------------------
DATABASE                  CLI,.NET,ESQL BLUDB
HOSTNAME                  CLI,.NET,ESQL dashdb-entry-yp-dal10-01.services.dal.bluemix.net
PORT                      CLI,.NET,ESQL 50001
SECURITYTRANSPORTMODE     CLI,.NET      SSL

===============================================================================
Connection attempt for data source name "aact":
===============================================================================

[SUCCESS]

===============================================================================
The validation is completed.
===============================================================================

INFO: Extracting CTGOV dashboard data.
INFO: Creating DB2 Warehouse table.
IBM DATABASE 2 Interactive CLI Sample Program
(C) COPYRIGHT International Business Machines Corp. 1993,1996
All Rights Reserved
Licensed Materials - Property of IBM
US Government Users Restricted Rights - Use, duplication or
disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
> CREATE TABLE CTGOV (NCT_ID VARCHAR(16),OVERALL_STATUS VARCHAR(64),...
The SQL command completed successfully.
>
Loading messages into DB2 warehouse at dashdb-entry-yp-dal10-01.services.dal.bluemix.net. Logs being written at [/mnt/clientdir/clienthome/db2inst1/workdir/load.log]

   Database Connection Information

 Database server        = DB2/LINUXX8664 11.1.9.0
 SQL authorization ID   = DASH5396
 Local database alias   = BLUDB


Number of rows read         = 909432
Number of rows skipped      = 0
Number of rows loaded       = 909432
Number of rows rejected     = 0
Number of rows deleted      = 0
Number of rows committed    = 909432


DB20000I  The SQL command completed successfully.
DB20000I  The TERMINATE command completed successfully.

And you should see a storage utilization around 5% in the service instance dashboard in the IBM Cloud console:

Using Watson Analytics as part of a Machine Learning process

2018-04-01T18:14:00.001-07:00

Machine Learning often boils down to detecting patterns ... if there are any to be found.

Though we want the machine to do the heavy lifting in detecting and reusing the patterns, we also need to understand the domain and its data in order to choose the features for the training. That understanding is equally crucial to determine whether the data is correct and complete in the first place.

Before getting into the main topic, I admit to being prone to using PivotTables in MS-Excel when trying to understand the interplay between different columns in a data set.

I know there are strong opinions (and some zealotry) around using MS-Excel in statistical analysis, but judged on its merits, it is hard to beat the point-and-click usability of grouping the data from dozens of different perspectives within minutes, specially when you consider the price point.

The pivot tables run out of breath when you are past the point of understanding the data at a macro level and want some form of insight about the data.

Enter Watson Analytics. In full disclaimer, I just received a professional account from IBM as part of my regular work. A couple of hours exploring the tool proved it was no I-have-to-use-it penalty box, quite the contrary.

Insight in hindsight

Watson Analytics took only 10 minutes to make me regret the choice of a training feature used in my previous blog entry ("Applying machine learning on the study of clinical trial data") .

I exported the data to a CSV file and imported it into Watson Analytics. The import gave me a 60% ranking for the quality of the data, which I presume is calculated primarily based on empty fields.

Clicking on the icon for the data set, I was presented with a list of free-formed questions I could choose, none of which involved the target column: overall_status. I then typed "What drives overall_status?", hit Enter, and was greeted with the following set of alternatives to explore the answer:

Free-formed requests for insights

I selected the first (and immediate) answer to my question, though I was tempted to follow my nose exploring the others. The next screen presented me with an exhaustive list of the predictive strength of other columns variables for overall_status, including combinations of two or more columns:

Assessment of drivers for a given metric

I expected enrollment targets to be a good predictor of overall status, since large centers have more access to patients, solid reputations, as a well as more resources to run the clinical trials. It was intriguing to see "sponsor type" as a co-driver, so I clicked on the "sponsor_type and enrollment" slot to drill down into it:

Graphical breakdown of relationship amongst different metrics

At this point, I am a handful of clicks and one typed question into the experience, and already looking at a new insight (for me) : a disproportionate number of terminated clinical trials had small values (less than 12 ) for the enrollment column. Going back to my AACT database instance, I isolated a few clinical trials matching that criteria and read up on their characteristics in clinicaltrials.gov, eventually realizing that there was no way the sponsors for clinical trials with complex eligibility criteria could have possibly set out to have such low number of patients in the studies. Inspecting the 'studies' table a little further, I realized that the enrollment column was further qualified by another column: enrollment_type.

enrollment_type can be either 'Anticipated' or 'Actual', with 95% of the clinical trials in my data set having the enrollment type being 'Actual'. Pause for the heart-sinking realization that enrollment data for those clinical trials is updated over time to reflect the number of patients currently enrolled in the clinical trial at the time of its completion or termination. This finding not only explained the low numbers for many terminated clinical trials, but also meant enrollment was largely an output field that could not be used to make predictions about the outcome of ongoing trials.

People familiar with clinical trials may be shaking their heads at this point: "of course that is what 'enrollment' means", though I could also make the case that the number gradually shifts from an output to an input as the clinical trial progresses. In other words, it may still be possible to use it in combination with the elapsed time and phase of a clinical trial, but that deserves a paper of its own.

A new beginning

Once I eliminated the enrollment column from the training model, a new run of the predictor had precision cratered to baseline levels and I was back to square zero. The silver-lining was that the exercise could get restarted on a more solid ground: by analyzing the data before choosing the best features for training a model.

Without the enrollment column, Watson Analytics showed me this list of predictors:

Drivers after removing enrollment column

I did not like to see the upper limit of 75% being discouragingly close to the baseline (in the 70%-75% range) , but at this point I did not want to let my newfound set-back with the enrollment column detract from further exploring the tool.

I chose the top pair of drivers (start_epoch and number_of_facilities) and Watson Analytics surfaced yet another insight I did not have, annotated with red in the next screenshot:

Higher rate of terminations for trials conducted in a single facility

The termination rate for trials is noticeably higher for studies involving a single facility. Looking at the next column (2-4 facilities) , the termination rate is a bit lower than for single facilities, but still noticeably higher than the termination rate for trials running in 5 or more facilities. In theory, this can be explained by the fact that the number of facilities correlates with larger organizations. As noted before, these organizations not only have the substantial material resources for planning and conducting the studies across a larger number of locations, they also have more access to patients required for the studies.

A new level of productivity

I now have a new batch of notes on choosing features to reboot the earlier exercise in predicting the final status of a clinical trial. It is also clear that exploring the data with a tool that can quickly group the data across multiple dimensions is almost a necessity before developing a machine learning prototype.

More than a competent assistant in exploring and finding insights within data, I almost forgot to mention Watson Analytics' flexible visualization mechanism, which stands on its own. With another couple of extra variables added to the previous chart, I had an insightful visualization of agency types, study types, volumes of studies and their final overall status, showing that corporations tend to be more effective in completing studies than other types of sponsors:

Flexible composition of new visualization through drag-and-drop of columns

I am sure there is more to be explored, but I am already quite impressed with what can be done with the tool within a couple of hours and without external training.

Applying machine learning to the study of clinical trial data

2018-03-25T09:38:00.000-07:00

Documenting my personal side project in studying and visualizing the data about clinical trials.

Disclaimer

In full disclosure, given the time I have for this project, the work is more focused on the technical building blocks than on the disciplined statistical science you will see in papers like "Estimation of clinical trial success rates and related parameters".

A technical exercise

As the basis for the exercise, I set out to answer a hypothetical question that could be useful for both patients and sponsors of clinical trials: "Will the clinical trial be completed or terminated?".

Whereas the final overall status of a clinical trial is not necessarily an indication of success or failure for either patient or sponsors, there is certainly some correlation. In terms of technology, I wanted to use TensorFlow to process a subset of structured fields in the clinical trial data to train a machine learning model that can predict the final overall status of the trial.

The data

I focused on a training set with interventional trials related to oncology, starting from 2009. I must give another shout out to the Clinical Trials Transformation Initiative (CTTI) folks for building the AACT database, which considerably shaved preparation time.

With Python and Pandas in tow, I created the data import module to query a hosted AACT database on IBM Cloud (See "Deploying the AACT database on a hosted Kubernetes cluster"). It was equally invaluable to have Adminer on hand to explore the database contents and prototype some of the SQL statements directly.

With the data source in place, it was time for the second most expensive piece: converting all the strings in the responses to numbers, bucket columns, identity columns, and embedding columns in TensorFlow.

With all the building blocks in place, it was time to spend countless hours adding and removing features, then running the training and estimation to find out which features worked better than others to increase the accuracy of the prediction.

(Current) Results

Given that a high percentage of clinical trials ends up being completed - in the range of 70% to 75% depending on the selection criteria for the trials - there was not a whole lot of room for improvement over the baseline, and at some point the accuracy of this new model maxed out at 81.5% against a baseline of 72.8% for the test set, yielding a gain of 12% over simply guessing 'Completed' for all trials.

To recreate the results, if you have docker installed and running, simply execute this command from a terminal:

docker run --rm -it dnastacio/tensorflow-ct

After a couple of minutes of data loading and training, you should see the accuracy for the trained model and the prediction for a few examples on the screen.

Note that I am still using random slices of the data set on each run to avoid optimizing the model for a given slice, so results will vary across runs. I still need to go back and pick a set of clinical trials as a fixed ground truth.

Conclusion and next steps

Though early results are not a major gain over the baseline, that is expected at this stage of development. This is still largely a exercise to assess and integrate building blocks into a solution for this kind of problem, such as transforming data from relational databases into usable tensors inside TensorFlow. I still want to go through a few more exercise steps: (1) building a deep-wide model to separate dense from sparse features in the current set and (2) using word2vec on the contents of clinical trial description and eligibility criteria.

Beyond the technology, I have also invested time reading articles on the causes of clinical trial termination and talking to experts in the field, which points at promising additions to the model. Unfortunately, some of these features are not well-populated in the clinical trials database, so that their inclusion will require concerted and localized work to clean up the data series for each feature.

One simple example of such a feature is the allocation strategy for the clinical trial (randomized vs. non-randomized allocation across different arms) . Despite consensus from experts about randomized allocation being a big factor in the patient's decision to join or leave a clinical trial, there was no gain in accuracy of prediction after adding that field to the model. I then charted the distribution of the input values in the database, just to find out that it is empty on the majority of clinical trials. Cleaning up that one data series can easily become a machine learning project of its own.