There are quite a few blog posts out there detailing this, but none of them seem to be up to date for use with the HCL style syntax, introduced in Packer 1.5, which has a number of advantages over using the original JSON syntax, namely that it is much more human friendly, flexible, and easy to work with.

So in this post I will share a very quick and dirty example packer configuration that uses the HCL syntax to create, provision and upload a Windows server AMI to AWS.

You’ll need the following bootstrap_win.txt file in place in order for Packer to be able to provision the server using PowerShell commands over WinRM.

<powershell>

write-output "Running User Data Script"
write-host "(host) Running User Data Script"

Set-ExecutionPolicy Unrestricted -Scope LocalMachine -Force -ErrorAction Ignore

# Don't set this before Set-ExecutionPolicy as it throws an error
$ErrorActionPreference = "stop"

# Remove HTTP listener
Remove-Item -Path WSMan:\Localhost\listener\listener* -Recurse

# Create a self-signed certificate to let ssl work
$Cert = New-SelfSignedCertificate -CertstoreLocation Cert:\LocalMachine\My -DnsName "packer"
New-Item -Path WSMan:\LocalHost\Listener -Transport HTTPS -Address * -CertificateThumbPrint $Cert.Thumbprint -Force

# WinRM
write-output "Setting up WinRM"
write-host "(host) setting up WinRM"

cmd.exe /c winrm quickconfig -q
cmd.exe /c winrm set "winrm/config" '@{MaxTimeoutms="1800000"}'
cmd.exe /c winrm set "winrm/config/winrs" '@{MaxMemoryPerShellMB="1024"}'
cmd.exe /c winrm set "winrm/config/service" '@{AllowUnencrypted="true"}'
cmd.exe /c winrm set "winrm/config/client" '@{AllowUnencrypted="true"}'
cmd.exe /c winrm set "winrm/config/service/auth" '@{Basic="true"}'
cmd.exe /c winrm set "winrm/config/client/auth" '@{Basic="true"}'
cmd.exe /c winrm set "winrm/config/service/auth" '@{CredSSP="true"}'
cmd.exe /c winrm set "winrm/config/listener?Address=*+Transport=HTTPS" "@{Port=`"5986`";Hostname=`"packer`";CertificateThumbprint=`"$($Cert.Thumbprint)`"}"
cmd.exe /c netsh advfirewall firewall set rule group="remote administration" new enable=yes
cmd.exe /c netsh firewall add portopening TCP 5986 "Port 5986"
cmd.exe /c net stop winrm
cmd.exe /c sc config winrm start= auto
cmd.exe /c net start winrm

</powershell>

Here is what the full Packer HCL configuration looks like.

variable "aws_region" {
  type    = string
  default = "us-west-2"
}

variable "instance_type" {
  type    = string
  default = "t3.medium"
}

variable "subnet_id" {
  type = string
}

variable "vpc_id" {
  type = string
}

variable "ami_users" {
  type = string
}

source "amazon-ebs" "windows_server" {
  ami_description             = "A custom Windows Server AMI"
  ami_name                    = "windows-example"
  ami_users                   = ["${var.ami_users}"]
  associate_public_ip_address = true
  communicator                = "winrm"
  instance_type               = "${var.instance_type}"
  region                      = "${var.aws_region}"
  force_deregister            = true
  force_delete_snapshot       = true
  source_ami_filter {
    filters = {
      architecture        = "x86_64"
      name                = "Windows_Server-2019-English-Full-ContainersLatest-*"
      root-device-type    = "ebs"
      virtualization-type = "hvm"
    }
    most_recent = true
    owners      = ["801119661308"]
  }
  subnet_id      = "${var.subnet_id}"
  user_data_file = "./bootstrap_win.txt"
  vpc_id         = "${var.vpc_id}"
  winrm_insecure = true
  winrm_port     = 5986
  winrm_use_ssl  = true
  winrm_username = "Administrator"
}

build {
  sources = ["source.amazon-ebs.windows_server"]

  # Extra configuration
  provisioner "file" {
    destination = "C:\\ProgramData\\someconfig.txt"
    source      = "./myconfig.txt"
  }

  provisioner "powershell" {
    # Reinitialize the server to generate a random password on first boot
    inline = [
      "C:\\ProgramData\\Amazon\\EC2-Windows\\Launch\\Scripts\\SendWindowsIsReady.ps1 -Schedule",
      "C:\\ProgramData\\Amazon\\EC2-Windows\\Launch\\Scripts\\InitializeInstance.ps1 -Schedule",
      "C:\\ProgramData\\Amazon\\EC2-Windows\\Launch\\Scripts\\SysprepInstance.ps1 -NoShutdown"
    ]
  }
}

Notice the two provisioners. The first enables a way for you to inject local configurations into the image before baking it, useful for adding extra configurations. The second is specific to AWS Windows Server images but essentially allows the machine to act like it is being booted for the first time, using the SendWindowsIsReady.ps1, InitializeInstance.ps1 and SysprepInstance.ps1 scripts. These scripts are important pieces to ensuring that this AMI can be created and started the exact same way every time.

Some of the configuration options may not be necessary so you will need to play around with the configuration to make it suit your needs. For example, you may want to create a unique image on every build, which can be done using timestamps. If you use unique identifiers, the force_deregister and force_delete_snapshot options can be omitted.

Likewise, since this Packer image is specific to Windows, it uses a number of winrn_ options, which would be replaced by ssh_ options if this were being provisioned for Linux.

One other trick that I found to be helpful was setting some of these variables on the fly via a script. You can use PKR_VAR_<var> via shell to set some environment variables. This is especially useful when the configuration needs to be shared across different environments for things that change, e.g. vpc_id and subnet_id.

The post Create Windows Server 2019 AMIs using Packer first appeared on Practical System Administration.

In this post I will describe some of the interesting discoveries I have made for a recent side project I have been working on, which include a few nice discoveries to automate and manage WordPress deployments with Kubernetes.

Automating WordPress

I am very happy with the patterns that have emerged from this project. One thing that I have always struggled with (I’m sure others have as well) in the world of WP has been finding a good way to create completely reproducible and immutable code and configurations. For example, managing plugins and themes has been a painful experience because WP was designed back in a time before configuration as code and infrastructure as code. Due to this different paradigm, WP is usually stood up once, then managed via it

Tools have evolved since then and a perfect example of one of the bridges between the old and new way is the wp-cli. The wp-cli is basically a way to automate all kinds of things you would otherwise do in the UI. For example, wp-cli provides a way to manage plugin and themes, which as I mentioned has been notoriously difficult to do in the past.

The next step forward is the combination of a tool called bedrock and its accompanying way of modifying it and building your own Docker image. The roots/bedrock method provides the wp-cli in the build scripts so if needed, you can automate tasks using extra entrypoint scripts and/or wp-cli commands, which is just a nice extra touch and shows that the maintainers of the project are putting a lot of effort into it.

A few other bells and whistles include a way to build custom plugins into Docker images for portability rather than relying on some external persistent storage solution which can quickly add overhead and complexity to a project, as well as modern tools like PHP Composer and Packagist which provide a way to install packages (Composer) and a way to manage WP plugins via the Composer package manager (packagist).

Sidenote

There are several other ways of deploying WP into Kubernetes, unfortunately most of these methods do not address multitenancy. If multitenancy is needed, a much more complicated approach is needed involving either NFS or some other many -> many volume mapping.

Deploying with Kubernetes

The tricky part to all of this is the fact that I was unable to find any examples of others using Kubernetes to deploy bedrock managed Docker containers. There is a docker-compose.yaml file in the repo that works perfectly well, but the next step beyond that doesn’t seem to be a topic that has been covered much.

Luckily it is mostly straight forward to bring the docker-compose configuration into Kubernetes, there are just a few minor adjustments that need to be made. The below link should provide the basic scaffolding needed to bring bedrock into a Kubernetes cluster. This method will even expose a way to create and manage WP multisite, another notoriously difficult aspect of WP to manage.

https://gist.github.com/jmreicha/aae3ce024c13be1c561189946f1a0efc

There are a couple of things to note with this configuration. You will need build/maintain your own Docker image based off the roots/bedrock repo linked above. You will also need to have some knowledge of Kubernetes and a working Kubernetes cluster in place. The configuration will require certificates, and DNS so cert-manager and external-dns will most likely need to be deployed into the cluster.

Finally, in the configuration the password, domain (example.com), environment variables for configuring the database and Docker image will need to be updated to reflect your own environment. This method assumes that the WordPress database has already been split out to another location, so will require the Kubernetes cluster to be able to communicate with wherever the database is hosted.

To see some of the magic, change the number of replicas in the Kubernetes manifest configuration from 1 to 2, and you should be able to see a new, completely identical container come up with all the correct configurations and code and start taking traffic.

Conclusion

Switching to the immutable infrastructure approach with WP nets a big win. By adopting these new methods and workflows you can control everything with code, which removes the need for manually managing WP instances and instead allows you to create workflows and pipelines to do all of the heavy lifting.

These benefits include much more visibility in controlling changes, because now Git becomes the central source of truth which allows you to get a better picture of the what, when and why than any other system I have found. This new paradigm also enables the use of Continuous Integration as it is intended – the automatic builds and deploys because of Docker and Kubernetes integrations producing immutable artifacts (Docker), and deployments (Kubernetes manifests) create a clean and simple way to manage the aspects of running the WordPress site.

The post Immutable WordPress Installations with Kubernetes first appeared on Practical System Administration.

What begins with “T”? I have been thinking about various ways of testing infrastructure and resources lately and have been having a difficult time parsing out the various tools that are available. This post is meant to be a reference for “finding the right tool for the right job” as part of testing various infrastructures.

Often times when you start talking about testing, you will hear about the testing pyramid, which is described along with some other interesting aspects of testing Terraform in this blog post, it covers a lot of the pitfalls and gotchas you might run into.

My aim originally was to find a good tool for unit testing Terraform and as part of that adventure have uncovered a number of other interesting projects, that while not directly applicable, could be very useful for a number of different testing scenarios. Read the above blog post for more info but suffice to say, unit testing (the bottom layer of the testing pyramid) is quite difficult to do with Infrastructure as Code and Terraform and at this point is mostly not a solved problem.

Here is the list of tools that I uncovered in my research. Please let me know if there are any missing.

Terratest

Terratest is written and maintained by the folks at Terragrunt and it provides a comprehensive testing experience for deploying Infrastructure as Code, testing that it works as expected and then tears down the IaC when it is finished. From what I can tell this is probably the most comprehensive testing tool out there, still doesn’t cover unit tests, but is a great way to ensure things are working end to end.

Kitchen-Terraform

Follows the BDD philosophy and spins up, tests and spins down various Terraform resources. Works in a similar fashion to Terratest to bring up the environment, test and then tear things down.

serverspec

This testing tool is one of the original tools I ran across when originally mulling over the idea of unit testing infrastructure back in the days when configuration management tools like Chef, Puppet and Salt ruled the earth.

inspec (aws-resources)

Very similiar to awspec, this tool provides a framework for testing various AWS resources. This one is nice because it uses inspec to build on and so it has a lot of extra capabilities.

awspec

Very similar to the other “spec” tools, awspec is built in the same style as serverspec/inspec and provides a very nice interface for testing various AWS resource types. Since it is modeled after serverspec, you will need to deal with Ruby.

Testinfra

Another Python based unit testing style framework. This tools varies from Terraform Validate in that it mainly focuses on testing the lower level server and OS, but it does have integration for testing a number of other things.

terraform-compliance

A nice tool for testing (and enforcing Terraform compliance rules).

Terraform Validate

This is a nice tool for expressing various test conditions as compliance, especially if you are already familiar with Python and its testing landscape. This tool parses configs using pyhcl and allows you to write familiar unittest style tests for Terraform configurations.

Pulumi

This one is a bit of a wild card but could prove interesting to some readers. Pulumi provides direct integration into your language of choice (Python for me) to enable you to write pulumi code using language native unit tests. Obviously Pulumi isn’t Terraform but I had to mention it here because there is a significant amount of crossover between the tools.

The post Terraform Testing Tools first appeared on Practical System Administration.

I have been experimenting with Linux on a recently acquired Lenovo X1 Carbon and as part of the process I have been exploring and testing out various productivity tools, including terminal emulators.

The first thing I discovered in this process is that there’s a lot of terminals out there. And they’re all slightly different, and in my experience, almost none of them were able to do all of the tweaks I like.

Here is the list of terminals I have tested out (so far).

• Alacritty
• Gnome Terminal
• Kitty
• Simple Terminal (ST)
• Terminator
• Termite
• Tilix
• urxvt

As you can see, that is a bunch of terminals. To keep things short, in my exploration and testing, the best 3 I found were as follows.

Alacritty – 3rd

It is fast, defaults are great, works perfectly with the tiling window managers. This one would would probably have been first on the list if it was able to provide a blinking cursor but as I found with many of the options there seemed to almost always be a gotcha.

A few high points for this terminal, written in Rust, it uses the GPU to offload rendering, cross platform, and maybe my favorite feature, the configuration file is yaml based. Docs are also good and the community is really taking off so I have a strong feeling this one will continue improving.

urxvt – 2nd

The biggest problem with this terminal for me is that it is painful to configure. There are no preferences configured out of the box, which to some is preferred, but also, digging through old blog posts and perl scripts is not how I like to spend my time.

This terminal is second on the list because it was the only other terminal that was able to do all of the small little tweaks and adjustments that suit my preferences. And it is fast and light weight, which are good things.

Here is the configuration I ended up with if interested.

Kitty – 1st

This terminal was a clear winner. It was able to do all of my custom tweaks and settings and because it has nice defaults I only needed to add about 10 lines of extra configuration.

This terminal has a lot of other stuff going for it. It is written in Rust, which makes it fast, it uses the GPU to offload rendering, which also makes it fast, easy to configure, and for the most part just works. Only gotcha I found was that I needed to explicitly turn on copy to selection in my configuration, but that was easy.

Here is the configuration I ended up with if interested.

Conclusion

I plan on keeping this list updated to some extent as I find more Terminals to try out. As you can see, there are many different options and seem to be more and more all the time.

Your experience may differ so obviously take these musings with a pinch of salt, and please do look through the various options and try things out to see if they will work for you. That said, I do think I have a specific enough use case that these recommendations should be helpful in guiding most users.

Here is the repo with all of my various configurations if you want to check something out. There were a few terminal configs that didn’t end up there just because they were too minimal or I didn’t like them.

The post Exploring Linux Terminals first appeared on Practical System Administration.

One challenge when dealing with Terraform is keeping things clean and repeatable. My current favorite approach to accomplish this task using shell scripts is by using a combination of null_resources and triggers to control when scripts should be updated. These controls combined with Terraform provisioners and template_files provide a nice flexible way to deal with otherwise potentially messy scripts.

I like to provide more than one trigger so that they can be recomputed when either a variable that gets passed into the script changes, or, the script itself changes. This trick is handy when you need to bend Terraform into doing something that it usually doesn’t handle, like some startup or bootstrap process.

Below is the full example with templated scripts and triggers in Terraform v0.11. The same logic should also work with Terraform v0.12 with minimal changes.

data "template_file" "cool_script" {
  template = "${file("${path.module}/script.sh")}"

  vars {
    my_cool_var = "${var.my_cool_var}"
  }
}

resource "null_resource" "script" {

  # Trigger when the script when variables change
  triggers = {
    my_trigger = "${var.my_cool_var}"
    script_sha = "${sha256(file("${path.module}/script.sh"))}"
  }

  provisioner "local-exec" {
    command   = "${data.template_file.cool_script.rendered}"
    interpreter = ["/bin/bash", "-c"]
  }
}

You can provide as many variables to the template_file as the script needs and any time those get updated, the null_resource will pick these changes up and update/rerun your script for you. Notice that the provisioner in the null_resource is basically set up to call bash against the rendered script that we created, using the values in the vars.

If you call the script again, nothing should change because we already computed the SHA of the rendered script and told Terraform to keep track of that state. The below example is a simple way for telling Terraform to keep track of changes to the script/template.

script_sha = "${sha256(file("${path.module}/script.sh"))}"

Once the script is updated, the next time the script gets called, its values and output should update accordingly. Likewise, since we are also using a var as a trigger, when that changes the script will also be updated.

Terraform is flexible enough to allow us to do things like this because often times there are situations and edge cases where Terraform can’t really perform some actions. The above example is a common workaround for provisioning resources that either don’t have an API that Terraform can tap into, or are just tasks that are only handled via some startup or bootstrap script/process.

The post Idempotent Shell Scripts with Terraform first appeared on Practical System Administration.

One thing that I love about bash is that there is never a shortage of new tips and tricks to learn. I have been using bash for over 10 years now and just stumbled on this little trick.

This one (as the title implies) allows you to quickly substitute a string into the previous command and rerun the command with the substitution.

Quick substitution is officially part of the Bash Event Designators mechanism and is a great way to fix a typo from a previous command. Below is an example.

# Simple example to highlight substitutions
echo foo

# This will replace the string "foo" with "bar" and rerun the last command
^foo^bar

This shorthand notation is great for most use cases, with the exception of needing to replace multiple instances of a given string. Luckily that is easily addressed with the Event Designators expanded substitution syntax, shown below.

# This will substitute ALL occurrences of foo in the previous command
!!:gs/foo/bar/

# Slightly different syntax allows you to do the same thing in ZSH
^foo^bar^:G

The syntax is slightly more complicated in the first example but should be familiar enough to anyone that has used sed and/or vim substitutions, and the second example is almost identical to the shorthand substitution.

fc

Taking things one step further, we can actually edit the previous command to fix anything more than a typo of different argument. fc is actually a bash builtin function so it is available almost everywhere.

fc is especially useful for dealing with very long, complicated commands.

# Oops, we messed this up
echo fobarr | grep bar

# To fix it, just open the above in your default editor
fc

# when you write and quit the file it will put the contents into your current command

There are many great tutorials available so I would recommend looking around to see all the options and get more ideas.

The post Bash Quick Substitution first appeared on Practical System Administration.

Update (2/7/2020): Added jc

The landscape of command line driver text manipulation and processing tools is somewhat large and confusing, with more and more tools emerging all the time. Because I am having trouble keeping them all in my head, I decided to make a little reference guide to help remember which tool to choose for the correct task at hand.

• jq
• jc
• yq (both python and go versions)
• oq
• xq
• hq
• jk
• ytt
• configula
• jsonnet
• sed (for everything else)

jq

Reach for jq first when you need to do any kind of processing with JSON files.

From the website, “jq is like sed for JSON data – you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text.”

sudo apt-get install jq
# or
brew install jq

# consume json and output as unchanged json
curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | jq '.'

jc

This project looks very interesting and is a refreshing thing to see in the Linux world. JC is basically a way to create structured objects (ala PowerShell) as JSON output from running various Linux commands. And from the GitHub repo, ” This tool serializes the output of popular gnu linux command line tools and file types to structured JSON output. This allows piping of output to tools like jq”.

Transforming data into structured objects can massively simplify interacting with them by pairing the output with jq to interact with.

pip3 install --upgrade jc
df | jc --df -p
[
  {
    "filesystem": "devtmpfs",
    "1k_blocks": 1918816,
    "used": 0,
    "available": 1918816,
    "use_percent": 0,
    "mounted_on": "/dev"
  },
  {
    "filesystem": "tmpfs",
    "1k_blocks": 1930664,
    "used": 0,
    "available": 1930664,
    "use_percent": 0,
    "mounted_on": "/dev/shm"
  },
  ...
]

yq (Python) (Go)

This tool can be confusing because there is both a Python version and a Go version. On top of that, the Python version includes its own version of xq, which is different than the standalone xq tool.

The main differences between the Python and Go version is that the Python version can deal with both yaml and xml while the Go version is meant to be used as a command line tool to deal with only yaml.

pip install yq
cat input.yml | yq -y .foo.bar

oq

From the website, oq is “A performant, portable jq wrapper thats facilitates the consumption and output of formats other than JSON; using jq filters to transform the data”.

The claim to fame that oq has is that it is very similar to jq but works better with other data formats, including xml and yaml. For example, you can read in some xml (and others), apply some filters, and output to yaml (and others). This flexibility makes oq a good option if you need to deal with different data formats oustide of JSON.

snap install oq
# or
brew tap blacksmoke16/tap && brew install oq

# consume json and output xml
echo '{"name": "Jim"}' | oq -o xml .

xq

From the Github page, “Apply XPath expressions to XML, like jq does for JSONPath and JSON”.

The coolest use case I have found for xq so far is taking in an xml file and outputting it into a json file, which surprising I haven’t found another tool that can do this (oc authors say there are plans to do this in the future). The simplest example is to curl a page, pipe it through xq to change it to json and then pipe it again and use jq to manipulate the data.

pip install yq
curl -s https://mysite.xml | xq .

hq

Like xq (and jq) but for html parsing. This tool is handy for manipulating html in the same way you would xml or json.

pip install hq
cat /path/to/file.html | hq '`Hello, ${/html/head/title}!`'

jk

This is a newer tool, with a slightly different approach aimed at helping to automate configurations, especially for things like Kubernetes but should work with most structured data.

This tool works with json, yaml and hcl and can be used in conjunction with Javascript, making it an interesting option.

curl -Lo jk https://github.com/jkcfg/jk/releases/download/0.3.1/jk-darwin-amd64
chmod +x jk
sudo mv jk /usr/local/bin/

// alice.js
const alice = {
  name: 'Alice',
  beverage: 'Club-Mate',
  monitors: 2,
  languages: [
    'python',
    'haskell',
    'c++',
    '68k assembly', // Alice is cool like that!
  ],
};

// Instruct to write the alice object as a YAML file.
export default [
  { value: alice, file: `developers/${alice.name.toLowerCase()}.yaml` },
];

jk generate -v alice.js

ytt

This is basically a simplified templating language that only intends to deal with yaml. The approach the authors took was to create yaml templates and sanbdox/embed Python into the templating engine, allowing users to call on the power of Python inside of their templates.

The easiest way to play around with ytt if you don’t want to clone the repo is to try out the online playground.

curl -Lo ytt https://github.com/k14s/ytt/releases/download/v0.25.0/ytt-darwin-amd64
chmod +x ytt
sudo mv ytt /usr/local/bin/

https://github.com/k14s/ytt.git && cd ytt
ytt -f examples/playground/example-demo/

configula

From the GitHub page, ” Configula is a configuration generation language and processor. It’s goal is to make the programmatic definition of declarative configuration easy and intuitive”.

Similar in some ways to ytt, but instead of embedding Python into the yaml template file, you create a .py file and then render the py file into yaml using the Configula command line tool.

git clone https://github.com/brendandburns/configula
cd configula

# tiny.py
# Define a YAML object where the 'foo' field has the value of evaluating 1 + 2 (e.g. 3)
my_obj = foo: !~ 1 + 2
my_obj.render()

./configula examples/tiny.py

jsonnet

Jsonnet bills itself as a “data templating language for app and tool developers”. This tool was originally created by folks working at Google, and has been around for quite some time now. For some reason always seems to fly underneath the radar but it is super powerful.

This tool is a superset of JSON and allows you to add conditionals, loops and other functions available as part of its standard library. Jsonnet can render itself into json and yaml output.

pip install jsonnet
# or
brew install jsonnet

// example.jsonnet
{
  person1: {
    name: "Alice",
    welcome: "Hello " + self.name + "!",
  },
  person2: self.person1 { name: "Bob" },
}

jsonnet -S example.jsonnet

sed

For (pretty much) everything else, there is Sed. Sed, short for stream editor, has been around forever and is basically a Swiss army knife for manipulating text, and if you have been using *nix for any length of time you have more than likely come across this tool before. From their docs, Sed is “a stream editor is used to perform basic text transformations on an input stream”.

The odds are good that Sed will likely do what you’re looking for if you can’t use one of the aforementioned tools.

The post Untangling CLI Text Processing Tools first appeared on Practical System Administration.

If you have spent any time managing infrastructure you have probably run into the issue of needing to set environment variables in order to connect to various resources.

For example, the AWS Terraform provider allows you to automatically source local environment variables, which solves the issue of placing secrets in places they should be, ie. writing the keys into configurations or state. This use case is pretty straight forward, you can just set the environment variables once and everything will be able to connect.

The problem arises when environments need to be changed frequently or other configurations requiring secrets or tokens to connect to other resources need to be changed.

There are various tools available to solve the issue of managing and storing secrets for connecting to AWS including aws-vault and aws-env. The main issue is that these tools are opinionated and only work with AWS and aren’t designed to work with Terraform.

Of course this problem isn’t specific to Terraform, but the solution I have discovered using ondir is generic enough that it can be applied to Terraform as well as any other task that involves setting up environment variables or running shell commands whenever a specific directory is changed to or left.

With that said, the ondir tool in combination with Terragrunt, gives you a set of tools that vastly improves the Terraform experience. There is a lot of Terragrunt reference material already so I won’t discuss it much here. The real power of my solution comes from combining ondir with Terragrunt to enhance how you use Terraform.

Note: Before getting into more details of ondir, it is worth mentioning that there are several other similar tools, the most well known of which is called direnv. Direnv provides a few other features and a nice library for completing actions so it is definitely work checking out if you are not looking for a solution specific to Terraform/Terragrunt. Other tools in the space include autoenv, smartcd and a few others.

Direnv should technically work to solve the problem, but when managing and maintaining large infrastructures, I have found it to be more tedious and cumbersome to manage.

Ondir

From the Github page, Ondir is a small program to automate tasks specific to certain directories, so isn’t designed specifically to set environment variables but makes a perfect fit doing so.

Install ondir

# OSX
brew install ondir
# Ubuntu
apt install ondir

Configure ondir

After installing, add the following lines to your ~/.zshrc (or these lines if you are using bash) and restart your shell.

val_ondir() {
  eval "`ondir \"$OLDPWD\" \"$PWD\"`"
}
chpwd_functions=( eval_ondir $chpwd_functions )

Ondir leverages an ~/.ondirrc file to configure the tool. This file basically consists of enter and leave directives for performing actions when specific directories are entered and left. To make this tool useful with Terragrunt, I have setup a simple example below for exporting a database password given a path that gets switched into (if the path ends in stage/database), which is defined in the ~/.ondirrc file. One very nice feature of ondir is that it allows regular expressions to be used to define the enter/leave directives, which makes allows for complex paths to be defined.

enter .+(stage\/database)
    echo "Setting database password"
    export TF_VAR_db_password=mypassword
leave .+(stage\/database)
    unset TF_VAR_db_password

With this logic, the shell should automatically export a database password for Terraform to use whenever you switch to the directory ending in stage/database.

cd ~/project/path/to/stage/database
# Check your environment for the variable
env | grep TF_VAR
TF_VAR_db_password=mypassword

cd ~
# When you leave the directory it should get unset
env | grep TF_VAR

In a slightly more complicated example, you may have a databases.sh script containing more logic for setting up or exporting variables or otherwise setting up environments.

#!/usr/bin/env bash

db_env=$(pwd | awk -F '/' '{print $(NF-1)"/"$NF}')

if [[ "$db_env" == "stage/database" ]]; then
    # do stuff
    export TF_VAR_db_password="mypassword"

else [[ "$db_env" == "prod/database" ]]; then
    # do other stuff
    export TF_VAR_db_password="myotherpassword"
fi

Conclusion

That’s pretty much. Now you can automatically set environment variables for Terraform to use, based on the directory we are currently and no longer need to worry about setting and unsetting environment variables manually.

I HIGHLY recommend checking out direnv as well. It offers slightly different functionality, and some neat features which in some cases may actually be better for specific tasks than ondir. For example, the direnv stdlib has some great helpers to help configure environments cleanly.

For me, using ondir for managing my Terragrunt configs and direnv for managing other projects makes the most sense.

The post Automatically set Terraform environment variables first appeared on Practical System Administration.

First, there have been some relevant announcements regarding the changes and deprecations recently. The first being the API Depractions in 1.16 announcement, which describes the changes to the API and some of the things to look at and do to fix problems

The next post is the Kubernetes 1.16 release announcement, which contains a section “Significant Changes to the Kubernetes API” that references the deprecation post.

Another excellent resource for learning about how Kubernetes deprecations work is the API deprecation documentation, highlighted in the deprecation post, but not widely shared.

In my opinion, the Kubernetes community really dropped the ball in terms of communicating these changes and missed an opportunity to describe and discuss the problems that these changes will create. I understand that the community is gigantic and it would be impossible to cover every case, but to me, the few blog posts describing the changes and not much other official communication or guides for how to handle and fix the impending problems is a little bit underwhelming.

The average user probably doesn’t pay attention to these blog posts, and there are a lot of old Helm charts out in the wild still, so I’m confident that the incoming changes will create headaches and table flips when people start upgrading. As an example, if you have an old API defined and running in a pre 1.16 cluster, and upgrade without fixing the API version first, APPS IN YOUR CLUSTER WILL BREAK. The good news is that new clusters won’t allow the old API versions, making errors easier to see and deal with.

Testing for and fixing deprecated APIs

With that mini rant out of the way, there is a simple but effective way to test your existing configurations for API compatibility.

Conftest is a nice little tool that helps write tests against structured configuration data, using the Rego language using Open Policy Agent (OPA). Conftest works with many file types including JSON, TOML and HCL, which makes it a great choice for testing a variety of different configurations, but is especially useful for testing Kubernetes YAML configurations.

To get started, install conftest.

wget https://github.com/instrumenta/conftest/releases/download/v0.15.0/conftest_0.15.0_Linux_x86_64.tar.gz
tar xzf conftest_0.15.0_Linux_x86_64.tar.gz
sudo mv conftest /usr/local/bin

Then we can use the handy policy provided by the deprek8 repo to validate the API versions.

curl https://raw.githubusercontent.com/naquada/deprek8/master/policy/deprek8.rego > deprek8.rego
conftest test -p deprek8.rego sample/manifest.yaml

Here’s what a FAIL condition might look like according to what is defined in the rego policy file for an outdated API version.

FAIL - sample/manifest.yaml - Deployment/my-deployment: API extensions/v1beta1 for Deployment is no longer served by default, use apps/v1 instead.

The Rego policy is what actually defines the behavior that Conftest will display and as you can see, it found an issue with the Deployment object defined in the test manifest.

Below is the Rego policy that causes Conftest to spit out the FAILure message. The syntax is clean and easy to follow, so writing and adjusting policies is easy.

_deny = msg {
  resources := ["DaemonSet", "Deployment", "ReplicaSet"]
  input.apiVersion == "extensions/v1beta1"
  input.kind == resources[_]
  msg := sprintf("%s/%s: API extensions/v1beta1 for %s is no longer served by default, use apps/v1 instead.", [input.kind, input.metadata.name, input.kind])
}

Once you know what is wrong with the configuration, you can use the kubectl convert subcommand to fix up the existing deprecated API objects. Again, attempting to create objects using deprecated APIs in 1.16 will be rejected automatically by Kubernetes, so you will only need to deal with converting existing objects in old clusters being upgraded.

From the above error, we know the object type (Deployment) and the version (extensions/v1beta1). With this information we can run the convert command to fix the object.

# General syntax
kubectl convert -f <file> --output-version <group>/<version>

# The --output-version flag allows specifying the API version to upgrade to 
kubectl convert -f sample/manifest.yaml  --output-version apps/v1

# Omitting the --output-version flag will convert to the latest version
kubectl convert -f sample/manifest.yaml

After the existing objects have been converted and any manifest files have been updated you should be safe to upgrade Kubernetes.

Bonus

There was a fantastic episode of TGIK awhile back called Kubernetes API Removal and You that describes in great detail what all of the deprections mean and how to fix them – definitely worth a watch if you have the time.

Conclusion

OPA and testing configurations using tools like conftest and Rego policies is a great way to harden and help standardize configurations. Taken a step further, these configuration testing tools can be extended to test all sorts of other things.

Conftest looks especially promising because of the number of file types that it understands. There is a lot of potential here for doing things like unit testing Kubernetes configuration files and other things like Terraform configs.

I haven’t written any Rego policies yet but the language looks pretty straight forward and easy to deal with. I think that as configurations continue to evolve, tools like Conftest (OPA), Kubeval and Kustomize will gain more traction and help simplify some of the complexities of Kubernetes.

The post Testing for Deprecated Kubernetes APIs first appeared on Practical System Administration.

Inspired by the aws-vault tool which is a simple but secure way of storing local AWS credentials in environment variables using a local password store, in this post I will show you a quick and dirty way to add an extra layer of security to your (other) local environment by injecting sensitive secrets stored in an encrypted location (password store) into your local terminal. This method works for both OSX and Linux and is just a few lines of configuration and examples for both OSes are shown below.

In OSX the keychain is a good starting place for storing and retrieving secrets and in Linux the combination of GPG and the standard unix password manager “pass” work well together. Pass also works on OSX if you aren’t a fan of keychain.

Below are steps for storing and retrieving local secrets using the Linux pass tool. There are installation instructions and full documentation for how to use the tool in the link above. It should also be noted that the system needs to have GPG installed in order to write and read secrets.

One you have GPG configured, create the password store. I am skipping most of the GPG configuration because there is a lot to know, the command below should be enough to get things started. If you already have GPG set up and configured you can skip the setup.

Set up GPG and pass.

gpg2 --full-gen-key # follow prompts to create a gpg store with defaults
pass init <email> # use the same email address used with gpg
pass git init # optionally set pass up as a git repo

To create/edit a secret.

#pass insert/edit <secret>
pass insert mysecret
pass edit mysecret

Pass allows for hierarchies but in the example we are just going to put the secret at the top level. The command above will open the default editor. After closing the editor, the password will be written to an encrypted file in ~/.password-store. Once you have added the password you can show the contents of the newly added secret.

To read a secret into the terminal.

#pass show <secret>
pass show mysecret

You can also quickly list all of your secrets.

pass ls

Now that we have a created secret, we can write a little bash function to pull out the contents of the password and export them as an environment variable when the shell gets sourced. Put the following snippet into your ~/.bashrc, ~/.zshrc or ~/.bashprofile to read secrets.

get_password () {
  pass show "$1"
}

A similar result can be achieved in OSX using the “security” command line tool.

get_password () {
  security find-generic-password -ga "$1" -w
}

In your shell configuration file you can simply export the result of calling the get_password() function into an environment variable.

export MYSECRET="$(get_password mysecret)"

Source the shell profile to pickup the new changes. After that, you should now see the contents of the secret inside an environment variable in your terminal.

source ~/.bashrc
env | grep MYSECRET

Conclusion

Obviously this isn’t a perfect way to secure your environment since the secret is available to anyone who is able to connect to this user so make sure you practice good security in as many other ways as possible.

What this method does do though is cuts down the amount of sensitive information that can be gleaned from a user account by ensuring that shell secrets are encrypted at rest and unavailable as clear text.

The post Quickly securing local secrets first appeared on Practical System Administration.