David Chua | Dev Notes for a Ruby Server Geek

The Signal Pattern

2017-06-28T00:00:00+08:00

The signal pattern can be used to ensure your long-lived processes (ie. servers) are able to handle signals being sent to the process.


import ...

type Handler struct {
}

func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
  w.Write([]byte(`{}`))
}

func main() {

  h := &Handler{}

  sigCh := make(chan os.Signal, 1)
  errCh := make(chan error, 1)

  signal.Notify(sigCh, syscall.SIGTERM, syscall.SIGINT, syscall.SIGHUP)

  go func() {
    err := http.ListenAndServe(":3000", h)
    errCh <- err
  }()

  for {
    select {
    case sig := <-sigCh:
      switch sig {
      case syscall.SIGHUP:
        log.Println("woot woot sighup ", sig)
        os.Exit(0)
      case syscall.SIGTERM:
        log.Println("sigterm ", sig)
        os.Exit(0)
      case syscall.SIGINT:
        log.Println("sig interrupt ", sig)
      }

    case err := <-errCh:
      log.Println(err)
      os.Exit(1)

    }
  }
}

Many thanks to @chuyeow for introducing this pattern to me.

References

golang.org - os/signals

Sending your Structs across the wire (tcp connection)

2017-06-23T00:00:00+08:00

Sending your structs across the wire and receiving them on the other side. Using encoding/gob can help ensure your data structures can receive it on the other side.

// server.go

package main

import ...

// Create your custom data struct
type Message struct {
  ID   string
  Data string
}

func main() {
  // for purpose of verbosity, I will be removing error handling from this
  // sample code

  server, err  := net.Listen("tcp", ":12345")
  conn, err = server.Accept()

  // create a temp buffer
  tmp := make([]byte, 500)

  // loop through the connection to read incoming connections. If you're doing by
  // directional, you might want to make this into a seperate go routine
  for {

    _, err = conn.Read(tmp)
    
    // convert bytes into Buffer (which implements io.Reader/io.Writer)
    tmpbuff := bytes.NewBuffer(tmp)

    tmpstruct := new(Message)

    // creates a decoder object
    gobobj := gob.NewDecoder(tmpbuffer)

    // decodes buffer and unmarshals it into a Message struct
    gobobj.decode(tmpstruct)

    // lets print out!
    fmt.Println(tmpstruct) // reflects.TypeOf(tmpstruct) == Message{}

  }

}

// client.go

import ...

type Message struct {
  ID   string
  Data string
}

func main(){
  // lets create the message we want to send accross
  msg := Message{ID: "Yo", Data: "Hello"}
  bin_buf := new(bytes.Buffer)

  // error handling still truncated
  conn, err

  // create a encoder object
  gobobj := gob.NewEncoder(bin_buf)

  // encode buffer and marshal it into a gob object
  gobobj.Encode(msg)

  conn.Write(bin_buf.Bytes())
}

References

Gobs of Data - golang.org*
Network Fundamentals

Using S2i like buildpacks to deploy apps

2017-06-21T00:00:00+08:00

Openshift’s S2i (Source-2-image) is a pretty cool replacement to buildpacks using Dockerfiles and Openshift

Objective

The goal is to be able to create a container image that contains the necessary application dependencies just by uploading the application code without having to write Dockerfiles.

Creating S2i builder images

S2i builder images are the base images that your deploying application will be injected its code/binary into to create a new deployment image.

An easy way to think about S2i builder images is that it is analogous to Buildpacks in which they set the environment for applications.

Directory Structure

S2i builder images have the following directory structure:

/
/s2i/
/s2i/assemble
/s2i/usage
/s2i/run
Dockerfile

Key details to note is that when you create a new build that uses an s2i builder image as its base layer, your source code will be injected into /tmp/src of the builder image. You will need to keep this in mind when you write your assemble scripts.

S2i Builder Image Default Files

/s2i/assemble

This is a bash script that contains instructions that will be run right after your deploying app source code is injected into /tmp/src of the S2i builder image.

#!/bin/sh

set -ex

echo "---> Preparing source..."

mkdir -p /tmp/myapp

cp -Rf /tmp/src/src/. /tmp/myapp

/s2i/usage

This script is run when the builder image is run w/o any additional commands. Its sole responsibility is to print out the usage of the builder image in text form.

/s2i/run

This script is run during runtime when the derived application image runs. Should contain the actual running of a long-lived process.

#!/bin/sh

set -e

echo "Running myapp..."
cat /tmp/myapp/*

echo Sleeping...
while true; do sleep 1; echo -n .;done

S2i Builder Image Dockerfile

A typical Dockerfile should contain at minimum the following steps:


# This image creates a very simple s2i builder image
# We are basing our builder image on openshift base-centos7 image

FROM openshift/base-centos7

# Set labels used in OpenShift to describe the builder images

LABEL io.k8s.description="simple s2i builder example" \
      io.k8s.display-name="simple s2i builder example" \
      io.openshift.s2i.scripts-url="image:///usr/libexec/s2i" \
      io.openshift.tags="builder"

# Copy the example S2I scripts to the expected location
COPY ./s2i/ /usr/libexec/s2i

# Set the default user for the image
USER 1001

# Set the default CMD to print the usage of the image, if somebody does docker run
CMD /usr/libexec/s2i/usage

Bonus: The io.openshift.s2i annotations can provide you with more configurable settings to help you with your s2i image creation. For example, you can use io.openshift.s2i.destination="/path/where/appcode/is/injected". If not your application code will be injected into /tmp/src of the base image.

Cheat Code

To quickly generate the necessary files to get your S2i builder image started (including the above directory structure), you can actually cheap by using the s2i cli. You can grab it at http://github.com/openshift/source-to-image

$ s2i create <image name> <relative path to name of directory to use to contain the files>

Deploying your Builder Image onto Openshift

oc new-build --name <name of builder image> --binary=true # Create the build configuration (using Docker build strategy)
oc start-build <name of builder image> --from-dir=.       # Start the build, using the current directory
oc logs bc/<name of builder image> -f                     # Check the build output for errors

When this is complete, this creates an image called <name of builder image> into Openshift which you can then use as the source base image.

Deploying your App onto Openshift

Now that you have your app, all you really need is to send an instruction to have your app baked into your builder image and return an output image which youc an then use to deploy.

In the directory of your app source code, create a new build configuration. Like the steps above, oc new build creates a build instruction to tell what is the expected output image.

$ oc new-build --binary=true -i <name of builder image> --name <name of output image/appname>

binary=true just instructs Openshift to expect the incoming files to be from a tarball/raw files

At this stage, you should have a working build configuration. Now let’s start to build an output image. The following command will push all the files in the current directory up start a build.

# make sure you're in your app source code directory
$ oc start-build <name of output image/appname> --from-dir=.
$ oc logs bc/<name of output image> -f

After this is completed, just do an oc new-app <name of output image/appname> and your app is deployed!

Bonus: If you want your application to overwrite any of the assemble/run/usage scripts in your builder image, you can add them in .s2i/bin/ of your application root. The S2i process will find these file and inject them into the builder image during build time.

References

Load env variables from ConfigMaps and Secrets upon Pod boot

2017-04-21T00:00:00+08:00

One of the coolest stuff I’ve picked up just today is that you can keep environment variables that you want to be loaded into every deployment pod in a neatly configured ConfigMap or Secret which gets injected back into the Pod during deploys.

Lets say you have a Secret that looks like:

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  username: YWRtaW4=
  password: MWYyZDFlMmU2N2Rm

And you want username and password to be easily accessible in ENV['username'] and ENV['password'] on your application pods, all you need is a envFrom within your TemplateSpec

ie.

apiVersion: extension/v1beta1
kind: Deployment
spec:
  replicas:1
  template:
    spec:
      containers:
        - name: test-container
          image: gcr.io/google_containers/busybox
          command: [ "/bin/sh", "-c", "env" ]
          envFrom:
            - secretRef:
                name: mysecret

Because envFrom expects an array, you can do multiple references like:

envFrom:
  - secretRef:
      name: hello
  - configMapRef:
      name: hello2
  - configMapRef:
      name: hello3

This will take all the data keys from the 2 ConfigMaps and 1 Secret and load it into your pod.

References

Kubernetes - Configmap

How to access unexported embedded structs within composite literals in Golang

2017-04-18T00:00:00+08:00

Short Answer, you can’t. BUT you can still declare it.

If you have two struct types which is embedded within one another but declared as an unexported type, you will not be able to declare it in a composite type. You can only declare it outside of an composite literal

Came across this while attempting to make use of Kubernetes’ API to declare an Deployment object.

type Deployment struct {
  metav1.TypeMeta
  Spec  DeploymentSpec

}

# in the metav1 pacakge
type TypeMeta {
  Kind  string
}

With the above structure, when we try to declare Kind from TypeMeta within Deployment, we will get an error indicating an unknown field of TypeMeta.

test := Deployment{metav1.TypeMeta: {Kind: "Test"}, Spec: Spec{}}

This is because metav1.TypeMeta is declared as a unexported type within Deployment with its lowercase reference.

In order to declare a Kind, you have to declare it from outside.

test := Deployment{Spec: Spec{}}
test.Kind = "Hello"

This is because exported fields (Kind) keep their exported status when that type is embedded.

References

Exported/Unexported Identifiers in Go

Forwarding Vault audit logs to a remote Syslog server (like Graylog)

2017-03-06T00:00:00+08:00

Using Vault’s Audit Backend to send logs to a remote Syslog server.

Objective

Send audit_logs from Hashicorp’s Vault to an Graylog instance

Prerequisite

Setup a Syslog TCP/UDP Input on Graylog (if you’re using graylog)
Has a remote syslog server running

Steps

Important Notice: Vault has a Syslog Audit Backend as part of its suite but it currently does not allow remote forwarding.
In order to do that we will have to make use of rsyslog’s rules forwarding.

In your vault server instance,

Setup your local syslog audit backend

$ ./vault audit-enable syslog tag="<TAG>"

# TAG - A recognizable name that represents what this vault is for or where is it located. eg. mysupersecretserver.domain

This audit backend sends all logs to your local syslog’s auth facility with the tag that you specify

Now, lets create a syslog configuration file to lookout for the logs.

Create a new configuration file:

# /etc/rsyslog.d/30-vault.conf

if $syslogfacility-text == "auth" and $syslogtag startswith "<tag>" then @@<remote_syslog_host>:<remote_syslog_port>

If you are running graylog, replace remote_syslog_host and remote_syslog_port with your Graylog Syslog listening host and port.

Restart your local syslog

$ systemctl restart rsyslog.service

Now when you do any vault activity, you should see the log appear in both /var/log/auth.log and your remote syslog server.

References

Alerting with Prometheus and AlertManager

2017-02-28T00:00:00+08:00

How to setup Prometheus AlertManager and get a whole alerting pipeline setup.

Objectives and Goals

Write and Deploy Prometheus Alert Rules
Configure Prometheus to send Alerts to Alert Manager
Setup AlertManager to receive Prometheus Alert
Send a Slack message on Alert

Prerequisite

Prometheus is already setup and running

Alert Rules

To begin writing and deploying alerts, you’ll need to modify your prometheus config file. Usually, its located at /etc/prometheus/prometheus.yml.

If there’s no rule_files key in root, add it. It should look something like:

rule_files:
  - "alert.rules"

This tells prometheus that there’s a rules file that’s located at alert.rules in a path that’s relative to the location of the current config file.

Rules are basically instructions to Prometheus and can be used for Alerting or Recording. Alerting is to tell Prometheus to raise an alert if certain conditions are met and Recording rules is used to precompute/process time-series as they come in. For example, to rewrite labels into a new time_series.

With your alert.rules file, now you can start writing ALERTING expressions to monitor a Prometheus Metrics and its value.

# Alert Rules

ALERT AppCrash
  IF firehost_value_metric_bbs_crashed_actual_lr_ps > 10
  FOR 15s
  LABELS { severity="critical" }
  ANNOTATIONS {
   summary = "Number of consecutive crashes in the past 30 seconds"
   description = "The number of consecutive crashes in the past 30 seconds is at: . This is dangerous and rectified immediately"
  }

The FOR indicates the duration to wait before firing an alert out. This is important if you want to prevent false positives. By adding a higher duration, it allows the failure to resolve itself before sending out a message.

Now, let’s reload your Prometheus by sending a SIGHUP to the process.

$ ps auwx | grep prometheus
$ kill -1 <prometheus_pid>

When you access http://:, you should see your alert appear in the "ALERT" tab. If you don't, check that your rules and prometheus configuration are not malformed.

Setup AlertManager

When a alert is triggered, Prometheus sends a payload to the urls defined in Prometheus’ /api/v1/alertmanagers.

It might be possible to build your own AlertManagers but in the meantime, let’s use the one provided by Prometheus.

Download the binaries here.

Download it onto a VM which you want to run, un-tar it and you should see two files here. One simple.yml and alertmanager.

Lets keep it simple for now and get alertmanager hooked up.

Run the binary on a server that is internet accessible. I would place it in the same server as Prometheus to start off.

$ ./alertmanager -config.file=simple.yml

Remember to note the port that its listening to.

Now on your Prometheus startup script, re-run Prometheus with an -alertmanager.url http://<server with prometheus>:<port>

If everything is ok, when you curl Prometheus’ alertmanagers endpoint, you should see a new URL under activeAlertmanagers.

curl http://localhost:900/api/v1/alertmanagers

Configure AlertManager to send a Slack Alert

Moving back to AlertManager, right now if an alert is triggered, you’ll probably be unaware. Lets hook it into something that’s more visible. Like a Slack Channel.

First thing to do is to get your Incoming Webhooks url ready. It should look something like https://hooks.slack.com/services/T1Ydfdsfab/B31uy2389/Edfsdfdsfsdf.

Add\Modify the following fields in your simple.yml file in your AlertManager.

global:
  ...
  slack_api_url: "https://hooks.slack.com/services/sdfsfdsf..."

route:
  receiver: slack-alert # replace this field

receivers:
- name: "slack-alert" # you can name this anything you want
  slack_configs:
  - channel: "#channel"
    username: "The Watchmen"
    text: ""
    send_resolved: true

Do a SIGHUP (kill -1 <pid>) on the AlertManager process.

And you’re done!

You should see something like:

References

Deleting files and getting your diskspace back without rebooting

2017-02-03T00:00:00+08:00

When you run out of diskspaces and need to delete files quickly to recover them, most of the time, your deleted files will not free up the diskspace until the process that is using it is restarted or deleted.

To force the filesystem to free up the lock to the file so that the files can be cleared up, you’d need to find the process that is using the file and truncate it.

Finding and deleting the file

First, find the process PID that is using it and look out for deleted files that the process has a lock on.

$ ls -l /proc/<PID>/fd

lr-x------ 1 tmp tmp 64 lut  1 00:06 3 -> /home/tmp/x (deleted)

Once you’ve found out which is the fd that the deleted file is linked to, in the above example its, 3, you can immediately truncate it to release the file lock and allow the filesystem to release the diskspace.

$ :>/proc/<PID>/fd/3

You should immediately see your file diskspace released after the above command.

Note: It should be emphasized that this might cause some unintended consequences to your running process. Please use this with caution.

References

Stack Overflow - Best way to free disk space from deleted files

Using Multiple Buildpacks in Deis

2017-01-22T00:00:00+08:00

To use multiple buildpacks with your Application Deployment on Deis Workflow, is pretty straight forward.

As heroku-buildpack-multi is already built into deis’ slugbuilder, all you need to do is to create a .buildpacks file in your repository, insert your required buildpacks, commit and push away.

An example .buildpack could look like:

# .buildpacks
https://github.com/cloudfoundry/heroku-buildpack-ruby.git
https://github.com/gaumire/heroku-buildpack-mysql

Enjoy.

References

Github - multi-buildpacks

Fixing OmniAuth Redirecting to a URI with Port

2017-01-16T00:00:00+08:00

Came across this problem the other day when deploying a Rails application that is sitting behind Kubernetes and Deis

With my Rails application using OmniAuth and its Facebook strategy, after a user logins, hits Facebook and returns a redirection, the URL returned has its port 80 embedded in the return URI like: https://domain.com:80.

In most cases, this wouldn’t be a problem but Facebook would return an error as the return URI (with the port 80) is not whitelisted hence creating a bad flow for your user.

In order to fix this, you just need to tell OmniAuth where to redirect.

Just add the following in either your environments/*.rb file or create a new initializer file in your config/initializers.

OmniAuth.config.full_host = "https://domain.com"