Some projects require a quick way to deploy code to a Raspberry Pi (or other small device running Linux behing NAT) and it's not critical enough to need things like Kubernetes, Ansible or Docker. It was exactly the case for a small RGB LED matrix we set up at our office to show some KPIs. It was running an OG Raspberry Pi A and we just want to be able to update the code remotly.

The setup

To make the whole thing a bit easier we decided on a Git based setup. The Pi will keep checking a remote repo for changes and if there are any, it will pull them and restart the systemd service driving the display.

Git setup

The Pi needs to be able to access the repository therefore we needed to add the public SSH key of the Pi as a Deploy key:

1. Generate a new key on the Pi (only if it doesn't have one yet) with:

$ ssh-keygen -t ed25519
$ cat ~/.ssh/id_ed25519.pub

2. Copy the resulting key and paste it in Settings -> Deploy keys -> Add deploy key in your GitHub repo

3. Clone the repo on the Pi via ssh

Main application service

1. First you need to define a systemd service for your main application:

[Unit]
Description=rgb-matrix

[Service]
Type=simple
Restart=always
RestartSec=5
ExecStart=/home/pi/rgb-matrix/run.sh
WorkingDirectory=/home/pi/rgb-matrix
StandardOutput=append:/home/pi/rgb-matrix/stdout.log
StandardError=append:/home/pi/rgb-matrix/stderr.log
User=root
Group=root

[Install]
WantedBy=multi-user.target

In our case the application has to run as a root but in your case there might be a better user for that. Once created, you need to enable it:

$ sudo ln -s ${PWD}/rgb-matrix.service /etc/systemd/system/rgb-matrix.service
$ sudo systemctl daemon-reload
$ sudo systemctl enable rgb-matrix.service
$ sudo systemctl start rgb-matrix.service

Auto-deployment service

Now we need a small script that will be periodically checking the remote repo:

#!/bin/bash

# Configuration
SERVICE_NAME="rgb-matrix"  # The name of the systemd service you want to restart

# Function to check for updates and pull them
check_and_pull() {
    git fetch origin

    # Check if there are updates available
    if ! git diff --quiet origin/main; then
        echo "Changes detected, pulling updates..."
        git pull
        echo "Updates pulled successfully."

        # Restart the systemd service
        echo "Restarting the service: $SERVICE_NAME..."
        systemctl restart "$SERVICE_NAME"
        echo "Service restarted successfully."
    fi
}

# Main loop to periodically check for updates
while true; do
    check_and_pull
    # Wait for a specified interval before checking again
    sleep 60  # Checks every 60 seconds, adjust as needed
done

Similarily to the main application, we need to create a systemd service for the watcher:

[Unit]
Description=watcher

[Service]
Type=simple
Restart=always
RestartSec=5
ExecStart=/home/pi/rgb-matrix/watcher.sh
WorkingDirectory=/home/pi/rgb-matrix
StandardOutput=append:/home/pi/rgb-matrix/watcher-stdout.log
StandardError=append:/home/pi/rgb-matrix/watcher-stderr.log
User=root
Group=root

[Install]
WantedBy=multi-user.target

and enable it as well:

$ sudo ln -s ${PWD}/watcher.service /etc/systemd/system/watcher.service
$ sudo systemctl daemon-reload
$ sudo systemctl enable watcher.service
$ sudo systemctl start watcher.service

Done

Now any changes pushed to the remote repo will get automatically fetched and deployed. This of course does not take into account any migrations, installation of dependencies, rollbacks etc. but that's ok :)

While building the Web Microphone app with IPFS I sumbled upon an error when trying to use older examples but using the latest js-ipfs:

Error: no valid addresses were provided for transports [WebSockets,WebRTCStar,Circuit]

It got me puzzled but Vasco cleared it out in this thread:

It turns out that it's necessary to use a new discovery mechanism, webrtc-star. This seemed like an easy way to fix it, just change the server to the hosted randezvous server mentioned in readme and we're done! But nothing is ever that easy. It was failing for me with a 500 error and I shouldn't be surprised as the documentation clearly states:

it should not be used for apps in production

The recommended route is to deploy it on your own. So I tried to get it done in shortes amount of time and preferably without a need to pay for the whole server. Fortunately, Heroku supports Docker containers now and they can be deployed literally with couple of lines! Here's how you do it:

Instructions

First install the Heroku CLI and then run:

# Login to Heroku
$ heroku login
# Login to the Container Registry
$ heroku container:login
# Clone the webrtc-star repo
$ git clone https://github.com/libp2p/js-libp2p-webrtc-star.git
$ cd js-libp2p-webrtc-star
# Create a Heroku app
$ heroku create
# Build and push the image
$ heroku container:push web
# Release the image to your app
$ heroku container:release web
# Scale to one free worker
$ heroku ps:scale web=1
# Open the app in the browser
$ heroku open

Now the server should welcome you with its address:

Use this address in your IPFS app and you're done!

With the release of TinyGo 0.13, you can now write firmware for Particle Argon, Boron and Xenon in Go:

There are some caveats for now, namely:

1. TinyGo can't coexist with the Particle Device OS. Meaning flashing your Go app to the device will erase Particle bootloader and system. It is still possible to restore the device to "stock" but it requires manual flashing of all necessary parts.
2. TinyGo doesn't support Argon's WiFi coprocessor nor Boron's cellular modems yet, meaning for now you can only use the devices offline. Ayke is making progress on support for the nRF Soft Device that allows BLE support (and Mesh in the future) but it's still experimental.
3. You need to use the Particle Debugger to do all the flashing. It might be possible to use Adafruit nRF52 Bootloader (same as used for CircuitPython) in the future to allow flashing over the USB.

That being said, running TinyGo might be a nice alternative to CircuitPython if you need more speed. How fast is it? Well, it can handle a real-time flight controller for a drone on Arduino Nano33 IoT with SAMD21 Cortex-M0 at 48 MHz:

Or drive six 32x32 RGB LED matrices on Cortex-M4:

Additionally, a strong advantage of TinyGo is that TinyGo is written in Go itself! Meaning, you don't need to know C/C++ to understand/contribute or write low level code.

Got you interested? Lets proceed to the...

Instructions

Prerequisites

• Particle Argon/Boron/Xenon $11-$55
• Particle Debugger $12

Necessary software

1. Install OpenOCD. If you have Particle Workbench installed, it should already download it for you to ~/.particle/toolchains/openocd directory. Make sure the openocd binary is in your PATH.
2. Install TinyGo

Blinky

Well, that's it really. Now you're ready to go write Go:

package main

import (
	// machine package exposes all hardware available on the microcontroller
	"machine"
	"time"
)

func main() {
	// All the pins are listed here: https://tinygo.org/microcontrollers/machine/particle-xenon/
	led := machine.LED
	// Make it an output
	led.Configure(machine.PinConfig{Mode: machine.PinOutput})
	// Repeat forever
	for {
		// Turn it on (it's active low)
		led.Low()
		time.Sleep(time.Millisecond * 500)

		// And off
		led.High()
		time.Sleep(time.Millisecond * 500)
	}
}

Now you can flash your app with:

$ tinygo flash -target=particle-xenon blink

(assuming you save the Go code above in blink directory). Also, you should set the -target flag to the module you're using.

Summary

Even although Argon and Boron can't connect to the internet yet, peripherals like UART, GPIO, SPI, I2C, ADC, and PWM are working, and there are already plenty of drivers for devices like displays, sensors, etc.

I hope you'll have fun with TinyGo and if you like to chat about it or contribute, feel free to join the Slack channel and check out the GitHub repos.

In different contexts like development, running tests or serving production, our apps have different needs when it comes to their Docker images. For development, you might want a live-reload server with all necessary dependencies. For testing you probably need some more packages that you definitely don't want on production.

There are ways to go about it, the first one that pops in mind is using multiple Dockerfiles and just telling Docker which one to use. But this approach has its limits, especially along the road, where each file can very easily develop its own life, diverging from each other beyond recognition.

Fortunately since Docker 17.05 we have the concept of multi-stage builds. An ability to declare separate images in one Dockerfile and freely inherit or copy data between each other. This is the exact approach I used when building images for Lox:

#############################################
# Base container with all necessary deps
FROM tiangolo/uvicorn-gunicorn:python3.7-alpine3.8 AS base

ENV HOME=/app 
    BUILD_DEPS="build-base linux-headers postgresql postgresql-dev libffi-dev"
WORKDIR ${HOME}

# Copy the pipenv files
COPY Pipfile ${WORKDIR}/
COPY Pipfile.lock ${WORKDIR}/

# 1. Install system dependencies
# 2. Install pipenv
# 3. Use pipenv to install app deps
# 4. Remove system deps to save space
RUN apk add --no-cache ${BUILD_DEPS} \
    && pip install --no-cache-dir pipenv \
    && pipenv install --system \
    && apk del ${BUILD_DEPS}


#############################################
# Test container from a common base
FROM base AS test
# Same as in the base image but this time we also install the --dev packages
RUN apk add --no-cache ${BUILD_DEPS} \
    && pip install --no-cache-dir pipenv \
    && pipenv install --system --dev \
    && apk del ${BUILD_DEPS}


#############################################
# Live container with Webpack watcher
FROM base AS live
# Install Node.js dependencies
RUN apk add --no-cache nodejs npm
# Install all Webpack dependencies
COPY package.json package-lock.json ./
RUN npm install
# Finally copy the current sources and build the bundle
COPY . .
RUN npm run bundle:build


#############################################
# Final container with the app
FROM base AS production
# Only copy the ready app dir from the live step
COPY --from=live ${WORKDIR} ${WORKDIR}

Note: for the sake of brevity, this Dockerfile doesn't contain all size improvements and will produce slightly bigger images.

Now you're able to build separate images for each of your needs:

Testing

$ docker build -t suda/lox/test --target test .

Development w/ live-reload

$ docker build -t suda/lox/live --target live .

Production

$ docker build -t suda/lox --target production .

P.S.: You might've noticed I used the great Python base image from Sebastián Ramírez and if you want to deploy a Django project using ASGI, I have some instructions that can help you.

With Django 3 released, it's a great moment to jump in on all the async goodies it provides. Unfortunately for me, it means dropping my uWSGI Docker config and figuring out a new approach.

Fortunately, Sebastián Ramírez of fastapi fame, created a very nice base image using uvicorn: uvicorn-gunicorn-docker. One thing missing was the ability to serve static files, but here's where WhiteNoise comes in.

Let's jump in to the setup.

Instructions

Compared to my uWSGI setup, here we just need to create a Dockerfile:

ARG PYTHON_VERSION=3.7

# Build dependencies in separate container
FROM tiangolo/uvicorn-gunicorn:python${PYTHON_VERSION}-alpine3.8 AS builder
ENV WORKDIR /app
COPY Pipfile ${WORKDIR}/
COPY Pipfile.lock ${WORKDIR}/

RUN cd ${WORKDIR} \
    && pip install pipenv \
    && pipenv install --system

# Create the final container with the app
FROM tiangolo/uvicorn-gunicorn:python${PYTHON_VERSION}-alpine3.8

ENV USER=docker \
    GROUP=docker \
    UID=12345 \
    GID=23456 \
    HOME=/app \
    PYTHONUNBUFFERED=1
WORKDIR ${HOME}

# Create user/group
RUN addgroup --gid "${GID}" "${GROUP}" \
    && adduser \
    --disabled-password \
    --gecos "" \
    --home "$(pwd)" \
    --ingroup "${GROUP}" \
    --no-create-home \
    --uid "${UID}" \
    "${USER}"

# Run as docker user
USER ${USER}
# Copy installed packages
COPY --from=builder /usr/local/lib/python3.7/site-packages /usr/local/lib/python3.7/site-packages
# Copy the application
COPY --chown=docker:docker . .
# Collect the static files
RUN python manage.py collectstatic --noinput

Serving the static files

For this purpouse, I'm using WhiteNoise which needs to be added to dependencies:

$ pipenv install whitenoise

and then added whitenoise.middleware.WhiteNoiseMiddleware to the top of the MIDDLEWARE array in settings.py, right below the SecurityMiddleware:

MIDDLEWARE = [
  'django.middleware.security.SecurityMiddleware',
  'whitenoise.middleware.WhiteNoiseMiddleware',
  # ...
]

Side note: I know many people are wondering why I'm so set on serving static files, instead of just using S3 buit I think David explained it well:

Shouldn’t I be pushing my static files to S3 using something like Django-Storages?
No. (...) problem with a push-based approach to handling static files is that it adds complexity and fragility to your deployment process.

Running the image

uvicorn needs to know what is the module is the main application, therefore we need to set the APP_MODULE environment variable to myapp.asgi:application replacing myapp with the name of your app. This assumes you have the asgi.py file (ASGI equivalent of the wsgi.py) which will be generated automatically when creating new Django 3.0 project but if you're migrating from an older version, you can use this one:

"""
ASGI config for myapp project.

It exposes the ASGI callable as a module-level variable named ``application``.

For more information on this file, see
https://docs.djangoproject.com/en/3.0/howto/deployment/asgi/
"""

import os

from django.core.asgi import get_asgi_application

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myapp.settings')

application = get_asgi_application()

Notes

Running in async mode, means you should not use possibly blocking, synchronous methods (like the ORM) in the main thread. Quick tip here is, that if you get SynchronousOnlyOperation exception you might want to wrap it with sync_to_async:

from asgiref.sync import sync_to_async

def my_db_function():
    <do orm stuff>

await sync_to_async(my_db_function)()

When building desktop applications, whether it's with native frameworks or with web ones, we take for granted how easy it is to handle. I'm in the process of building a portable synthesizer that runs on Particle Xenon with limited resources and on a very slow SPI screen which made me appreciate web frameworks a bit more.

The problem

The problem I had, boiled down to figuring out a straightforward (and fast) way to reconcile many inputs/outputs.

There are many ways to go about it but I wanted a clear and straightforward way of describing what happened and how to react. The last word of the previous sentence might give you some indication about what I went with ;)

React and Flux

React is a popular frontend library, enabling reusable, reactive components for the user interface. Flux is an architecture of how events should flow through the system, that complements React. Note that Flux isn't a library but rather a concept that libraries like Redux took and implemented. But for the purpouse of this project, Flux was simple enough.

Just to be clear, I didn't implement React/Flux in C but rather used some of the principles to develop a pattern. From React, I took partial rendering of only the things that changed and from Flux the idea of actions, stores and dispatchers.

Actions, stores, views and dispatchers

There's a great explanation of all those principles in Flux's documentation. I made some adjustments, mostly to accomodate limitations of the microcontroller and statically typed C. What I ended up with is:

Actions

To identify each Action, they are being defined with unique identifier, so when they are dipatched, they can be passed to the correct View.

#define ACTION_INCREMENT_COUNTER 1

Stores

A Store (rather than many stores) was implemented as a struct:

struct Store {
  uint32_t counter;
}

#define STORE_PROP_COUNTER 1

The Store is being shared between Views mostly to allow mixing global state with local one. Additionally this allows a easy way to serialize/deserialize and store it in EEPROM.

For each Store property there's a #define that is used to indicate which property has been changed.

Views

A View is just a class that implements two methods:

• void handleAction(uint8_t action, int16_t args[])
• void handleStoreUpdate(uint8_t storeProp)

If the View is currently active (there's an assumption that only one View can be active at the time), it will receive all the dispatched Actions. The handleAction method should decide if it will react to it. If it does, it can dispatch another Action or modify the Store. If it does update the Store, it should call handleStoreUpdate() with respective Store property, to update the UI.

An example action handler might look like this:

void MyView::handleAction(uint8_t action, int16_t args[]) {
  switch (action) {
    // For the increment action...
    case ACTION_INCREMENT_COUNTER:
      // increment the counter in the store by the first argument
      this->store->counter += args[0];
      // Update the UI that displays the counter
      this->handleStoreUpdate(STORE_PROP_COUNTER);
      break;
  }
}

and the update handler like this:

void MyView::handleStoreUpdate(uint8_t storeProp) {
  switch (storeProp) {
    // For the counter property...
    case STORE_PROP_COUNTER:
      // ..update the UI
      this->screen->drawCounter();
      break;
  }
}

Dispatchers

A Dispatcher is a main class that contains the list of Views including which of them is currently active and routes all dispatched Actions to it.
Actions can take arguments and for simplicity, it's an array of four int16_t. To make calling easier, there are two macros to help:

#define MK_ARGS(name, arg0, arg1, arg2, arg3) int16_t name[] = {arg0, arg1, arg2, arg3};
#define NO_ARGS(name) MK_ARGS(name, 0, 0, 0, 0)

the idea behind this is to change the macros if the type (or number) of arguments changes instead of changing each piece of code that dispatches an Action.

To dispatch an Action the dispatchAction() method has to be called:

MK_ARGS(args, 1, 0, 0, 0)
dispatcher.dispatchAction(ACTION_INCREMENT_COUNTER, args);

all of this can be summarized in one, nifty chart:

Where next?

I developed the pattern above while working on my ps-01 synthesizer and it's the main place where a "reference implementation" lives.

I also made a video going through the early iteration of this pattern (where View is called a Page and Dispatcher an UI):

The slides from the video are also available on SpeakerDeck.

If you have any comments or ideas about this pattern, please let me know in the comments (or on social media) 🙂

Updated on Jan 10, 2021 to use Helm 3.

After getting tired with instability of my QNAP TS-251B (random reboots once a day, official apps not working) I decided to turn my Intel NUC into a home server. I could've used a Raspberry Pi, but I do need a x86 for most of the Docker images I use. I decided to use Kubernetes, which for many might come off as a huge overkill (and possibly it is) but that's what I'm interacting with in my professional life thus it's easier for me to do what I need in a familiar environment :)

I looked through single node Kubernetes installation options, I found microk8s which Kelsey Hightower called "the easiest way to provision a single node Kubernetes cluster". Oh boy was he right! Ubuntu Server comes with an option to install microk8s making it practically zero effort installation!

Second part for me was to add some GUI to manage. I choose Rancher as it has a great integration with both Kubernetes and kops.

Installation

First start with a fresh installation of Ubuntu Server. I used 18.04 LTS but feel free to use latest version. To make a bootable USB drive with the image, you can use Balena Etcher. Once you have it installed on your machine, ssh into it and issue following commands:

# Install microk8s from the 1.19 channel (Rancher doesn't support Kubernetes 1.20 yet)
$ sudo snap install microk8s --classic --channel=1.19
# Enable useful plugins
$ sudo microk8s.enable dns dashboard storage ingress helm3

# Allow running priviledged Pods (required by Rancher's `cattle-node-agent`)
$ sudo sh -c 'echo "--allow-privileged=true" >> /var/snap/microk8s/current/args/kube-apiserver'
$ sudo systemctl restart snap.microk8s.daemon-apiserver.service

# Install cert-manager user by Rancher
$ sudo microk8s.helm3 repo add jetstack https://charts.jetstack.io
$ sudo microk8s.kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.9/deploy/manifests/00-crds.yaml
$ sudo microk8s.kubectl create namespace cert-manager
$ sudo microk8s.kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true
$ sudo microk8s.helm3 install --name cert-manager --namespace cert-manager --version v0.9.1 jetstack/cert-manager

# Install stable Rancher
$ sudo microk8s.kubectl create namespace cattle-system
$ sudo microk8s.kubectl label namespace cattle-system cattle-system.k8s.io/disable-validation=true
$ sudo microk8s.helm3 repo add rancher-latest https://releases.rancher.com/server-charts/latest
$ sudo microk8s.helm3 repo update
$ sudo microk8s.helm3 install rancher rancher-latest/rancher --namespace cattle-system  --set replicas=1 --set hostname=${HOSTNAME}.home

Now you should be able to see Rancher interface at https://SERVER_IP! There are two manual tasks you need to do on your machine:

• the SSL certificate will be marked as invalid so you'll need to add it to trusted certificates
• Rancher is installed with a ${HOSTNAME}.home hostname which you need to add to your /etc/hosts file (.local domain can't be used with Rancher)

Hope this worked and let me know if you have any comments!

Thanks to Harald Löbig for helping with the Helm 3 update!

P.S: If you're using Raspberry Pi, the Rancher folk created k3s which is a single node Kubernetes installation that requires <512MB of RAM!

During restructuring of my Digital Ocean Kubernetes cluster I needed to move some Persistent Volumes across namespace and some of them across the clusters.

One approach was to manually copy all the data using dvsync script but fortunately both clusters existed under the same Digital Ocean account which meant it should be possible to reuse already created DO volume.

Note: this approach should work as well with Amazon AWS, Google Cloud GKS and Azure AKS.

Here are the steps I took:

First make sure removing PV won't remove the underlying volume

By default, the Persistent Volumes have persistentVolumeReclaimPolicy set to Delete. This would automatically delete the underlying EBS/DO volume. We can change it to Retain by patching the PV:

$ kubectl patch pv ${PV_NAME} -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

Then export both Persistent Volume and its Claim

$ kubectl get pv/${PV_NAME} --export -o yaml > ${PV_NAME}.yaml
$ kubectl get pvc/${PVC_NAME} --export -o yaml > ${PVC_NAME}.yaml

Create all necessary objects in the new cluster

If you're using Helm to manage deployments and create PVC's, you need to run it first and allow it to create the PVC on its own.
After this, delete the created PVC (you'll replace it with the exported one in the next step).

$ kubectl delete pvc/${PVC_NAME}

Note: this will also delete the new PV due to default persistentVolumeReclaimPolicy mentioned above.

Insert the exported PV and PVC

Kubernetes uses UID to determine connection between PVC and PV therefore you'll need to patch the PV with correct one:

$ kubectl apply -f ${PVC_NAME}.yaml
$ PVC_UID=$(kubectl get pvc/${PVC_NAME} -o jsonpath='{.metadata.uid}')
$ # If you're operating in the same cluster, you don't need to do the next step
$ kubectl apply -f ${PV_NAME}.yaml
$ kubectl patch pv ${PV_NAME} -p "{\"spec\":{\"claimRef\":{\"uid\":\"${PVC_UID}\"}}}"

Stop/remove the old deployment

You won't be able to use this volume in the new cluster as long it is attached to the old node, so you need to remove the old PV/PVC.

You're done!

Now the new deployment should mount the volume. Usually it takes a bit to reattach the EBS/DO volume, so don't get discouraged if you need to wait a minute or two.