Billy Overton

Smart Meter Dashboard: Collecting the Data

Sun, 19 Jun 2016 00:00:00 -0400

Previous Post: Smart Meter Dashboard: Installing and Configuring InfluxDb

Once I had a place to store my power usage data, I needed to start actually collecting it. I already had my radio and software picked out during my initial testing, but now I needed a more permanent setup. With the idea being to lower my power usage, I couldn’t just leave my desktop running all the time. On my first attempt I tried an older Raspberry PI Model B I had laying around, but it couldn’t keep up with the processor load. While it would catch a few entries, for the most part it would drop the data.

A perfect excuse to get a new PI. I went with Raspberry PI 3, although I expect this will work with a Model 2 B.

Installing RTL-SDR

I installed the requirements for RTL-SDR based on Adafruit’s SDR scanner tutorial. I only needed the requirements for rtl-sdr since I’m not using a python library to interact with the radio directly.

sudo apt-get update
sudo apt-get install cmake build-essential libusb-1.0-0-dev git

cd ~
git clone git://git.osmocom.org/rtl-sdr.git
cd rtl-sdr
mkdir build
cd build
cmake ../ -DINSTALL_UDEV_RULES=ON -DDETACH_KERNEL_DRIVER=ON
make
sudo make install
sudo ldconfig

Setting rtl-tcp To Start At Boot

Rtlamr uses rtl-tcp to control and read data from the radio. For testing purposes you can start it on its own terminal by running sudo rtl-tcp, but I wanted something that would start automatically on boot. To do this, I created a systemd unit file at /etc/systemd/system/rtl-tcp.service that will always restart the service if it fails.

[Unit]
Description=Software Defined Radio TCP Server
Wants=network.target
After=network.target

[Service]
ExecStart=/usr/local/bin/rtl_tcp
Restart=always

[Install]
WantedBy=multi-user.target

Once that is created I set it to start on boot with sudo systemctl enable rtl-tcp.service.

Installing Go and Rtlamr

The version of Go in the Rasbian repository was high enough to allow rtlamr to run, so I used it instead of compiling or installing a later ARM Go version: sudo apt-get install golang

Once that was installed, I went through the standard Go setup procedure to create my Go path folder and adding the Go environment variables to my .profile file.

...
GOPATH=$HOME/go
PATH="$HOME/go/bin:$PATH"
...

You can reload your .profile by running . ~/.profile. With that done, the installation of Rtlamr was as simple as go get github.com/bemasher/rtlamr

For simplicity, I copied the binary to a dedicated location as root.

sudo mkdir /opt/powermon
sudo cp /home/pi/go/bin/rtlamr /opt/powermon

Testing

To make sure everything was working I ran rtlamr at the console to make sure it was receiving data. There are quite a few apartments near mine using the same meter type, so it didn’t take long to see a large number of results coming through.

$ sudo systemctl start rtl_tcp.service
$ /opt/powermon/rtlamr
05:10:44.703399 decode.go:79: CenterFreq: 912600155
05:10:44.703907 decode.go:79: SampleRate: 2359296
05:10:44.704002 decode.go:79: DataRate: 32768
05:10:44.704083 decode.go:79: SymbolLength: 72
05:10:44.704161 decode.go:79: PreambleSymbols: 21
05:10:44.704242 decode.go:79: PreambleLength: 3024
05:10:44.704322 decode.go:79: PacketSymbols: 96
05:10:44.704403 decode.go:79: PacketLength: 13824
05:10:44.704481 decode.go:79: Preamble: 111110010101001100000
05:10:44.704564 recv.go:93: GainCount: 29
<DATA>

You can filter the results to only include your meter using the --filterid switch. My meter had the ID printed on it, but it wouldn’t have been hard to compare my current meter reading to all of the results to find the ID.

Influx Python Upload Script

The final step of the collection was to send the data to InfluxDB. I decided to go with a simple Python script to parse then forward that information on to the database. Rtlmar is capable of outputting its data in JSON format which made reading the data simple line by line. Python also has a nice Influx client library (that I was already planning to use on the API side of things), however the version that is in the Rasbian package is out of date and does not work with the latest Influx version. The one in pip works just fine.

$ sudo apt-get install python-pip
$ sudo pip install influxdb # Do not use the python-influxdb package

The script itself is pretty simple. It starts rtlamr with a filter for my meter’s ID and a JSON output format. The output is parsed line by line, converted into a format for Influx, and then sent onward. Below is the current state of the script (with placeholders for my db info and my meter ID).

#!/usr/bin/env python
import sys
import json
import subprocess
import time
from influxdb import InfluxDBClient


# Delay. Useful for now while I try and figure out systemd targets. I've had
# issues where if both powermon and rtlamr start immediately at boot, no data is
# reported. The delay solves the problem but I don't like it.
time.sleep(2*60)

client = InfluxDBClient('{influx host}', 8086, '{username}', '{password}', '{database}', True)

proc = subprocess.Popen(['/opt/powermon/rtlamr', '-filterid={meter ID}', '-quiet=true', '-format=json'],stdout=subprocess.PIPE)

while True:
    line = proc.stdout.readline()
    if not line:
        break

    data=json.loads(line)
    message = [{"measurement":"consumption", "time": data['Time'], "tags": {"meter": data['Message']['ID'] }, "fields": {"value": data['Message']['Consumption']}}]
    client.write_points(message)

Just like rtl_tcp, I created a systemd unit file to start powermon at boot.

[Unit]
Description=Python PowerMon Influx Client
Wants=network-online.target rtl_tcp.service
After=network-online.target rtl_tcp.service

[Service]
ExecStart=/opt/powermon/powermon.py
Restart=always

[Install]
WantedBy=multi-user.target

I then enabled it sudo systemctl enable powermon.service

Problems

For now the biggest problem I’m having is the clock on the Raspberry Pi. Any time I lose power (a very infrequent event), the clock gets out of sync. I’ve tried some of the recommended fixes, but so far nothing has seemed to work. I don’t lose any data when this occurs, but the it does throw off the data.

I’m also not a fan of having the 2 minute delay in the script, but that’s a relatively minor problem for now.

Smart Meter Dashboard: Installing InfluxDB

Mon, 30 May 2016 00:00:00 -0400

For this project I knew I didn’t want to store a log of the information from the meter on the Pi. Through some testing, I determined that my power meter broadcasted it’s consumption about once every 30 seconds. While there isn’t much data overall that I wanted to store, I was kind of hesitant to write to a file on the Pi’s SD card that frequently. Looking around for a simple time series database, I found InfluxDB which seemed perfect for my use case.

Installation

After spinning up a new Debian droplet, the installation was pretty simple.

# Enable the influxDB
curl -sL https://repos.influxdata.com/influxdb.key | sudo apt-key add -
source /etc/os-release
echo "deb https://repos.influxdata.com/debian jessie stable" | sudo tee /etc/apt/sources.list.d/influxdb.list

# Install the packages
sudo apt-get update && sudo apt-get install influxdb
sudo service influxdb start

Creating a Database

Creating a database was pretty simple using the provided command line client.

$ influx
> CREATE DATABASE power
>

Enable SSL

Since I planned on having authentication to prevent direct database access, I didn’t want the communication between my local machine and the server to be in clear text. By default influxDB does all of it’s communication over HTTP, but it does have support for SSL. To enable it I needed to create a PEM encoded SSL key and certificate. For my purposes a self-signed certificate was good enough, but for added security (namely SSL validation) a signed certificate should be used.

# Create self-signed certificate for influx
cd /etc/ssl
openssl genrsa -out key.pem 2048 # Generate a private key
openssl req -new -key key.pem -out cert.csr # Create a certificate signing request
openssl x509 -req -days 365 -in cert.csr -signkey key.pem -out cert.pem # Sign the certificate with the key

Influx uses a combined file for both the private key and the certificate file, so I had to append them together.

cat key.pem > influxDB.pem
cat cert.pem >> influxDB.pem

Once the combined file was created, I enable ssl by editing /etc/influxdb/influxdb.conf. I scrolled down to the sections for [admin] changed the https-enabled option to true and changed the https-certificate to the path of the newly created combined PEM file. For me that was https-certificate = "/etc/ssl/influxDB.pem". I did the same in the [http] section.

I tested if this was successful by restarting the influxdb service with sudo service influxdb restart and attempted to connect using the influx command line tool. Since SSL is enabled you need to pass a few more flags in order for this to work: influx -ssl -unsafeSsl. Since I used a self-signed certificate, I had to use the -unsafeSsl flag to disable the clients validation step.

Enable Authorization and Authentication

By default influxDB does not have authorization and authentication enabled. To enable it you must first create at least one administrator user account. I used the command line client to create two users: an admin user with all privileges and a power user that only has permissions to the power database I created earlier.

influx -ssl -unsafeSsl
> CREATE USER admin WITH PASSWORD '<password>' WITH ALL PRIVILEGES
> CREATE USER power WITH PASSWORD '<password>'
> GRANT [READ,WRITE,ALL] ON power TO power

Once the users were created, I enabled authentication by editing /etc/influxdb/influxdb.conf again. In the [http] section, I changed auth-enabled to true. I restarted the influx service and relaunch the influx command line client. It should deny all requests to read data till you authenticate with > auth <username> <password>. Alternatively you can launch with influx -ssl -unsafeSsl -username <username> -password <password> to authenticate before the client launches.

Next Step

Once I had a place to store my power usage information, I needed to setup the radio to pickup and send the information over to influxDB. That will be the topic of the next post.

Smart Meter Dashboard: Introduction

Thu, 05 May 2016 00:00:00 -0400

While browsing around some older posts on Hackaday, I came across a post about reading the broadcasted data from smart meters using a software defined radio (SDR). It got me wondering if my apartment’s meter broadcasted similar data. I knew that nobody physically checked the meter each month, so there had to be some kind of remote reporting capability. A few google searches later I found a timeline for my local power company’s installation of smart meters in the area, which roughly matched up with the date of the Hackaday article. On the off chance that the software would work with with my meter, I ordered a RTL-SDR supported tuner (a NooElec NESDR Nano 2+). A couple days later I had a successful test and a new project to try.

I’m wanting to make a dashboard to display a few stats to help track (and maybe reduce) my home power usage:

Current usage rate (kWh/m)
Final estimated bill based on historical rates and usage
Current estimated bill since last reading
Usage for the last 24 hours (by hour)
Usage for the last 30 days (by day)

There will be a follow up post soon as I’ve already done some work on collecting and recording the data. Posts on the dashboard creation will follow as time allows.

Horrible, Semi-Anonymous, Secure Communication

Thu, 03 Sep 2015 00:00:00 -0400

Recently I’ve been thinking about how to make a secure chat program that also hides who is talking to who. I really don’t have a need for it, but it has been a fun thought exercise. The main problem I’ve been having is figuring out how to make an efficient scheme that accomplishes it.

To handle the secure side of things, using a public key system seemed to make sense. It allows confirmation of who you’re talking to by comparing the public key fingerprint they provide to one you confirm in person, while also making it easy to transmit block level encryption keys securely for messaging. However it does have the drawback that now it’s obvious who is talking to who. If Alice uses Bob’s public key to send him a message… it’s obvious that Alice is talking to Bob. At the very least it reveals that Bob is receiving messages.

To get around this, I came up with a horribly inefficient “public wall” system that hides most of the knowledge about who is talking to who, while still allowing confirmation of who you’re talking to. It’s horribly inefficient however because of the number of messages that end up on a system with too many users.

Here is an example of a conversation with four online users $U:\{A,B,C,D\}$ and a shared wall $W$ . In this case user $A$ wants to talk with user $C$ without letting the other users know.

$\begin{eqnarray} A &\rightarrow& W:\{\mbox{I want to talk}\} \\ B &\rightarrow& A:\{\{R_1\}_{K_B}\}_{K_A} \\ C &\rightarrow& A:\{\{R_2\}_{K_C}\}_{K_A}\\ D &\rightarrow& A:\{\{R_3\}_{K_D}\}_{K_A} \\ A &\rightarrow& W:R_2,\{\{R_4,\mbox{Hello}\}_{K_A}\}_{K_C} \\ C &\rightarrow& W:R_4,\{\{R_5,\mbox{Greetings}\}_{K_C}\}_{K_A} \end{eqnarray}$

User $A$ starts by stating publicly it wants to talk to somebody. All other users respond to this request by sending an encrypted and signed randomly chosen identifier $R_x$ . The suffix in this case doesn’t show a sequence, just that the identifier is unique from the others. Once the identifier from the intended recipient is received $A$ sends their message publicly. The message is comprised of the unencrypted random identifier, and then a signed and encrypted message containing a new identifier and the actual message. The recipient knows the message is for them because of the identifier, and can decrypt and respond using the identifier it received.

Problems

There are quite a few attacks against the system. It’s vulnerable to collusion. If enough of the users talk to each, it’s possible to discover who is sending messages to who. In the example above, if $B$ and $D$ both talk and agree they are not speaking with A, then any messages seen becomes known to be between $A$ and $C$ . Ideally this should require collusion by $|U|-2$ users in order to become absolutely certain.

There are probably other issues related to replay attacks. Using multiple keys to send the same message, etc. I haven’t quite thought through all the of issues yet. It’s more of a general idea than something concrete at this point.

World of Warcraft Analysis Fail

Tue, 18 Aug 2015 00:00:00 -0400

Over the last two weeks I’ve been running a script to download the auction snapshots for my old World of Warcraft realm Madoran as a follow up to my Explaining World of Warcraft Auction Snapshots post. Sadly I didn’t get nearly as much data as I wanted, so I’m starting the whole process over again in the hopes of getting a more consistent “chunk” of data. In the meantime I thought I would explain a few interesting things I’ve noticed about the snapshot data I did get, and explain why my collection method failed.

Mistakes Were Made

One of the early mistakes was my auction downloader script never accounted the wowapi library failing to get the auction data. When I was working on the library, I noticed that occasionally a request for an auction file would fail. To counteract that, the library attempts to download the file multiple times. If it still fails, it throws an exception. I, embarrassingly, didn’t account for my own exception so after three days my script crashed. I didn’t notice till five days later, cutting my available data in half.

My second mistake was not putting my download script under revision control. Apparently between starting my script the first time and the crash of 8/6, I made a change the altered the output I was saving. Once I restarted the script, it started saving snapshots with the new, different format. This made my parsing annoying as I had to detect what output the snapshot file was before I was able to parse it for metadata.

Finally, when I started parsing for metadata I didn’t account for anything but item id number when it came to per item metadata. This lead to my output showing Pet Cage as the most common item for almost every snapshot. While that was true, it missed out on some subtly in the data that I plan to fix for next time

Some Data

Even with my script checking every five minutes for a new snapshot, it appeared like snapshots jumped from 11pm to 1am with no snapshot for the 12am hour. Maybe this is a artifact of my downloader script, but it might be that that they snapshot generator script on Blizzard’s side resets at midnight. This wasn’t consistent and only seemed to happen if a snapshot was supposed to be generated near midnight.

Of course, I had to at least graph something. Here is a graph of the total auction buyout value over a week. For each snapshot I simply summed the buyout values. You can see an interesting spike of data from the 14th through the 15th. Kind of expected since this was a Friday and Saturday. I would have expected Sunday to be up there as well. Something to look for once I get a at least two weeks worth of data.

Explaining World of Warcraft Auction Snapshots

Wed, 05 Aug 2015 00:00:00 -0400

In preparation to do some simple analysis on World of Warcraft auction house data, I thought I would take the time to explain some of the data that Blizzard provides through their Battle.Net Community API…with a touch of self-promotion by using the wowapi python library to pull the data in my examples. To use this library, you need an API key from the Battle.net dev site.

On a semi-regular hourly basis, Blizzard publishes snapshots of the current state of each realm’s auction houses. This contains a list of active auctions, pricing information, items being sold, etc.

Connected Realms

When you retrieve data from an auction house, it can contain data for several realms. In 2013, Blizzard started connecting lower population realms together in order to support more active communities. These connected realms share a common auction house, so retrieving a snapshot for one realm will return data for all realms in that set.

Which realms are connected can be seen in the realm status API result. The status contains a lot of information about a realm such as PvP zone status, server timezone, and population size. For auction houses the main parameters are slug which represents a short name for a realm and is useful for other api calls and connected_realms which contains the slug for all realms connected to the one represented by the JSON object. It also contains the slug for that realm.

import wowapi

api = wowapi.API('BattleNetAPIKey')
realm_status = api.realm_status()

auction_houses = []

for realm in realm_status['realms']:
    # I'm unsure what the internal-record objects are, but they are not valid
    # realms so they can be filtered out of the lists
    auction_house = filter(lambda x: "internal-record" not in x, realm['connected_realms'])
    auction_house.sort()
    if auction_house not in auction_houses:
      auction_houses.append(auction_house)

print("Number of unique auction houses: ", len(auction_houses)
for auction_house in auction_houses:
    print(auction_house)

This prints out a grand total of unique Auction Houses and a comma delimited list of the realms that make up each house.

Number of unique auction houses: 120
aegwynn, bonechewer, daggerspine, gurubashi, hakkar
aerie-peak
agamaggan, archimonde, burning-legion, jaedenar, the-underbog
aggramar, fizzcrank
...

Pulling a Snapshot

Snapshots are published to the API in two parts: an Auction House Status and the auction house data file (a link to which is provided in the status). For example api.auction_status("madoran") returns the following status for the Madoran realm (madoran is the slug for that realm):

{
  "files": [
    {
      "url": "http://us.battle.net/auction-data/52f853e2b29decd29e027d3a17fe1fba/auctions.json",
      "lastModified": 1438741937000
    }
  ]
}

While this data structure can support multiple files, I have yet to see an auction status that contains more than one file. Each snapshot is comprised of a url where the actual data can be downloaded from and a lastModified timestamp field that contains the milliseconds since Epoch. There is a helper function in wowapi.utility for downloading the additional file that takes in an auction status and returns a combined data structure with the auction status and an appended field data that holds the contents of the files in url.

import json
import wowapi
from wowapi.utility import retrieve_auctions

api = wowapi.API('BattleNetAPIKey')
auction_data = retrieve_auctions(api.auction_status("madoran"))
print(json.dumps(auction_data, indent=2, separators=(',', ': ')))

Which prints the following (the contents of auctions is explained in more detail in the following section):

{
  "files": [
    {
      "url": "http://us.battle.net/auction-data/52f853e2b29decd29e027d3a17fe1fba/auctions.json",
      "lastModified": 1438741937000,
      "data": {
        "realm": {
          "name": "Madoran",
          "slug": "madoran"
        },
        "auctions": {
          "auctions": [...]
        }
      }
    }
  ]
}

Auctions

Inside of a snapshot, the auctions field contains a list of individual auctions with a lot of data. Here is an example of one auction:

{
  "rand": 0,
  "auc": 841736577,
  "timeLeft": "VERY_LONG",
  "bid": 500000,
  "item": 109137,
  "seed": 0,
  "ownerRealm": "Dawnbringer",
  "context": 0,
  "owner": "CharacterName",
  "buyout": 600000,
  "quantity": 10
}

Auctions have a unique identifying number in the auc field. No information that I can find guarantees this is unique across Auction Houses or even within a single Auction House over time, but it does appear to be unique within a snapshot and for a single auction throughout its lifetime (if it appears in more than one snapshot).

The owner of the auction is identified by the owner and the ownerRealm fields. Because users in a connected realm set can share a username if they are on different realms, both fields are required to uniquely identify an auction owner.

The item(s) being sold in an auction is identified by the item, quantity, context, rand, and seed fields. The item field is the item id. Information about the actual item can be retrieved through another API call with api.item(itemID). The context provides information about the loot context for a item (what kind of dungeon it was looted from at what level) and was newly added in the 6.0.2 Update to the API. The rand field relates to the random enchantment that is applied to the item being sold . You can see a list of what the numbers mean on gamepedia. The quantity describes the number of items being sold in this auction.

The seed field is a tricky one because I’m not sure what it means. Looking around it seems to match the uniqueID from the in-game API’s item string. Some explanation on that can be found here. As far as I can tell it contains no useful information without access to internal Blizzard tools.

The “timeLeft” field appears to be intentionally vague. This can take one of four different string values indicating how long the auction will remain on the auction house before it expires:

SHORT - < 30 minutes left
MEDIUM - 30 minutes - 2 hours
LONG - 2 hours - 12 hours
VERY_LONG - 12 hours - 24 hours

The money related fields for an auction are bid and buyout. The numbers are given in copper (100 copper in 1 silver, 100 silver in 1 gold, 100 gold in the platinum I wish they would implement). When a user creates an auction they define a starting bid value and an optional buyout. If an auction has a buyout users can bypass the bid process and simply pay the buyout price. If the user doesn’t set a buyout price, the returned auction data has a buyout of 0.

References / Further Reading

Setting up Jekyll, Gulp, and Automated Git Deployments

Mon, 27 Jul 2015 00:00:00 -0400

I have been using Jekyll for a while for my personal site and long-abandoned blog, but with the latest redesign I decided to play with a more automated workflow for dependency management, development, and deployments. With a little luck that means more frequent postings, since I don’t have to go through the hassle of manually building and deploying my content.

So far I’ve settled on using a mixture of Gulp, Bower, and Jekyll for building the site and Git for deployment to my server. It’s still a work in progress that is being tweaked, but it’s finally settling down to where I felt comfortable writing about it.

Installing Global Dependencies

For my current setup there are a few global dependencies that need to be installed: node, gulp, bower, compass, and jekyll. After installing node (following the instructions from their site), the following commands install bower and gulp.

npm install -g bower
npm install -g gulp

I already had a ruby install through homebrew, so installing Jekyll and Compass is just as easy as gem install jekyll and gem install compass.

Creating the Folder Structure

Once all of the global dependencies are installed, I created the initial folder structure using Jekyll’s new command jekyll new personalsite. This creates a new folder called personalsite with the basic folder structure required by Jekyll. This is where I stopped before, using the normal jekyll build and jekyll serve commands to build and post my site.

For the new workflow, I made a few changes to the default folder structure. First I removed the css folder Jekyll made. Instead I created an assets folder with two sub folders: css and fonts. These folders are going to be generated by gulp and bower, so I also added the whole assets folder to my .gitignore file. That’s the only real change to the default folder structure.

Setting up Bower

For right now my only two bower requirements for my site are fontawesome and normalize.scss. I added those dependencies by creating a bower.json file at the root of personalsite with the following contents.

{
  "name": "personalsite",
  "version": "0.0.0",
  "authors": [
    "Billy Overton <overton.billy@gmail.com>"
  ],
  "homepage": "http://billyoverton.com",
  "ignore": [
    "**/.*",
    "node_modules",
    "bower_components",
    "test",
    "tests"
  ],
  "dependencies": {
    "fontawesome": "~4.3.0",
    "normalize.scss": "~3.0.2"
  }
}

Other than some boilerplate, the main part is the list of dependencies which gives the name and the version. These dependencies can be installed by running bower install at the root of the site. This will create a bower_components folder which contains the files for each of the dependencies.

Setting up Node/NPM

To manage the Node dependencies, I created a package.json file. This mostly has the dependencies for gulp to manipulate css and integrate with browserSync and compass.

{
  "name": "BillyOvertonBlog",
  "version": "0.0.0",
  "description": "Personal Blog",
  "main": "gulpfile.js",
  "author": "Billy Overton",
  "devDependencies": {
    "browser-sync": "^2.7.12",
    "gulp": "^3.9.0",
    "gulp-bower": "0.0.10",
    "gulp-compass": "^2.1.0",
    "gulp-minify-css": "^1.2.0",
    "gulp-notify": "^2.2.0",
    "gulp-ruby-sass": "^1.0.5"
  }
}

This sets up everything so gulp can compile sass files, minify them, and place them in the assets folder so Jekyll can pick it up during the build. The dependencies can be installed with npm install from the root of the site folder.

Setting up Gulp

The most complicated part of this setup is the gulp.js file. The full file for this can be found here. Below is a breakdown of the parts.

var gulp         = require('gulp');
var cp           = require('child_process');
var minifyCss    = require('gulp-minify-css');
var notify       = require("gulp-notify") 
var sass         = require('gulp-ruby-sass') ;
var bower        = require('gulp-bower');
var browserSync  = require('browser-sync');

The first part imports the requirements we installed with our package.json file. The only one not from the requirements file is child_process which is used later for running the Jekyll build commands.

var config = {
     sassPath: './_sass',
     bowerDir: './bower_components',
    assetDir: './assets',
    outputDir: './_site' 
}
var messages = {
    jekyllBuild: '<span style="color: grey">Running:</span> $ jekyll build'
};

gulp.task('bower', function() { 
    return bower()
         .pipe(gulp.dest(config.bowerDir)) 
});

This sets up some paths used in the later tasks, plus a message to indicate when a Jekyll build is running (for browserSync). The outputDirectory matches the build directory for Jekyll. It’s used so instead of having to trigger a build every time a file changes (like a css file), I can directly place the css into the output directory and stream the changes through browserSync.

The bower task configures the gulp-bower plugin by passing it the config.bowerDir.

gulp.task('icons', function() { 
    return gulp.src(config.bowerDir + '/fontawesome/fonts/**.*') 
        .pipe(gulp.dest(config.assetDir + '/fonts'))
        .pipe(gulp.dest(config.outputDir + '/assets/fonts')); 
});

This is the gulp task for moving the fontawesome font files from the bower directory to both the asset directory (so jekyll will pick it up during a build) and to the output directory’s assets folder. I don’t really use the second part, but it matched what I was doing with the css task so I left it in.

gulp.task('css', function() { 
    return sass(config.sassPath + '/main.scss', {
             style: 'compressed',
             loadPath: [
                 config.sassPath,
                config.bowerDir + '/normalize.scss/',
                config.bowerDir + '/fontawesome/scss',
             ],
            compass: true
         }) 
        .pipe(minifyCss())
        .pipe(gulp.dest(config.assetDir + '/css'))
         .pipe(gulp.dest(config.outputDir + '/assets/css'))
        .pipe(browserSync.stream()); 
});

The css task compiles the main scss file from which I import all other files, adds the paths for the bower scss files, minifies it, outputs it to the build assets directory and the final assets directory, then triggers a browserSync stream to cause a reload.

gulp.task('jekyll-build', ['css','icons','bower'], function (done) {
    browserSync.notify(messages.jekyllBuild);
    return cp.spawn('jekyll', ['build'], {stdio: 'inherit'})
             .on('close', done);
});

gulp.task('jekyll-rebuild', ['jekyll-build'], function () {
    browserSync.reload();
});

These two tasks handle the integration between gulp and Jekyll. The jekyll-build command has dependencies on the css, icons, and bower tasks so those are completed beforehand. The first part sends a notification that there is a build occurring to browserSync. The second runs the jekyll build command.

jekyll-rebuild is the actual rebuild command for development, it depends on jekyll-build and runs a browserSync reload when the build is complete.

gulp.task('build', ['bower', 'icons', 'css' ,'jekyll-build']);
gulp.task('serve', ['build'], function() {
    browserSync.init({
        server: {
            baseDir: "./_site"
        }
    });
    // Start a watch for rebuilds
    gulp.watch(['_sass/*.scss'], ['css'])
    gulp.watch(['index.html', '_layouts/*.html', '_includes/*', '_posts/*'], ['jekyll-rebuild']);
});
gulp.task('default', ['serve']);

The rest of the file defines the entry points into the system. The build task runs the other tasks to create an initial build. This is the task I use during the deployment step on my server.

The serve task is what I use during development. In addition to running an initial build, it starts browserSync against the output directory and sets up watch commands on the files so browserSync can keep the browser up to date with any changes. Changes to the scss files triggers the css task which streams changes to browserSync. Changes to html files or post files trigger a full jekyll-rebuild, which will run a new build and reload the browser.

Configure Jekyll

Because of how Jekyll processes files, the gulp, bower, and npm files will end up being sent to the output directory _site. To avoid this, I added the following to my _config.yml file

exclude: [gulpfile.js, package.json, node_modules, bower.json, bower_components]

Using for Development and Writing

Once all that is setup, developing or writing for my site can be handled by running gulp serve. This compiles the scss files, builds the jekyll site, and loads the output in a browser window with browserSync.

If it is a fresh checkout, the gulp serve command needs to be preceded by

npm install
bower install
gulp serve

Adding the Site to Git

For my automated deployment, I went with a git work flow. The first thing I did was put the site code under revision control by running git init from the root of the personalsite folder.

Before adding and committing the first set of files, I made sure my .gitignore file ignores everything that is either auto-generated or doesn’t need to be added for the build processes.

_site
.sass-cache
node_modules
assets
bower_components
.DS_Store

Then I commited the files with a normal git workflow.

git add *
git add .gitignore
git commit -m "Initial Import"

Configuring the Server

So far all this is local to my workstation. On my remote server I initiated a bare git repository to act as the “deploy” base.

cd ~
mkdir personalsite.git
cd personalsite.git
git init --bare

To support automated building and deploying, I also installed all of the global dependencies I installed on the local machine.

Finally I added a post-receive git hook by creating the file ~/personalsite.git/hooks/post-receive and setting it as executable chmod +x ~/personalsite.git/hooks/post-receive This file will be executed every time there is a push to this repository. In my case, I use this script to create a temporary checkout, build the site, and then move it to the directory being served by my web server.

GIT_REPO=$HOME/personal-site.git
TMP_GIT_CLONE=/tmp/billyoverton_blog
PUBLIC_WWW=/var/www/billyoverton.com/

# Create a temporary checkout
git clone $GIT_REPO $TMP_GIT_CLONE
cd $TMP_GIT_CLONE

# Build the site and rsync it to the public html folder
npm install
bower install
gulp build
rsync -a --delete _site/ $PUBLIC_WWW

# Remove the temporary checkout
rm -Rf $TMP_GIT_CLONE
exit

Configuring Git Deployments

Back on my local machine, I added the repo on my server as a remote named deploy. I’m using ssh to add the remote, so there is no need for any additional server configuration beyond normal remote access (and installing git)

git remote add deploy username@remotehost.com:~/personal-site.git

With that done, the only thing required to deploy my site is to push my changes to the remote server with git push deploy master

Automating Backup Downloads With WinSCP

Wed, 02 Oct 2013 00:00:00 -0400

On the Server

The first thing I did was create a user that handles nothing else but the copying of backups. While I could have used an account that already exists, I decided to create an account that I could restrict as much as possible.

addusers -s /usr/sbin/nologin backup_user

The -s /usr/sbin/nologin option changes the default shell so that this user does not have shell access and creates a little extra security. Since it was not already there on my Debian install, I added /usr/sbin/nologin to the /etc/shells file.

For my purposes, the backup_user was only used for copying backups from my server to my Windows desktop. As such, I decided to limit it to only using ssh for sftp. To do this, I added the following lines to the \etc\ssh\sshd_config file:

Subsystem sftp internal-sftp
Match user backup_user
    ChrootDirectory %h
    X11Forwarding no
    AllowTcpFowarding no
    ForceCommand internal-sftp
Match

I also commented out all other lines that referenced Subsystem sftp.

The ChrootDirectory %h option locks the user to their home directory. Because of the way SSH’s chroot works, the directory it uses must by wholly owned by root, and can only be writable by root. The command below takes care of that for my backup_user’s home directory.

chown root:root /home/backup_user

On the Client/Desktop

First I created a SSH key without a password for the backup_user. This makes it so I can run this automatically in the background without user intervention. This key without a password is the main reason I placed so many limitations on the backup_user’s account and ssh access.

I also created a WinSCP script for transferring backups from the server that used the above ssh key. I saved it as C:\backup.txt to make the command line entry easier.

option batch abort
option confirm off
option echo off
open sftp://backup_user@example.com -privatekey="C:\backup_key.ppk"
synchronize local -criteria=none ""
exit

The script can be ran with WinScp with the command-line parameter /script="C:\backup.txt" to automatically copy documents from the remote directory to the local one. I created a Scheduled Task to run the above once a week.

Windows PowerShell and SmartAVI CATSwitch

Mon, 21 May 2012 00:00:00 -0400

I was working on some room automation stuff last year when I found myself in need of a command line tool to control a SmartAVI CATSwitch video switcher. Thankfully they posted the details for their RS-232 protocol at the end of their manual, so I was able to throw together a quick and dirty script for connecting inputs and outputs on the switch. I thought it would be interesting to share what I put together, since it might help someone else.

I was playing around with PowerShell at the time, so that is what the script is written in. It takes two strings as inputs corresponding to the physical ports on the video switcher. For the model I was working with valid inputs were “01” – “16” for both the input and the output string.

$output = $args[1]
$input = $args[0]

$command = '//F00M'
$command += $output
$command += "I"
$command += $input

$errorCode = 0
foreach($character in $command.ToCharArray()) {
    $errorCode = $errorCode -bxor [int][char]$character
}

$command += [char]$errorCode

$command += "`r"

$port= new-Object System.IO.Ports.SerialPort COM1,9600,None,8,one
$port.open()
$port.Write($command)
$port.close()
exit

Graphical Sequences with Python

Sun, 22 Jan 2012 00:00:00 -0500

I got bored, so I quickly put together a function in python to determine if a given degree sequence is graphical. It uses the Erdős–Gallai theorem to do the checking. I know it isn’t the most difficult function in the world to write, but I find it useful.

def isGraphicalSequence(sequence):
  # sort the list in reverse order
  sequence.sort(reverse=True)
  # Ensures it is a non-negative sequence
  if sequence[len(sequence)-1] < 0: return False      
  # Check to ensure there are an even number of odd degrees
  if sum(sequence)%2 != 0: return False      
  # Erdos-Gallai theorem
  for i in range(1, len(sequence)+1):
    sumOfDegree = sum(sequence[:(i)])          
    partialSum = i * (i-1)
    for j in range(i+1, len(sequence)+1):
      partialSum = partialSum + min(i, sequence[j-1])
    if sumOfDegree > partialSum: return False
  return True

Update: 2012-01-23

After actually going to class, I did another implementation that uses the Havel-Hakimi algorithm instead. Both work, but I thought I would share this one too. Plus this one is recursive, and who doesn’t love recursion?

def isGraphicalSequence(sequence):
  if sequence.count(0) == len(sequence):
    return True 
  sequence.sort(reverse=True)
  if sequence[len(sequence)-1] < 0: return False
  if sum(sequence)%2 != 0: return False
  if sequence[0] >= len(sequence): return False
  count = sequence.pop(0)
  for i in range(count):
    sequence[i] = sequence[i] - 1
  return isGraphicalSequence(sequence)