Ian Gilham

Fixing Swapped Tilde Key on MacOS

2021-08-20T00:00:00+00:00

For the last few years I’ve been using an early 2015 MacBook Pro for work and have sometimes suffered from RSI due in part to the terrible ergonomics of the Magic Keyboard. I solved this by buying a Mac-compatible mechanical keyboard from Varmilo.

Unfortunately, MacOS has never correctly detected the layout of this keyboard. It thinks it’s ANSI layout (no extra key left of ‘z’) but it’s actually ISO layout (backtick/tidle left of z). The Mac’s default behaviour is to swap the section mark and plus-minus (§, ±) keys with the backtick/tilde key. It’s usable but annoying. Until recently I solved this by remapping the offending keys using Karabiner Elements.

Fast-forward a few years and I’ve got a newer MacBook Pro, circa 2019. No matter how many times I go through the setup, Karabiner Elements always fails to install its driver on this machine. I needed a new solution. Forunately, there’s a CLI for changing user keymappings on MacOS, hidutil. I found some scripts on the Apple Stack Exchange and rewrote them into a shell utility to make it easier to use.

#!/bin/sh
# Adapted from https://apple.stackexchange.com/a/353941 under CC BY 4.0
set -e

PLIST="/Library/LaunchDaemons/org.custom.tilde-switch.plist"

case $1 in
on)
  hidutil property --set '{"UserKeyMapping":[
    {
      "HIDKeyboardModifierMappingSrc": 0x700000035,
      "HIDKeyboardModifierMappingDst": 0x700000064
    },
    {
      "HIDKeyboardModifierMappingSrc": 0x700000064,
      "HIDKeyboardModifierMappingDst": 0x700000035
    }
  ]}'
  ;;
off)
  hidutil property --set '{"UserKeyMapping":[]}'
  ;;
install)
  sudo sh -c "cat > ${PLIST}" << EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>org.custom.tilde-switch</string>
    <key>Program</key>
    <string>${HOME}/bin/switch-tilde-keys</string>
    <key>ProgramArguments</key>
      <array>
        <string>on</string>
      </array>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <false/>
  </dict>
</plist>
EOF
  sudo launchctl load -w -- ${PLIST}
  ;;
uninstall)
  sudo launchctl unload -w -- ${PLIST} && sudo rm -f -- ${PLIST}
  ;;
*)
  echo "Usage: $0 on|off|install|uninstall" >&2
  exit 1
  ;;
esac

The way this swaps keymaps, it fixes my external keyboard but also makes the same change to the MacBook’s internal keyboard, making that one wrong. For that reason it’s convenient to have the commands to disable the change or uninstall the PList file in case I’m traveling or my hardware changes. Thanks to the Covid situation, I haven’t been in the office for a year and a half but that may eventually change.

Simplifying the Jekyll site

2021-08-04T00:00:00+00:00

It has been a while since I last wrote anything for this blog but I think about it occasionally. I was planning to get back into writing but the thought of Jekyll’s slow build process put me off.

To address the static site build time, I got most of the way through a port to Hugo before I realised that many of Jekyll’s performance problems had been solved in the last year or two. I was also concerned that I might struggle to get some of the features to work correctly, since I use an esoteric plugin to build tag index pages.

Well, the tag problem is easily solved - I have removed the plugin and the pages from the site, simplifying the build and reducing the number of things that can slow it down or break the site. This also seemed like a good opportunity to simplify article and post headers and I ended up fiddling with the colours a bit while I was playing around.

In the past, I’ve used Google Analytics on this site but I’ve never really paid any attention to the data it produces. Since the value of the data to me is almost zero, I removed it quite some time ago to keep things as simple as possible.

As for the Hugo port, I haven’t decided if I want to commit to it at this stage, but at least it should be easier without the extra weirdness of the tag plugin. I’ll stick with Jekyll for now, since my reasons to migrate have largely been mitigated.

Azure Functions Experience

2020-02-12T00:00:00+00:00

I’ve recently been working with Azure Functions, Microsoft’s functions-as-a-service (FAAS) platform. I’m already quite familiar with AWS Lambda so my perspective is coloured by that experience.

First, the good. Functions Apps are pretty flexible. You can host a single entry point per app if you want but it is often more cost-effective to host several functions within the same app and share some resources. I’m using Javascript so the whole app shares a package.json and a node_modules directory etc. The app also has its own Storage Account, which can host blobs (like S3), queues (like SQS) and tables (similar to DynamoDB but works differently) etc.. These resources are easy to use from any function within the app, making it easy to send a message down a queue to model event-driven workflows and perform multiple actions with the same files or database records.

The runtime of a function app environment is versioned separately from the language and language version. There is a list of supported languages and versions for each version of the runtime.

A Function App is connected to an App Service Plan, which defines the virtual machine billing model. There are options for consumption plans on Windows and Linux or you can reserve an auto-scaling pool of virtual machines. It’s fairly flexible but quite awkward to configure for Linux. The key when using Terraform or ARM is to make sure the instance is reserved, as that seems to mean “use Linux”.

The inputs and outputs of functions can be bound to HTTP requests/responses or tied directly to other Azure event sources, such as blobs, queues, tables etc. This allows you to avoid writing the code to send a message, write a row to a table or any number of other common integrations. This gives Functions a lot more flexibility than AWS Lambda functions, which only lets you bind the input event source.

The documentation for Linux apps is largely missing and often useless. You have to dig pretty deep to discover that the setting for configuring the version of NodeJS you want only works on Windows instances. There is a hidden setting called LinuxFxVersion in a different place (Site Config) that has no UI in the Azure Portal for configuring it on Linux. Fortunately Terraform has a way to address this undocumented property. It appears in ARM as well but I haven’t found a way to edit it from the Azure Portal.

I’ve seen various weird crashes when deploying and restarting function apps. There are at least 3 different undocumented things that can go wrong, leaving only a cryptic error message. Searching for these issues usually fails to yield any relevant results. I’ve seen more strange errors with version 3 of the functions app runtime than with version 2.

Another frustration is the Azure CLI. The best case when deploying a function app is the message Operation failed with status: 200 Operation completed successfully. As amusing as that is, more often the tool simply fails with a HTTP 400 error: Bad Request with no explanation. Since the CLI hits the API about 8 times during deployment, there is no way to know how far you got or what state it left your app in. This completely breaks any kind of deployment automation. In production, you’ll have to pay somebody to hand-hold the build and make sure changes make it out to your customers.

Setting SNS String Array Message Attributes with the AWS Javascript SDK

2019-06-04T00:00:00+00:00

AWS Simple Notification Service is a great tool for broadcasting messages to a topic. Subscribers receive all messages by default, or can use a filter on the messagee’s attributes to receive only a subset of the messages sent to the topic.

Message attributes have a few possible data types. Unfortunately the documentation for the Javascript SDK is pretty bad at the time of writing. It’s fairly obvious how to set an attribute of type String but it says nothing about how to set an attribute of type String.Array. Fortunately, I guessed correctly when I gave it a try.

const AWS = require('aws-sdk')
const config = require('./config')

AWS.config.region = 'eu-west-1' // or your region

const sns = new aws.SNS()

const notificationGroups = [
  'GroupA',
  'GroupB'
]

async function sendMessage(message, errorCode) {
  const params = {
    Message: message,
    Subject: 'Something happened',
    TopicArn: config.sns.arn,
    MessageAttributes: {
      errorCode: {
        DataType: 'String',
        StringValue: `${errorCode}`
      },
      group: {
        DataType: 'String.Array',
        StringValue: JSON.stringify(notificationGroups)
      }
    }
  }

  await sns.publish(params).promise()
}

The trick is to call JSON.stringify(someArray) and stuff it into the StringValue key in the MessageAttribute object.

Declaring required commands in bash scripts

2018-02-22T00:00:00+00:00

It’s funny how easy it is to overlook an obvious solution to a trivial problem and keep doing things the slow way for years at a time.

While writing a short bash script today, it occurred to me that I’ve been handling requirements poorly for years. I’ve previously used long chains of if !$(which COMMAND) &>/dev/null to declare and enforce requirements but it never occurred to me to wrap it in a simple function. This is what popped out of my head today:

#!/bin/bash
# Declare requirements in bash scripts

set -e

function requires() {
    if ! command -v $1 &>/dev/null; then
        echo "Requires $1"
        exit 1
    fi
}

requires "jq"
requires "curl"
# etc.

# ... rest of script

This makes it easy to declare simple command requirements without repeating the basic if not found then fail logic over and over. In hindsight it should have been obvious to wrap this stuff in a function years ago, but at least I’ve caught up now.

The requires function can of course be placed in a common file for inclusion into multiple scripts. I’ve shown it inline for simplicity.

This snippet also available as a Gist.

Add security headers to an S3 static site with Lambda@Edge

2017-08-22T00:00:00+00:00

As the web continues to evolve, the selection of HTTP response headers a site is expected to support grows. Some of the security-related headers are now considered mandatory by various online publishers, including Google and Mozilla. If like me you run your site using an S3 bucket of static files, it’s not easy to add the recommended headers and improve your site’s security scores in tools like Mozilla’s Observatory.

CloudFront and Lambda@Edge can fix that. Others have detailed the process more fully than I can but beware that some features may have changed since the older posts were written. I suggest following the article linked previously if you want to implement this for your site. I’ve listed some of the gotchas that slowed me down below.

Adding the right permissions to the Lambda IAM role

While creating the Lambda you will be asked to assign it a role. Make sure you add the Basic Edge Lambda template to the role you create to allow the function trigger to be created. I missed this step the first time and it took me a few tries to figure it out.

Beware of replication

When you deploy the Lambda and attach the trigger, CloudFront will create replicas of the Lambda in various regions. If you then update your function, publish a new version and redeploy, it will create more replicas. The replicas of older versions are not automatically deleted and cannot be deleted by the user at this time so they will pollute your account with potentially large numbers of unused Lambda functions. Hopefully Amazon will fix this issue at some point.

Choose your trigger wisely

CloudFront supports 4 triggers: origin-request, origin-response, viewer-request and viewer-response. The restrictions on run-time and memory size are different for different triggers so pay attention to the delays your functions introduce to the trafic flow.

The viewer triggers are run outside of CloudFront’s caching layer so viewer-request is triggered before the request hits the cache and viewer-response is triggered after the response has exited the cache.

origin-response is triggered after CloudFront has found the resource but before it writes its response to its own cache. That means you can add headers and cache the result, reducing Lambda invokations and delays, keeping the cost down.

Headers and their values

I’ve configured CloudFront to redirect HTTP requests to my site to use HTTPS but browsers like a header to make that even clearer. HSTS or Strict-Host-Security does this job. Other than that, there’s a range of headers that help the browser to mitigate the risks of XSS (Cross-site scripting) vulnerabilities. These are well documented by Mozilla so I won’t rehash it all here.

The interesting one is the CSP (Content-Security-Policy) header. There are a few versions of the syntax supported by different browsers and getting it right is a little tricky. The excellent CSP Evaluator by Google is very helpful for testing wordings of the CSP. Tunine the policy to work properly with Google Analytics and allow inline stylesheets while disabling most other avenues of attack took a few attempts to get right but I’m happy with what I ended up with.

I disabled everything by default with default-src: 'none' then added the permissions I needed for my site. I use a very small inline CSS stylesheet so I needed to enable that with style-src 'self' 'unsafe-inline'. I don’t make much use of images at present but if I do I’ll access them over HTTPS so I enabled that with img-src 'self' https:. Opening up just the needed permissions for scripts was a bit more difficult but the CSP Evaluator helped a great deal. It recommends a strict-dynamic policy for browsers that support it. I only use one script on my site for Google Analytics, so I had to extract the contents (including all whitespace) from the <script> tag, hash it with SHA256, then encode the hash with Base64 and add the result directly to the CSP policy. CSP Evaluator also recommends some fall-back options for browsers that do not yet support strict-dynamic so I end up with script-src 'strict-dynamic' 'sha256-my_script_hash' 'unsafe-inline' https:, where my_script_hash is the Base64-encoded SHA256 hash of the contents of my script. The complete example is in the below code.

Lambda code template

My basic Lambda template for adding custom headers to all HTTP responses on my site is shared below.

'use strict';

exports.handler = (event, context, callback) => {
    function add(h, k, v) {
        h[k.toLowerCase()] = [
            {
                key: k,
                value: v
            }
        ];
    }

    const response = event.Records[0].cf.response;
    const headers = response.headers;

    // hsts header at 2 years
    // Strict-Transport-Security: max-age=63072000; includeSubDomains;
    add(headers, "Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload");

    // Reduce XSS risks
    add(headers, "X-Content-Type-Options", "nosniff");
    add(headers, "X-XSS-Protection", "1; mode=block");
    add(headers, "X-Frame-Options", "DENY");
    // TODO: fill in value of the sha256 hash
    const csp = "default-src 'none'" +
        "; frame-ancestors 'none'" +
        "; base-uri 'none'" +
        "; style-src 'self' 'unsafe-inline'" +
        "; img-src 'self' https:" +
        "; script-src 'strict-dynamic' 'sha256-my_script_hash' 'unsafe-inline' https:"
    add(headers, "Content-Security-Policy", csp);

    console.log('Response headers added');

    callback(null, response);
};

Preventing rpmbuild from stripping symbols using cpack

2016-12-08T00:00:00+00:00

I’ve been trying to analyse a core dump generated by a C++ application when it seg-faulted. I use CMake 3 to build it and create an RPM with CPack. This application is currently built in debug mode using -DCMAKE_BUILD_TYPE=Debug on the command line that invokes CMake.

The generated binaries have all their debug symbols as expected but the binaries in the RPM package do not. After some searching, I learned that rpmbuild strips binaries by default on some distributions. This makes analysing a core dump way harder than it needs to be so I found a way to turn this feature off using CPack. The trick is to set a variable in the CMakeLists.txt file:

# prevent rpmbuild from stripping binaries when in debug mode
if(CMAKE_BUILD_TYPE STREQUAL "Debug")
  set(CPACK_RPM_SPEC_INSTALL_POST "/bin/true")
endif()

Now my debug packages retain their debug info after installation so it’s possible to get a lot more information out of gdb when looking at a core dump.

This is documented in a roundabout way online, but it took me a while to figure it out so I thought I’d write it up.

Getting a core dump out of CentOS 7.2

2016-12-08T00:00:00+00:00

Since Systemd took over as the main init system in Red Hat Linux and derrivatives like CentOS, it has become more difficult to get a core dump out of a daemon application. The traditional approach of running ulimit -c unlimited before executing the binary works when running the application from the command line but does nothing for a daemon managed by Systemd’s unit files.

There is a lot of misleading information online about how to solve this so I thought I’d add a correct solution to the mix in the hope that it’s helpful.

The suggestions I found online include editing /etc/security/limits.conf, adding LimitCore=infinity to the Unit file, and messing around with /etc/systemd/coredump.conf. None of these methods work without customising the kernel configuration first.

Systemd is not configured to handle core dumps by default on CentOS (and by extension RHEL) distributions. The default behaviour is to write to the file core in the process’s working directory, which for daemons is often the root directory.

The obvious problem here is that the daemon probably doesn’t have write access to the root directory (if running as a non-root user). If is possible to change the working directory with the Systemd unit directive WorkingDirectory=/var/run/XXX. This is typically used with RuntimeDirectory=XXX, which creates and manages the lifecycle of /run/XXX (/var/run is a symlink to /run). Unfortunately, we can’t write the core file to a RuntimeDirectory because it gets deleted when the application terminates.

The simplest solution I found is to overwrite the kernel core_pattern setting. This can be edited at runtime by echoing a new value into /proc/sys/kernel/core_pattern:

echo /tmp/core-%e-sig%s-user%u-group%g-pid%p-time%t > /proc/sys/kernel/core_pattern

This will force the kernel to write all core files during the current OS uptime to /tmp with the filename pattern specified. The core manpage has more information on the syntax.

This change will be lost when the machine reboots. To effect the change at kernel startup, you need to edit /etc/sysctl.conf or a file in /etc/sysctl.d/.

kernel.core_pattern=/tmp/core-%e-sig%s-user%u-group%g-pid%p-time%t

Our solution at work was to write a script to create a file in /etc/sysctl.d/ at machine image creation time, so that the config is always there when we roll out to different environments (int, test, live etc.)

It should go without saying that there is no particular reason to use /tmp. The output can be redirected to any location the process has permission to write to. A network share may be more appropriate in some cases.

There may be another solution using systemd-coredump, but it is not part of this release of CentOS (7.2) and not in the yum repository at this time.

Azure Functions and Storage Tables

2016-04-08T00:00:00+00:00

I’ve written previously about how Amazon’s AWS Lambda feels like the future. I had a little time today to play with Azure Functions, Microsoft’s new competing service.

As a preview, Azure Functions isn’t quite polished yet, but the basic building blocks are in place. You create a Function App in the Azure Portal, much like you would create a Web or Mobile App, then add a few Functions in the Portal blade.

Since the service is built on the mature WebJobs product, there is already strong language support. Azure Functions can be created with specialised C#, NodeJS, PHP, Python, Powershell, Windows Batch scripts (I know), or any standard Windows executable. The Portal UI leaves a lot to be desired for now. I did’t find any obvious way of using different languages or packaging up apps and dependencies, but did manage to get it working. The documentation implies that there are many more features than those exposed by the Portal blade.

For the sake of a quick test, I created a small function to connect to the Azure Storage account and create an empty Table. I had a little trouble setting up the connection string to the storage account but found a solution using magic environment variables here. This may seem normal to regular dotNet and/or Azure developers but it was new to me. The documentation implied that I needed to use a CloudConfigurationManager from this NuGet package but I failed to figure out how to pull in the dependency and resolve the namespace. The solution is to copy a connection string from the Azure Storage account and paste it into the Functions App Service Settings blade’s Application Settings under “Connection Strings”. This effectively makes the string available to the Function in an environment variable at runtime. If our connection string is named StorageConnectionString, and the type is “Custom”, then the value will be available in the environment variable CUSTOMCONNSTR_StorageConnectionString.

Here’s the code I ended up with as shown in the editor in the Azure Portal.

#r "Microsoft.WindowsAzure.Storage"

using System;
using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Table;

public static void Run(object trigger)
{
    var storage = CloudStorageAccount.Parse(
        Environment.GetEnvironmentVariable(
            "CUSTOMCONNSTR_StorageConnectionString"));
    var tableClient = storage.CreateCloudTableClient();
    var membersTable = tableClient
        .GetTableReference("MyTable");

    membersTable.CreateIfNotExists();
}

This file is a little weird because it is a csx file. There is some documentation in the Azure Functions Developer Reference but it’s essentially C# with embedded assembly references (#r "Microsoft.WindowsAzure.Storage") and a top-level public static void Run method. Any other classes etc. you need can be included inline in the file.

With the code in place, you have to switch to the ‘integration’ tab to enable a trigger so that the function can be run. I did the minimum needed to test the function and just selected a manual trigger, changed the name to “trigger”, then switched back to the code view and hit the “Run” button at the bottom of the Function App blade. It took a few goes to get the code right, but with everything working, I was able to confirm that it worked by opening the storage account blade in the Azure Portal and looking at the list of Tables in the account. Sure enough, we have our shiny new table.

While creating a table programmatically is about the simplest use case I could think of for testing a Function, I must admit that it feels a bit forced to have to write code for this at all. I prefer Amazon’s model of providing Console-based UIs for most of the core functionality, like creating tables and adding records. There are Storage Explorer tools for this on Azure, but it would be nice to be able to manage everything from the Portal.

There’s a lot more ground to cover here: Function triggering, integration with other services, creating API endpoints and deployment automation are all considerations. I look forward to seeing how Azure Functions evolve in the future.

Triggering a Lambda from SNS using CloudFormation

2016-03-22T00:00:00+00:00

Amazon Web Services (AWS) provides many building blocks you can use to create just about anything in the world of web-connected services. A lot of the skill in using this toolkit is in figuring out how to make the various services work together. CloudFormation is critical to this effort, as it let’s you write a config file that can be used to automate the creation of all the infrastructure you need to deliver your product. The big win from using CloudFormation is reproducibility. It let’s you define your whole infrastructure requirement, put it in version control, and reproduce the same setup in a production environment without manually recreating everything.

CloudFormation is not without its problems, however. The Syntax is overly verbose and awkward to author. Some of the services it can create cannot be edited, and it is easy to get into circular reference hell with more complex configurations involving the interactions between multiple products. There are reams of documentation but most of it is useless noise without meaningful explanation. The emerging best-practices are to work in another format (like YAML) and convert it to CloudFormation’s preferred format mechanically, or to pay a third party for their cloud automation tools.

I’ve been struggling to get an SNS topic to trigger a Lambda this week. This is really easy using the AWS console because it automates many of the tricky and barely documented steps required to get it working. Wiring it up in CloudFormation is another hell entirely. You need a few different pieces in the right order to make this work so I’ll go through each one in turn. Each of the below can be pasted into the Resources section of a CloudFormation template.

For completeness, I’ve included a basic template skeleton here.

{
  "Description": "blah blah",
  "Parameters": { ... },
  "Outputs": { ... },
  "Resources": { ... }
}

Lambda Execution Role

Before you can build a Lambda Function, you need to create some permissions for it to assume at runtime. Here I present a fairly minimal role suitable for a basic Lambda Function with no external integration points. Additional permissions (e.g. reading from an S3 Bucket) can be added to the list of Statements in the PolicyDocument. This part is actually documented reasonably well.

{
  "ExecutionRole": {
    "Type": "AWS::IAM::Role",
    "Properties": {
      "Path": "/",
      "Policies": [
        {
          "PolicyName": "CloudwatchLogs",
          "PolicyDocument": {
            "Statement": [
              {
                "Action": [
                  "logs:CreateLogGroup",
                  "logs:CreateLogStream",
                  "logs:GetLogEvents",
                  "logs:PutLogEvents"
                ],
                "Resource": [ "arn:aws:logs:*:*:*" ],
                "Effect": "Allow"
              }
            ]
          }
        }
      ],
      "AssumeRolePolicyDocument": {
        "Statement": [
          {
            "Action": [ "sts:AssumeRole" ],
            "Effect": "Allow",
            "Principal": {
              "Service": [ "lambda.amazonaws.com" ]
            }
          }
        ]
      }
    }
  }
}

Lambda Function

The Lambda Function itself is quite easy to set up. I’ve hard-coded placeholders for the S3 Bucket and path to the zip file containing the code. It doesn’t matter what the code is for the purpose of this article.

{
  "Lambda": {
    "Type": "AWS::Lambda::Function",
    "Properties": {
      "Code": {
        "S3Bucket": "my-personal-bucket",
        "S3Key": "lambdas/test/my-lambda.zip" }
      },
      "Description": "Some Lambda Function",
      "MemorySize": 128,
      "Handler": {"Ref": "LambdaHandler"},
      "Role": {
        "Fn::GetAtt": [ "ExecutionRole", "Arn" ]
      },
      "Timeout": 5,
      "Runtime": "python2.7"
    },
    "DependsOn": [
      "ExecutionRole"
    ]
  }
}

The SNS Topic is very easy to create but it cannot be modified using CloudFormation after it has been created. You also have to create all the subscriptions at the same time, so if you use CloudFormation for reproducibility, you can never change the subscriptions of a running event pipeline that relies on SNS. This is a major drawback of SNS and CloudFormation and should be considered with care before you rely too heavily on this set of tools.

The unmodifiable nature of SNS Topics created this way won’t be a problem if you’re creating subscriptions via an API at runtime, but it limits how flexible SNS can be for some workflows, like fanning out to SQS Queues or triggering Lambda Functions.

{
  "Topic": {
    "Type": "AWS::SNS::Topic",
    "Properties": {
      "Subscription": [
        {
          "Endpoint": {
            "Fn::GetAtt": [ "Lambda", "Arn" ]
          },
          "Protocol": "lambda"
        }
      ]
    },
    "DependsOn": [ "Lambda" ]
  }
}

Permission for the Topic to invoke the Lambda

Unfortunately, creating all the pieces isn’t enough. We still need to grant our SNS topic permission to invoke the Lambda Function directly. This is really important and the documentation is almost completely useless so I’m putting a working example here.

It is absolutely critical to have the SourceArn property refer to the Topic we created earlier. Without this, the Lambda Console will give you inexplicable errors, while the SNS Console claims that everything is correct and working. It can be very frustrating trying to get this right.

{
  "LambdaInvokePermission": {
    "Type": "AWS::Lambda::Permission",
    "Properties": {
      "Action": "lambda:InvokeFunction",
      "Principal": "sns.amazonaws.com",
      "SourceArn": { "Ref": "Topic" },
      "FunctionName": {
        "Fn::GetAtt": [ "Lambda", "Arn" ]
      }
    }
  }
}

Update: An observant reader spotted a bug in the above CloudFormation. I originally had a reference to the SourceAccount in the Properties block:

  "SourceAccount": { "Ref": "AWS::AccountId" },

This was incorrect. It works after removing that line.

Conclusion

That should be enough to get it all working. There are some serious holes in the capabilities of CloudFormation for working with SNS, and the permission model is a poorly documented mess. The only thing I haven’t covered is how to get the Lambda itself to run, but that’s a topic for another day.

AWS Lambda feels like the future of software development

2016-01-26T00:00:00+00:00

Every now and then we find a technology that feels like the future, letting you achieve what you want to without getting in the way. Automatic and dual-clutch (semi-automatic) transmissions fall into this category, and some programmers occasionally get the bug from the latest experimental programming language or when rediscovering functional programming.

I’m getting that next-big-thing feeling from Amazon’s AWS Lambda service. I’ve been building enterprise data munging applications in Java and C++ for years and have done the usual little glue scripts in Python, Ruby, Bash, CMD just like most other software developers. As we all know, the code is the fun part that takes very little time compared to the brain-numbing effort of messing around with operating systems, deployment, scaling considerations, TLS (SSL) termination, fiddling with CORS settings etc.

When you move to cloud-rented servers, the configuration only gets worse, as the dev team has to micromanage details traditionally dealt with by operations staff running a data centre. At this level of abstraction you need to cake care of monitoring, availability and fault-tollerance while taking account of how to scale up and down with the load. And you always end up paying to leave at least one idle server sitting there waiting for something to do.

The beauty of Lambda is that it does most of the boring infrastructure stuff for you magically. You don’t have to think about how many hosts might be running and how to scale them up and down. You don’t have to pay for a running host when the application is sitting idle. You don’t even have to fiddle with the half-working settings of a Java Servlvet Container or the massive weight of the Spring framework.

All you have to do with Lambda is write a single-purpose application (currently in Java, Python or NodeJS), deploy it and trigger it to run. It can be given an endpoint with API Gateway or receive notifications from other Amazon services or from any SNS topic. You only pay for executing runtime and reserved memory capacity during execution.

The trade-off is obvious. Sacrifice some flexibility in exchange for a simpler life in which you can spend more of your time working on the code and delivering value. The biggest gain from adopting Lambda is the cost of developers’ time. The business saves money by paying developers to think about, design and write code full-time, not spend hours trying to wrap our heads around complex, multi-faceted configuration for a fleet of servers we don’t even need half the time. Configuration which inevitably contains faults and goes wrong in production at inconvenient times.

AWS Lambda isn’t best suited to monolithic applications. It works best if you build many small, single-purpose applications. This might mean each endpoint of your traditional monolith has a separate codebase, perhaps sharing a small library of common abstractions. The composable, single-purpose model encourages a scripting-like mindset, so complex frameworks become a hindrance. Even a basic CRUD application quickly decomposes into 4 or 5 tiny scripts. The Python and NodeJS support are ideal for supporting this approach, but it’s easy enough to extract functionality from existing Java monoliths as well.

In summary, AWS Lambda feels like the future of software development in ways that promising new programming languages don’t. It provides a great deal of flexibility within the problem domain while taking away most of the undifferentiated wastes of time in the software development process (internal corporate noise excepted). It encourages keeping delivered software very small and single-purpose, which makes it easy to keep the quality bar fairly high through peer reviews. I’m pretty excited to see how Lambda evolves over the next few years and what the competitors do to up their game.

Building Unity's space shooter tutorial

2015-12-11T00:00:00+00:00

This weekend I found time to continue learning about the Unity game engine and development environment. The second tutorial is a simple vertical scrolling space shooter game. My implementation of the project is on Github.

Space Shooter is a lot more complex than the Rollerball game so it’s a good introduction into game making. Rollerball was a great way to learn how to use the Unity editor and think about game objects. Space Shooter comes with some pre-built assets including models, materials and music useful for wiring something more complete together.

I was initially a little surprised to learn that the essentially 2D game was built with 3D models using a fixed camera. This made it easy rotate and tilt the on-screen objects and make the most of Unity’s lighting system. It would be possible to achieve many similar effects with the editor in 2D mode, with considerably more work animating sprites, but the 3D approach is easy and fairly intuitive.

The game is implemented as a fixed camera, a player-controlled ship object which can only move within a predefined area, and a background. Asteroids then appear and move from the top of the screen to the bottom and serve as the primary hazzard.

The tutorial spends some time explaining how to build the prefabricated game objects from the supplied assets then goes on to cover how to add behaviour one piece at a time. The basic input control is similar to that found in Rollerball. It gets more interesting when you start setting up collision detection, spawning hazzards, destroying objects when they’re off-screen or have expired. Adding the ability to shoot and handling collisions between the shot and the hazzards added another dimension to the game. Adding a scoring mechanism and simple restart function rounded out the basic playable experience.

Once the game is in a functional state, there are a few more extensions to make the player experience a little better. Getting the background texture to scroll creates the illusion of movement. Adding a particle effect over the background enhances the effect. Adding more variety to the hazzards and makes the game feel more natural. After all that, we add an enemy ship type to the hazzards and give it the ability to shoot and perform simple manoeuvers.

At this point we have a complete little game that my non-technical partner found equal parts fun and frustrating for a few minutes. It’s easy to imagine how the game could be expanded further with power-ups, a hitpoints/damage system, extra lives and more levels with different challenges to overcome.

I’m having a lot of fun playing with Unity. I’ll look at the next tutorial when I can.

Starting out with the Unity game engine

2015-11-13T00:00:00+00:00

Update (2012-12-11): I’ve put the project on Github. I don’t know the correct license to use, but it is safe to say it is a derivative work of this project by Unity Technologies.

I’ve had an interest in computer games for many years. Since I’m in the software trade, I’ve often thought about building a simple game as a way to learn how the technology works. I finally decided to have a go a couple of weeks ago and started looking into the technology.

The first engine that came to mind was Unity, a large cross-platform toolkit for building games. I considered a couple of other options but Unity seemed to be the easiest and most beginner-friendly tool to get something really simple off the ground.

I didn’t really know what to expect from Unity or games toolkits/engines in general. Unity, it turns out, provides a tool like an IDE with a level editor, property editors, animations tools, shader tweaking tools, and support for ‘scripts’ written in C# (amongst other things). I have since learned that some other game engines just provide a pile of small utility programs and libraries, and there is just about everything in between available. I’m used to working with IDEs so Unity felt pretty comfortable to get started in.

Over the course of a few evenings I’ve gone through the first tutorial, a simple game with a rolling ball and pick-ups to collect. Despite having no experience with games, I found the tutorials really easy to follow. The explanations helped me to understand the terminology and dig into the basic functionality of the engine.

I was surprised how little code I had to write to glue the game together. I think it was around 10 or 20 lines of C# in total. Loads more code than that ends up in the distribution of course, this is a complete interactive 3D gaming engine after all, but most of the basic building blocks are either visual tools or built in and configured with easy-to-use menus.

So far I’ve been really impressed with the ease-of-use and quality tutorials and documentation. Unity is obviously a very powerful toolkit. I’m looking forward to digging a little deeper with the next tutorial project when I have a bit more time.

A Tour of the Liberate Cable Platform

2015-08-03T00:00:00+00:00

Now that Glastonbury and Wimbledon are over with no major technical failures, it feels like a good time to talk about the interactive TV application I built for the events. For the last few years the BBC has been fielding an application that allows the audience to switch between different hidden video streams on broadcast TV. We always aimed to equally support all the broadcast digital platforms (terrestrial, satellite and cable) but for operational reasons this is the first time we’ve had a complete offering on the cable platform. I’d like to share some of what I learned developing software for cable TV in the UK.

For the last few months I’ve been working on a client application for the UK’s aging cable television platform. For the purpose of this article, I’m talking about the older boxes operated by Virgin Media, not the TiVo branded boxes. These set-top boxes run the Liberate TV Navigator middleware, which is essentially just a really old web browser with a few TV-specific extensions.

The set-top box browser supports a variant of HTML 3.2 and a very early JavaScript interpreter. It is largely based on Netscape Communicator 4.76 and was never updated to more modern web standards. The Liberate middleware (now owned by SeaChange did see at least a few revisions, but these never made it into consumer products in the UK.

There are a number of challenges to working with older browsers like this. Here’s a brief list of some of the things the browser cannot do:

CSS
JavaScript object literal syntax (JSON)
JavaScript function literal syntax (var f = function() { return "foo"; })
DOM API
The switch statement
Document scrolling

The document is a fixed-size rectangle of 720 by 576 pixels. The web page can embed and scale the TV’s video stream, but the complete document must be statically sized to fit within those dimensions. Since there are no advanced layout controls, the design is restricted to hard-coded HTML tables with fixed size rows and columns. The tables can’t be nested more than a few layers deep either, as that leads to serious performance issues on most of the older devices.

JavaScript can be used to get a reference to tables and fish out their elements. There is some limited ability to modify the contents by changing text, foreground and background colours (including transparent to hide them completely), but there is no way to dynamically add elements to the DOM or resize existing table cells, so each document is restricted to showing and hiding elements defined statically, and changing foreground and background colours.

The absence of object and function literals in JavaScript makes writing object-oriented code tricky. The only mechanism that works in the Liberate browser is to add named functions as methods on a constructor function’s prototype. This is more complete than functional programming support however, which has too many missing features to be workable.

var tblMain = document.tblMain;

function Foo() {
	this.name = "foo";
}

function foo_hello() {
	tblMain.rows[0].cells[0].write("Hello, " + this.name);
}
Foo.prototype.hello = foo_hello;

The above example assumes a document containing a table element with the attribute name="tblMain".

The limited syntax support creates issues when assembling a content pipeline for the website as well. Typical minification tools like uglifier and Closure Compiler tend to change object and array declarations to use object literals (= new Array() vs =[]), so they have to be used carefully to avoid breaking the scripts in the browser.

The browser can fetch resources from a broadcast carousel file system (known as in-band data) or via HTTP over the DOCSIS return path (out-of-band). The Cable operator places restrictions on what URLs can be retrieved for safety, so the network can be thought of as an Intranet, though out-of-band data is not immune to man-in-the-middle attacks.

Drawing a user interface

A simple user interface can be drawn with basic HTML 3.2 tables.

<html>
	<head>
		<title>test application</title>
		<script>
		// your scripts here
		</script>
	</head>
	<body onload="onLoad()" onunload="onUnload()" background="tv://">
		<table name="uiRoot"
				width="720" height="576"
				cellspacing="0" cellpadding="0" border="0">
			<!-- header row with borders -->
			<tr>
				<td width="50" height="30"></td>
				<td width="620"></td>
				<td width="50"></td>
			</tr>
			<!-- content row with borders -->
			<tr>
				<td width="50" height="30"></td>
				<!-- content area -->
				<td width="620" bgcolor="white">
					<font size="5" color="black">&nbsp;</font>
				</td>
				<td width="50"></td>
			</tr>
		</table>
	</body>
</html>

In this example, the content area is a table cell containing text. The cell can be manipulated as in the following code.

function hideContent() {
	window.uiRoot.rows[1].cells[1].bgcolor = "transparent";
	window.uiRoot.rows[1].cells[1].write(true, "");
}

function showContent() {
	window.uiRoot.rows[1].cells[1].bgcolor = "white";
	window.uiRoot.rows[1].cells[1].write(true, "Hello world!");
}

Similar methods are used to implement a navigable menu, with a highlight colour for the current row’s background. This would require the hard-coded HTML table to contain enough rows to represent the full menu, and some functions to navigate up and down, and highlight, show or hide each row.

When making multiple UI changes, it can be beneficial to lock the UI layer then allow the changes to draw when finished. This can be done at the level of the table or sub-table element in the HTML layer.

// lock UI
window.uiRoot.drawingDisabled = true;

// make some UI changes

// unlock UI
window.uiRoot.drawingDisabled = false;

Handling user input

Input handling is old-school on Liberate. Input events call the window.onkeyout function and provide the opportunity to decode navigator.input.curKey and run any required behaviours. Keys are represented by raw numbers, so a map of names to key codes can be useful.

function KeyMap() {
	this.HELP = 779;
	this.INFO = 775;
	this.RED = 2048;
	this.GREEN = 2049;
	this.YELLOW = 2050;
	this.BLUE = 2051;
	this.TV = 2053;
	this.PAGEUP = 1284;
	this.PAGEDOWN = 1285;
	this.VOLUP = 1026;
	this.VOLDOWN = 1027;
	this.MUTE = 1036;
	this.OK = 13;
	this.UP = 1280;
	this.DOWN = 1281;
	this.LEFT = 1282;
	this.RIGHT = 1283;
	this.BACK = 1288;
	this.N0 = 48;
	this.N1 = 49;
	this.N2 = 50;
	this.N3 = 51;
	this.N4 = 52;
	this.N5 = 53;
	this.N6 = 54;
	this.N7 = 55;
	this.N8 = 56;
	this.N9 = 57;
}
var K = new KeyMap();

The basic layout of an input handler function is below. This follows the previous example, toggling display of the UI’s ‘hello world’ box.

function doKeyout() {
	var key = navigator.input.curKey;

	// let the user alter volume/mute
	if (key ==K.VOLUP || key == K.VOLDOWN || key == K.MUTE) {
		return true; // invokes STB default behaviour
	}

	// handle application input
	if (key == K.OK) {
		// togle the UI content
		if (window.uiRoot.rows[1].cells[1].bgcolor == "transparent") {
			showContent();
		} else {
			hideContent();
		}
	}
	return false;
}
window.onkeyout = doKeyout;

Fetching resources with NetRequest

While the Liberate TV Navigator browser can’t parse JSON, it can fetch an in-band JavaScript file and eval it. This enables applications to use a configuration file in the programming language or to be fed a declaration of a string to be parsed. A common practice is to broadcast the basic application in-band and have it look up any changeable configuration data from either a separate ‘dynamic’ carousel (updating whenever the content changes) or over the network. Fortunately, both methods of fetching a file are modeled by the NetRequest object in the JavaScript API. In the below example, configNROnStart, configNROnCancel, configNROnError, configNROnLoad, and configNROnDone are functions called as event handlers.

function loadConfig() {
	configNR = new NetRequest();
	configNR.url = "http://www.example.com/config.js";
	configNR.cache = false;
	configNR.onstart = configNROnStart;
	configNR.oncancel = configNROnCancel;
	configNR.onerror = configNROnError;
	configNR.onload = configNROnLoad; // do something with response body
	configNR.ondone = configNROnDone;

	menuDataNR.fetch();
}

Dynamically updating resources at runtime

Another useful feature is timers. These work like in any older browser – calling var t = setTimeout("doSomething()", 5000); will trigger the string doSomething() to be evaluated asynchronously after 5 seconds. Calling clearTimeout(t) cancels it. The only caveat with timers is that the first argument must be a string to eval; it does not accept a function.

var world = new World();
var t = setTimout("world.hello()", 5000);
clearTimeout(t);

Together, NetRequests and Timers enable us to poll for an updating resource on an interval.

var configNROnloadCallback;

function configNROnLoad(event) {
	configNROnloadCallback(event.textData);
}

function loadConfig(callback) {
	configNROnloadCallback = callback;
	configNR = new NetRequest();
	configNR.url = "http://example.com/config.js";
	configNR.cache = false;
	configNR.onstart = configNROnStart;
	configNR.oncancel = configNROnCancel;
	configNR.onerror = configNROnError;
	configNR.onload = configNROnLoad;
	configNR.ondone = configNROnDone;

	menuDataNR.fetch();
}

var configReloadTimer = setTimeout(
	"reloadConfig();", 15000);

function reloadConfig() {
	loadConfig(loadConfig);
}

function loadConfig(config) {
	// TODO: do something with config

	// reset the timer
	configReloadTimer = setTimeout(
		"reloadConfig();", 15000);
}

Conclusion

That concludes our whirlwind tour of the Liberate platform. I have demonstrated the major interactive software building blocks on non-Tivo cable TV boxes in the UK. We’re now doing more complex things with the platform, such as tuning to TV channels and switching out alternative audio tracks, while periodically fetching configuration updates and updating on-screen assets.

I deliberately haven’t discussed much of the back-end infrastructure that bridges the gap from a web server to the broadcast transport stream. There is a large stack of legacy proprietary software behind the Liberate platform, compiling resources into transport streams and generating announcements so the set-top-box knows what content to run and make available to applications. The exact nature of the back-end is a bit more sensitive than the software environment and capabilities described above.

Spring Annoyances and the Trouble with Big Frameworks

2015-07-21T00:00:00+00:00

I’m going to pick on Spring a lot in this post because it has caused me the most frustration in the last few months, but the same arguments can be applied to many large and ungainly frameworks/libraries. This rant was sparked by trying to upgrade a production application from Spring Boot 1.1 to 1.2. It didn’t work; it didn’t tell me anything useful about why it didn’t work; and the upgrade notes said nothing about the parts of the framework which are now completely broken (hint: ErrorController).

There are many frameworks from massive to micro in the land of Java development. Spring is perhaps the largest and most mature of these. It provides libraries for just about everything these days with the promise that the many layers of magic should work together without too much developer pain.

But therein lies the problem. Spring is so large and fragmented that trying to use a few different Spring libraries together often exposes obscure compatibility problems. Sometimes things just work, but often the developer has to micro-manage the exact combination of specific library versions of their various dependencies in order to avoid bugs and rage-inducing compatibility errors.

Spring Boot steps in to solve this problem, repackaging many layers of Spring libraries and their dependencies as a single curated collection that works together, at least in theory. For the most part, Spring boot saves a lot of pain, but it is still subject to the same murderous upgrade path as the rest of Spring. The drill goes as follows:

Change the dependency version in the build file.
Compile and run all the tests.
Swear loudly when it all blows up.
Scour the release notes for the problem. Find nothing useful.
Scream bloody murder at the Spring maintainers for breaking the world without documenting it.

With time and effort, these backwards compatibility issues can always be worked around, but there comes a point when I have to ask if it is worth the trouble. The JDK, such as it is, always maintains backward compatibility and all the parts work together in the same release. Spring’s backward compatibility is shoddy at best and it can take a week to get two simple component libraries to work together.

Despite its merits, Spring Boot only seems to put lipstick on the pig while failing to address the underlying problems with Spring’s many interdependent and frequently unusable libraries. I appreciate that the project tries to make Spring usable again. The frustration comes from it getting close enough to build a product with but not close enough to maintain said product without considerable effort.

In my own travels I’m starting to appreciate more the way JavaScript frameworks are going, building mini libraries for different tasks and letting developers pick and choose between them. Spring was once supposed to offer something similar, but has since grown into a many-headed hydra of cyclically interdependent magic boxes with inconsistent documentation and a ‘throw it away and start again’ upgrade path. Good application design will attempt to separate most of what the app does from the technology used to build it, but there is only so far this can go before the level of abstraction gets silly. Spring Boot can both help and hinder here, depending on how heavily it is relied upon and which layer of APIs the developer uses.

I look forward to the day when upgrading the technologies underlying an application is as simple as changing the dependency version numbers. For anything built with a big, complex framework that day has not yet come.

Using Rubygems to Build RPM Packages

2015-01-19T00:00:00+00:00

Note: The examples in this post were ripped out of a real production project, simplified and renamed where appropriate. I have not tested them in their current state.

I recently did a bit of work to package a simple static website as an RPM to make it easier to manage versions and deploy automatically. The site is built with Jekyll and gem dependency management is handled by bundler. I ran into a few challenges using bundler from the specfile so it seemed like a good idea to document the process here.

The basic project layout looks like this:

root/
    webapp/
        _includes/
            app.js
        _layouts/
            main.html
        _config.yml
        Gemfile
        index.html
    Makefile
    webapp.spec

The build process for the Jekyll site is as follows:

cd webapp
gem install bundler
bundle install
bundle exec jekyll build

I used a Makefile to drive the build. Here’s a simplified version of it sufficient for building the Jekyll site:

NAME = webapp

# important files and directories
WEBAPPDIR = webapp
SITEDIR = $(WEBAPPDIR)/target
SPECFILE = $(NAME).spec

.PHONY: default all site clean

default: site
all: site

site: $(SITEDIR)

clean:
	@echo "==== making clean"
	rm -rf $(WEBAPPDIR)/target
	rm -f $(WEBAPPDIR)/Gemfile.lock
	@echo "==== making clean done"

$(SITEDIR):
	@echo "==== making jekyll site"
	cd $(WEBAPPDIR) && bundle install
	cd $(WEBAPPDIR) && bundle exec jekyll build
	@echo "==== making jekyll site done"

Building the RPM

I used the Makefile to create an RPM build root and copy the appropriate files in. This requires a few more targets to generate the intermediate files and the final package. I also added the VERSION and RELEASE variables to pass into rpmbuild later, making it a little simpler to control versioning from one file in the project root.

NAME = webapp
VERSION = 1.0
RELEASE = 1

# important files and directories
WEBAPPDIR = webapp
SITEDIR = $(WEBAPPDIR)/target
SPECFILE = $(NAME).spec
RPMBUILDDIR = $(CURDIR)/rpmbuild
SPECSDIR = $(RPMBUILDDIR)/SPECS
SOURCESDIR = $(RPMBUILDDIR)/SOURCES
BUILDDIR = $(RPMBUILDDIR)/BUILD
SRPMSDIR = $(RPMBUILDDIR)/SRPMS
RPMSDIR = $(RPMBUILDDIR)/RPMS/noarch
SPEC = $(SPECSDIR)/$(SPECFILE)
TARBALL = $(SOURCESDIR)/$(NAME).tgz
BUILDDIRS= $(SPECSDIR) $(SOURCESDIR) $(BUILDDIR) $(SRPMSDIR) $(RPMSDIR)

RPM_BASENAME = $(NAME)-$(VERSION)-$(RELEASE)
RPM = $(RPMSDIR)/$(RPM_BASENAME).rpm

.PHONY: default all site clean siteclean dist

default: dist
all: dist

site: $(SITEDIR)

clean: siteclean
	@echo "==== making clean"
	rm -rf $(RPMBUILDDIR)
	@echo "==== making clean done"

siteclean:
	@echo "==== making siteclean"
	rm -rf $(WEBAPPDIR)/target
	rm -f $(WEBAPPDIR)/Gemfile.lock
	@echo "==== making siteclean done"

dist: $(RPM)

$(SITEDIR):
	@echo "==== making jekyll site"
	cd $(WEBAPPDIR) && bundle install
	cd $(WEBAPPDIR) && bundle exec jekyll build
	@echo "==== making jekyll site done"

$(RPM): $(TARBALL) $(SPEC) $(BUILDDIRS)
	rpmbuild -v -ba $(SPEC) \
	  --define="_topdir $(RPMBUILDDIR)" \
	  --define="_version $(VERSION)" \
	  --define="_release $(RELEASE)"

$(SPEC): $(SPECSDIR) $(SPECFILE)
	@echo "==== making specfile"
	cp $(SPECFILE) $@
	@echo "==== making specfile done"

$(TARBALL): $(SOURCESDIR)
	@echo "==== making tarball"
	tar -czf $@ $(WEBAPPDIR)
	@echo "==== making tarball done"

$(BUILDDIRS): $(RPMBUILDDIR)
	mkdir -p $@

$(RPMBUILDDIR):
	mkdir -p $@

That should be enough to get rpmbuild to run. The hard part was getting bundler to run in the context of the specfile. Fortunately, the package rubygems-devel provides the necessary macros to support this use case:

%global bundler bundler-1.7.11
%global url_bundler https://rubygems.org/downloads/%{bundler}.gem

Name: webapp
Summary: Example webapp
Version: %{?_version}
Release: %{?_release}
Group: Applications/Internet
Vendor: Ian Gilham
License: CC BY 3.0 (http://creativecommons.org/licenses/by/3.0/)
Buildarch: noarch
BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root
Source0: %{name}.tgz
BuildRequires: ruby
BuildRequires: ruby-devel
BuildRequires: rubygems
BuildRequires: rubygems-devel
BuildRequires: gcc
BuildRequires: gcc-c++
BuildRequires: wget

%description
Example web app

%prep
%setup -n webapp

%build
wget %{url_bundler}
%gem_install -n %{bundler}.gem

export PATH=$PATH:./usr/bin
export GEM_PATH=./usr/lib/ruby/gems/1.8/
export GEM_HOME=$GEM_PATH
bundle install
bundle exec jekyll build

%install
INSTALL_PATH="home/server/webapp"
mkdir -p "%{buildroot}/${INSTALL_PATH}"
cp _site/index.html "%{buildroot}/${INSTALL_PATH}/"

%files
%defattr(644,dcabwww,dcabwww,755)
 "/home/server/webapp/index.html"

%clean
rm -rf %{buildroot}

%post

The BuildRequires dependencies had to be explicit to work in a chroot environment. ruby-devel, gcc and gcc-c++ are required to build Ruby native gems. wget is used to download a gem for local installation during RPM build. rubygems-devel provides a few useful rpmbuild macros needed to glue the build system together.

I knew from the Fedora Project Documentation that the %gem_install macro would allow me to install the prerequisite bundler gem locally, but it took a few attempts to figure out where it was installed and how to set up the local Ruby environment to use it correctly. This is particularly important when considering how to run the build in mock. In the end this was accomplished with the following snippet:

export PATH=$PATH:./usr/bin
export GEM_PATH=./usr/lib/ruby/gems/1.8/
export GEM_HOME=$GEM_PATH

Using mock to isolate rpmbuild from the host environment

At this point I had a workable build system, so the last step was to run it in our continuous integration environment without polluting the build agent. I used mock to isolate the package build-time dependencies from the host machine by running the RPM build inside a chroot jail.

With a working build, it is fairly easy to get mock working by adding a few things to the Makefile:

SRPM = $(SRPMSDIR)/$(RPM_BASENAME).src.rpm

# executables and their options
MOCK = mock
MOCK_OPTS = -q -D "_version $(VERSION)" -D "_release $(RELEASE)"
RPMBUILD_OPTS = --define="_topdir $(RPMBUILDDIR)" \
  --define="_version $(VERSION)" \
  --define="_release $(RELEASE)"

# ... skipped

clean: siteclean
	@echo "==== making clean"
	rm -rf $(RPMBUILDDIR)
	mock -q --clean
	@echo "==== making clean done"

# ... skipped

$(SRPM): $(TARBALL) $(SPEC) $(BUILDDIRS) siteclean
	@echo "==== making source rpm"
	rpmbuild -bs $(SPEC) $(RPMBUILD_OPTS)
	@echo "==== making source rpm done"

$(RPM): $(SRPM)
	@echo "==== making binary rpm"
	$(MOCK) $(MOCK_OPTS) --init
	$(MOCK) $(MOCK_OPTS) --installdeps $<
	$(MOCK) $(MOCK_OPTS) --offline --resultdir $(RPMSDIR) --rebuild $<
	@echo "==== making binary rpm done"

On the continuous integration server, I configured the build to run make RELEASE=$BUILD_NUMBER clean dist to make it easy to see where each version of the package came from. The top-level package version is edited manually in the Makefile and corresponds to our own tagging and versioning conventions.

The final complete Makefile follows:

NAME = webapp
VERSION = 1.0
RELEASE = 1

# important files and directories
WEBAPPDIR = webapp
SITEDIR = $(WEBAPPDIR)/target
SPECFILE = $(NAME).spec
RPMBUILDDIR = $(CURDIR)/rpmbuild
SPECSDIR = $(RPMBUILDDIR)/SPECS
SOURCESDIR = $(RPMBUILDDIR)/SOURCES
BUILDDIR = $(RPMBUILDDIR)/BUILD
SRPMSDIR = $(RPMBUILDDIR)/SRPMS
RPMSDIR = $(RPMBUILDDIR)/RPMS/noarch
SPEC = $(SPECSDIR)/$(SPECFILE)
TARBALL = $(SOURCESDIR)/$(NAME).tgz
BUILDDIRS= $(SPECSDIR) $(SOURCESDIR) $(BUILDDIR) $(SRPMSDIR) $(RPMSDIR)

RPM_BASENAME = $(NAME)-$(VERSION)-$(RELEASE)
SRPM = $(SRPMSDIR)/$(RPM_BASENAME).src.rpm
RPM = $(RPMSDIR)/$(RPM_BASENAME).rpm

# executables and their options
MOCK = mock
MOCK_OPTS = -q -D "_version $(VERSION)" -D "_release $(RELEASE)"
RPMBUILD_OPTS = --define="_topdir $(RPMBUILDDIR)" \
  --define="_version $(VERSION)" \
  --define="_release $(RELEASE)"

.PHONY: default all site clean siteclean dist

default: dist
all: dist

site: $(SITEDIR)

clean: siteclean
	@echo "==== making clean"
	rm -rf $(RPMBUILDDIR)
	mock -q --clean
	@echo "==== making clean done"

siteclean:
	@echo "==== making siteclean"
	rm -rf $(WEBAPPDIR)/target
    rm -f $(WEBAPPDIR)/Gemfile.lock
	@echo "==== making siteclean done"

dist: $(RPM)

$(SITEDIR):
	@echo "==== making jekyll site"
	cd $(WEBAPPDIR) && bundle install
	cd $(WEBAPPDIR) && bundle exec jekyll build
	@echo "==== making jekyll site done"

$(SRPM): $(TARBALL) $(SPEC) $(BUILDDIRS) siteclean
	@echo "==== making source rpm"
	rpmbuild -bs $(SPEC) $(RPMBUILD_OPTS)
	@echo "==== making source rpm done"

$(RPM): $(SRPM)
	@echo "==== making binary rpm"
	$(MOCK) $(MOCK_OPTS) --init
	$(MOCK) $(MOCK_OPTS) --installdeps $<
	$(MOCK) $(MOCK_OPTS) --offline --resultdir $(RPMSDIR) --rebuild $<
	@echo "==== making binary rpm done"

$(SPEC): $(SPECSDIR) $(SPECFILE)
	@echo "==== making specfile"
	cp $(SPECFILE) $@
	@echo "==== making specfile done"

$(TARBALL): $(SOURCESDIR)
	@echo "==== making tarball"
	tar -czf $@ $(WEBAPPDIR)
	@echo "==== making tarball done"

$(BUILDDIRS): $(RPMBUILDDIR)
	mkdir -p $@

$(RPMBUILDDIR):
	mkdir -p $@

Spring Boot Web MVC Testing

2014-12-11T00:00:00+00:00

Note: All the code here is licensed under the Apache License version 2.0.

Update: The information and in this post has been adapted back into the Getting Started Guide. I recommend reading the guide, as it is more complete and easier to follow.

For the last few weeks I’ve been using Spring Boot to build a REST application for an internal service. I started with the Getting Started Guide and built it up from there. Once I got the application in the guide working, I started to look at how to write tests for such an application to drive further development.

I started by forking the guide’s source code sample on GitHub and creating the requisite test source directories following the usual Maven conventions.

For reference, the web endpoint in the sample application can be found in class HelloController:

package hello;

// imports

@RestController
public class HelloController {

    @RequestMapping("/hello")
    public String index() {
        return "Greetings from Spring Boot!";
    }

}

The application just returns a document containing a plain string when clients request the /hello endpoint.

Unit testing

Getting set up for testing requires a dependency to be added to the pom.xml as follows:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-test</artifactId>
    <scope>test</scope>
</dependency>

This gives us JUnit and some other stuff useful for working with Spring applications.

For the test itself, we can hit the web endpoint defined by the HelloController class without spinning up a Servlet container and the rest of the application. This allows us to mock out any dependencies not under test (none for the example application) and verify the behaviour of the controller.

package hello;

// imports

@RunWith(SpringJUnit4ClassRunner.class)
@SpringApplicationConfiguration(classes = Application.class)
@ContextConfiguration(classes = MockServletContext.class)
@WebAppConfiguration
public class HelloControllerTest {
    private MockMvc mvc;

    @Before
    public void setUp() throws Exception {
        mvc = MockMvcBuilders.standaloneSetup(
                new HelloController())
            .build();
    }

    @Test
    public void getHello() throws Exception {
        mvc.perform(MockMvcRequestBuilders
            .get("/hello")
            .accept(MediaType.APPLICATION_JSON))
            .andExpect(status().isOk())
            .andExpect(content().string(
                is("Greetings from Spring Boot!")));
    }
}

The annotations on the test class orchestrate the setup of the application context here. @SpringApplicationConfiguration tells Spring to load the application context from the application’s normal entry point, building any required beans etc. @WebAppConfiguration sets up some machinery required to enable testing of web endpoints. @ContextConfiguration adds a @Configuration class to the context to enable some more beans, in this case to provide a mock Java Servlet container.

The test itself just uses Spring’s testing library to call GET /hello and verifies that the returned value is what was expected. There are more complex ways to assert on the result of hitting an endpoint but this is sufficient for the sample application.

Integration testing

Maven does not enable the integration testing build phase by default so we have to add the failsafe plugin to make it work:

<plugin>
    <artifactId>maven-failsafe-plugin</artifactId>
    <version>2.18</version>
    <executions>
        <execution>
            <goals>
                <goal>integration-test</goal>
                <goal>verify</goal>
            </goals>
        </execution>
    </executions>
</plugin>

With the failsafe plugin in place, maven will run our integration test.

package hello;

// imports

@RunWith(SpringJUnit4ClassRunner.class)
@SpringApplicationConfiguration(classes = Application.class)
@WebAppConfiguration
@IntegrationTest({"server.port=9090"})
public class HelloControllerIT {
    private String base;
    private RestTemplate template;

    @Before
    public void setUp() throws Exception {
        this.base = "http://localhost:9090/hello";
        template = new TestRestTemplate();
    }

    @Test
    public void getHello() throws Exception {
        ResponseEntity<String> response =
            template.getForEntity(base, String.class);
        assertThat(response.getBody(),
            is("Greetings from Spring Boot!"));
    }
}

The integration test is very similar to the unit test but it has a different runtime lifecycle. This time, the application will be run in an embedded Servlet container with all its dependencies before the test code is called. The @IntegrationTest annotation tells Spring to run the full application stack.

Integration testing like this is unnecessary for this simple application but it can be useful when a black-box test of the entire application is required. I have found this method particularly useful for tests that touch a database tier, as it is possible to customise the beans in play for each integration test class, enabling database compatibility testing where more than one vendor is supported, or enabling the tests to run against a local instance of a distributed NoSQL database.

My code can be found on GitHub.

A simple interactive TV slate in MHEG

2014-03-27T00:00:00+00:00

Note: All code examples in this post are licensed under a BSD style license.

There are a few players in interactive television in the UK, each using different technology to build interactive experiences. Freeview and Freesat both use profiles of MHEG-5 as the programming language and runtime environment. MHEG-5 is a fairly unusual language in that it uses an event-based programming model, with all state declared outside the scope of the events.

As far as programming languages go, MHEG-5 makes a developer’s life difficult. There are no symbolic names so we have to refer to everything by an integer object ID. There are no data structures, so arrays have to be implemented manually by iterating on the object IDs and being very careful about where to stop. Links, the functional blocks of imperative code, can fire asynchronously and have no local state so care must be taken to avoid race conditions. There are only a few basic library calls, referred to as “resident programs”, which enable basic string manipulation and limited interaction with the set-top-box. There isn’t even an if statement, so conditional behaviour has to be implemented as a convoluted Test instruction which then fires one of two links for the true and false cases. To program MHEG-5 is to live in interesting times.

Fortunately, there are some useful tools for developers. The main one I use is MHEG Plus. MHEG Plus includes a compiler for a superset of the core language which allows developers to write better code faster. There are a lot of useful tools in MHEG Plus, like symbolic variable names, conditional statements, loops, macros, and simplified syntax. The compiler also has some useful error messages for when mistakes are made that helps to speed up the development-test cycle considerably. The project also includes an emulator that implements an MHEG-5 runtime engine as a Java GUI application. It isn’t perfectly accurate but it’s a lot quicker to get off the ground with an emulator than by having to construct a DSMCC carousel, multiplex it into a DVB transport stream, then modulate it and play it into a set-top-box for testing.

For this post, I have written a simple Hello World application in MHEG Plus. MHEG plus gives us the ability to import headers containing macros and constants, so let’s first define some useful constants:

// src/constants.h.mheg+
{:IntegerConst SCREEN_WIDTH :ConstValue 720}
{:IntegerConst SCREEN_HEIGHT :ConstValue 576}

{:OctetStringConst FONT_ATTR_HEADING :ConstValue "plain.36.42.0"}
{:OctetStringConst FONT_ATTR_BODY :ConstValue "plain.26.32.0"}

{:OctetStringConst COLOUR_BLACK :ConstValue '=00=00=00=00'}
{:OctetStringConst COLOUR_WHITE :ConstValue '=FF=FF=FF=00'}

MHEG-5 applications begin with a file containing the definition of the application object. This file sits in the root of the carousel (the service gateway) and is either called startup or simply a.

// src/a
// application startup file
// application startup file

import(constants.h.mheg+)

{:Application
	( '/a' 0 )
	:Items (
		{:Link lnkMain
			:EventSource 0          // Check this application...
			:EventType IsRunning    // ... for the IsRunning event
			:LinkEffect (
				:TransitionTo(( '~/slate.asn' 0 ) )
			)
		}
	)
	// basic defaults
	:BackgroundColour COLOUR_BLACK
	:TextCHook 10
	:TextColour COLOUR_WHITE
	:Font "rec://font/uk1"
	:FontAttributes FONT_ATTR_BODY
}

The application object contains some default style values and at least a basic link, listening for the application’s IsRunning event. This link will be fired when you tune to a channel with this application mapped in its DVB Service Information, and the set-top-box has finished loading and parsing the code. There is only one statement in the body of this link, an instruction to transition to the slate scene. So let’s define a slate scene:

// src/slate.mheg+
import(constants.h.mheg+)

{:Scene
	( "~/slate.asn" 0 )
	:Items
	(
		{:Rectangle rectBackground
			:OrigBoxSize SCREEN_WIDTH SCREEN_HEIGHT
			:OrigPosition 0 0
			:OrigRefLineColour COLOUR_BLACK
			:OrigRefFillColour COLOUR_BLACK
		}

		{:Text txtHeading
			:OrigContent "Hello World!"
			:OrigBoxSize SCREEN_WIDTH 50
			:OrigPosition 0 [SCREEN_HEIGHT / 3]
			:FontAttributes FONT_ATTR_HEADING
			:TextColour COLOUR_WHITE
			:HJustification centre
		}
		
		{:Text txtExit
			:OrigContent "Press RED to exit"
			:OrigBoxSize SCREEN_WIDTH 30
			:OrigPosition 0 [(SCREEN_HEIGHT / 3) + 60]
			:FontAttributes FONT_ATTR_BODY
			:TextColour COLOUR_WHITE
			:HJustification centre
		}

		// Press RED to quit
		{:Link
			lnkKeyBlue
			:EventSource 0
			:EventType UserInput
			:EventData KeyRed
			:LinkEffect (
				:Quit()
			)
		}
	)
	:InputEventReg 3
	:SceneCS SCREEN_WIDTH SCREEN_HEIGHT
}

There is a bit more going on in the slate scene. First we import the symbolic constants as before. Next we define a scene, give it a name, and populate it with some items.

This scene declares a Rectangle to fill with the background colour and two Text objects containing hard-coded strings. That would be enough for a Hello world example, but for completeness I have also included a link listening for the Red key on the remote control, which simply causes the application to exit. Strictly speaking, a slate should not have exit behaviour, as that would defeat its purpose of hiding any video decoded by the receiver, but it’s a good example of how to enable basic user interaction via the remote control.

The complete code, documentation, build system and license are available on GitHub.

Exploring Software Architecture with OpenCaster and Golang

2013-10-13T00:00:00+00:00

Note: All code examples in this post are licensed under the GPL version 2 (or later). My code is a derivative work of OpenCaster.

Playing out MPEG transport streams is a fairly complex affair. At the office, we have our own in-house solution comprising SI generators, a software multiplexer and DSMCC carousel playout tools etc. covering the four major boradcast television platforms in the UK. In an effort to understand how all the moving parts slot together, I’ve been reading through some of the code and exploring some tools from the open source world as well. One open source tool, OpenCaster, has caught my attention for its simple, old-school software architecture.

OpenCaster is distributed as a collection of small utilities written in C. I haven’t dug into what they all do yet, but the idea is pretty simple. Each tool has one job and does it well. Communication between tools occurs via pipes, typically named FIFOs.

As an example, let’s say you want to broadcast a looped audio/video/data transport stream from a file on disk. We create a named pipe to pump the looped packets into and use a tool named tsloop to read packets from the source file, write them into the pipe, and loop when it runs out of input.

# given sample.ts
mkfifo loop.ts
tsloop sample.ts > loop.ts &

We might want to correct the PCR timestamp in the transport stream, so we use another tool to read from our pipe, correct the PCR, then write out to another pipe.

mkfifo corrected-pcr.ts
tsstamp loop.ts 13271000 > corrected-pcr.ts &

If we assume that we have done enough for the stream to be correctly decoded by a set top box (or VLC Media Player), we can now hook up a third tool to broadcast the PCR-corrected looped transport stream over a supported connection. OpenCaster supplies a few different broadcast tools so I’ll just rip an example from their user manual. This tool sends the stream out over a supported RF card.

tsrfsend myfirstfifo.ts -mt OFDM -mC QAM16 -mG 1/4 -mc 2/3 -mf 578

Now we have seen enough of the OpenCaster toolset to get the gist of how the software architecture works. Each tool filters a stream read from a file (really a named pipe), performs some modification, then writes it out for the next step in the chain to deal with. This is a perfect example of the Unix philosophy of building small, composable tools. The only thing that makes OpenCaster’s architecture unusual is that the tools filter MPEG transport stream packets rather than text strings, but the result is much the same.

Enter Go

For a while now I’ve had a fascination with Google’s Go programming language. It’s like C but easier to write, more type-safe, more modular, and in some ways even simpler, although it does make some of C’s more (potentially) dangerous features much harder to use.

I thought it might be a fun little challenge to reimplement some of OpenCaster in Go and see how the Go version differed from the C original. I have started with tsloop, a relatively simple application for looping a file-based transport stream into a pipe.

OpenCaster’s tsloop program is just a great big main function in C with no headers. It uses a couple of standard and POSIX library functions but is otherwise self-contained. While this may be efficient, I find it annoying to read through a giant wall of code with various duplicated bits that could be factored out easily.

I started by writing a straight port of the C code, changing to Go idioms where it made sense but keeping true to the original for the most part.

package main

import (
	"bytes"
	"encoding/binary"
	"flag"
	"fmt"
	"os"
	"syscall"
)

const (
	tsPacketSize = 188
	maxPid       = 8192
)

// PID table for the continuity counter of the TS packets
type ccTable [maxPid]byte

var (
	pidCcTable      ccTable
	previousCcTable ccTable
	openFile        = 0
)

func main() {
	flag.Parse()
	if flag.NArg() >= 1 {
		openFile = 1
		if tsFile, err := os.Open(flag.Arg(openFile)); err == nil {
			processFile(tsFile)
		} else {
			fmt.Fprintf(os.Stderr,
				"tsloop: unable to open file %s\n",
				flag.Arg(1))
			os.Exit(1)
		}
	} else {
		fmt.Fprintf(os.Stderr,
			"Usage: 'tsloop filename1.ts filename2.ts ... filenameN.ts'\n")
		os.Exit(1)
	}
}

func initTable(table ccTable) {
	for i, _ := range table {
		table[i] = 0x10
	}
}

func processFile(tsFile *os.File) {
	// initialize tables
	initTable(pidCcTable)
	initTable(previousCcTable)
	packet := make([]byte, tsPacketSize)
	// main loop
	for {
		// read packets
		count, err := tsFile.Read(packet)
		// handle short packet
		if count < tsPacketSize {
			openFile %= flag.NArg()
			openFile += 1
			tsFile.Close()
			initTable(previousCcTable)
			tsFile, err = os.Open(flag.Arg(openFile))
			if err != nil {
				fmt.Fprintf(os.Stderr,
					"Can't open file %s\n",
					flag.Arg(openFile))
				os.Exit(1)
			}
			count, err := tsFile.Read(packet)
			if count <= 0 || err != nil {
				os.Exit(1)
			}
		}

		// check packet cc
		buf := bytes.NewBuffer(packet[1:3])
		var pid uint16
		binary.Read(buf, binary.LittleEndian, &pid)
		
		pid = syscall.Ntohs(pid)
		pid = pid & 0x1fff
		if pid < maxPid {
			if pidCcTable[pid] == 0x10 {
				fmt.Fprintf(os.Stderr,
					"new pid entry %d\n", pid)
				// new stream to track cc
				pidCcTable[pid] = packet[3] & 0x0f
				previousCcTable[pid] = packet[3] & 0x0f
			} else {
				adaptationField := (packet[3] & 0x30) >> 4
				if adaptationField == 0x0 || adaptationField == 0x2 {
					// reserved, no increment
					packet[3] = (pidCcTable[pid] | (packet[3] & 0xf0))
				} else if (adaptationField == 0x1) &&
						((packet[3] & 0x0f) == previousCcTable[pid]) {
					// double packet accepted only once
					packet[3] = (pidCcTable[pid] | (packet[3] & 0xf0))
					previousCcTable[pid] = 0x10
				} else if (adaptationField == 0x3) && 
						((packet[3] & 0x0f) == previousCcTable[pid]) {
					// double packet accepted only once
					packet[3] = (pidCcTable[pid] | (packet[3] & 0xf0))
					previousCcTable[pid] = 0x10
				} else {
					// increase the cc and restamp
					previousCcTable[pid] = 
						packet[3] & 0x0f
					pidCcTable[pid] = 
						(pidCcTable[pid] + 1) % 0x10
					packet[3] = 
						(pidCcTable[pid] | (packet[3] & 0xf0))
				}
			}
		}

		// write packet
		os.Stdout.Write(packet)
	}
}

After I had the basic code structure compiling, I sought to simplify by refactoring distinct pieces of behaviour to separate functions, and generally making it easier to read and maintain.

package main

import (
	"bytes"
	"encoding/binary"
	"flag"
	"fmt"
	"io"
	"os"
	"syscall"
)

const (
	tsPacketSize = 188
	maxPid       = 8192
)

type ccTable [maxPid]byte

var (
	pidCcTable      ccTable
	previousCcTable ccTable
	openFile        = 0
	packet          = make([]byte, tsPacketSize)
)

func main() {
	flag.Parse()
	if flag.NArg() >= 1 {
		if tsFile, err := os.Open(flag.Arg(openFile)); err == nil {
			initTable(pidCcTable)
			initTable(previousCcTable)
			for {
				readPacket(tsFile)
				checkPacketCc()
				os.Stdout.Write(packet)
			}
		} else {
			fmt.Fprintf(os.Stderr,
				"tsloop: unable to open file %s\n", flag.Arg(1))
			os.Exit(1)
		}
	} else {
		fmt.Fprintf(os.Stderr,
			"Usage: 'tsloop filename1.ts filename2.ts ... filenameN.ts'\n")
		os.Exit(1)
	}
}

// initialize a cc table
func initTable(table ccTable) {
	for i, _ := range table {
		table[i] = 0x10
	}
}

// read a packet from the file(s) provided
// loops onto the next available file on EOF
func readPacket(tsFile *os.File) {
	count, err := tsFile.Read(packet)
	// handle short packet
	if count < tsPacketSize || err == io.EOF {
		openFile %= flag.NArg()
		openFile += 1
		tsFile.Close()
		initTable(previousCcTable)
		tsFile, err = os.Open(flag.Arg(openFile))
		if err != nil {
			fmt.Fprintf(os.Stderr,
				"Can't open file %s\n", flag.Arg(openFile))
			os.Exit(1)
		}
		count, err := tsFile.Read(packet)
		if count <= 0 || err != nil {
			os.Exit(1)
		}
	}
}

// extract the 16 bit PID from 2 bytes of the packet.
// Performs some binary magic to make it conform to the correct format.
// Example:
//     pid := extractPid(packet[1:3])
func extractPid(s []byte) (uint16) {
	buf := bytes.NewBuffer(s)
	var pid uint16
	binary.Read(buf, binary.LittleEndian, &pid)
	pid = syscall.Ntohs(pid)
	pid = pid & 0x1fff
	return pid
}

func checkPacketCc() {
	pid := extractPid(packet[1:3])
	if pid < maxPid {
		if pidCcTable[pid] == 0x10 {
			fmt.Fprintf(os.Stderr,
				"new pid entry %d\n", pid)
			// new stream to track cc
			pidCcTable[pid] = packet[3] & 0x0f
			previousCcTable[pid] = packet[3] & 0x0f
		} else {
			adaptationField := (packet[3] & 0x30) >> 4
			if adaptationField == 0x0 || adaptationField == 0x2 {
				// reserved, no increment
				packet[3] = (pidCcTable[pid] | (packet[3] & 0xf0))
			} else if (adaptationField == 0x1) && 
					((packet[3] & 0x0f) == previousCcTable[pid]) {
				// double packet accepted only once
				packet[3] = (pidCcTable[pid] | (packet[3] & 0xf0))
				previousCcTable[pid] = 0x10
			} else if (adaptationField == 0x3) && 
					((packet[3] & 0x0f) == previousCcTable[pid]) {
				// double packet accepted only once
				packet[3] = (pidCcTable[pid] | (packet[3] & 0xf0))
				previousCcTable[pid] = 0x10
			} else {
				// increase the cc and restamp
				previousCcTable[pid] = packet[3] & 0x0f
				pidCcTable[pid] = (pidCcTable[pid] + 1) % 0x10
				packet[3] = (pidCcTable[pid] | (packet[3] & 0xf0))
			}
		}
	}
}

After some very basic testing, I started to look at how the architecture might be improved. Specifically, I looked at how I might turn a collection of these filter applications into a reusable library of functionality, where a chain of filters could be built up in one process by reading a config file and assembling all the required pieces for broadcast. Obviously, I don’t have all the pieces yet, but it often pays to think about architecture early.

Go has native support for communicating synchronous processes (CSP) known as goroutines communicating over channels, which can be used much like the OpenCaster tools use FIFOs. An MPEG packet is an array (or slice) of 188 bytes. We can easily send a slice of bytes down a channel, so this communication mechanism should suffice nicely.

In the tsloop application, we can fire up a goroutine to read packets from a file (and loop at the end), sending each packet synchronously down a channel. We can then fire up another goroutine to read from that channel, perform some filtering, and send the resulting packet down another channel. The result of the final channel can be fed into another filter or written to standard output for redirection to a file or named pipe as before. Converting the architecture to send data over channels in this way, we end up with something like this:

package main

// ... imports, declarations etc.

func main() {
	flag.Parse()
	if flag.NArg() >= 1 {
		if tsFile, err := os.Open(flag.Arg(openFile)); err == nil {
			initTable(pidCcTable)
			initTable(previousCcTable)
			input := make(chan []byte)
			output := make(chan []byte)
			quit := make(chan bool)
			nGoroutines := 3
			go readPackets(tsFile, input, quit)
			go checkPacketsCc(input, output, quit)
			go writePackets(output, quit)
			for i := 0; i < nGoroutines; i++ {
				<- quit
			}
		} else {
			fmt.Fprintf(os.Stderr,
				"tsloop: unable to open file %s\n", flag.Arg(1))
			os.Exit(1)
		}
	} else {
		fmt.Fprintf(os.Stderr,
			"Usage: 'tsloop filename1.ts filename2.ts ... filenameN.ts'\n")
		os.Exit(1)
	}
}

// ... functions elided to avoid duplication

// read a packet from the file(s) provided
// loops onto the next available file on EOF
func readPackets(tsFile *os.File, input chan []byte, quit chan bool) {
	packet := make([]byte, tsPacketSize)
	for {
		// content of readPacket(*os.File), above
		input <- packet
	}
	quit <- true
}

func checkPacketsCc(input chan []byte, output chan []byte, quit chan bool) {
	for {
		packet := <-input
		// contents of checkPacketCc(), above
		output <- packet
	}
	quit <- true
}

func writePackets(c chan []byte, quit chan bool) {
	for {
		packet := <-c
		os.Stdout.Write(packet)
	}
	quit <- true
}

In porting an existing application to a new language, we have had the opportunity to practice refactoring and to think more in-depth about software architecture. The Unix philosophy of simple, composable tools is alive and well, but we can make the idea work harder for us by composing the tools at a finer level of granularity and exposing their components via library calls. Getting the API design right requires further refactoring. Here is a starting point:

// src/gomux/util/tsloop.go
package util

import (
	"bytes"
	"encoding/binary"
	"fmt"
	"io"
	"os"
	"syscall"
)

const (
	TsPacketSize = 188
	MaxPid       = 8192
)

// PID table for the continuity counter of the TS packets
type CcTable [MaxPid]byte

var (
	previousCcTable CcTable
)

// read a packet from the file(s) provided
// loops onto the next available file on EOF
func ReadPackets(filenames []string, openFile int, tsFile *os.File,
		input chan []byte, quit chan bool) {
	packet := make([]byte, TsPacketSize)
	for {
		count, err := tsFile.Read(packet)
		// handle short packet
		if count < TsPacketSize || err == io.EOF {
			openFile %= len(filenames)
			openFile += 1
			tsFile.Close()
			InitCcTable(previousCcTable)
			tsFile, err = os.Open(filenames[openFile])
			if err != nil {
				fmt.Fprintf(os.Stderr,
					"Can't open file %s\n", filenames[openFile])
				os.Exit(1)
			}
			count, err := tsFile.Read(packet)
			if count <= 0 || err != nil {
				os.Exit(1)
			}
		}
		input <- packet
	}
	quit <- true
}

// initialize a cc table
func InitCcTable(table CcTable) {
	for i, _ := range table {
		table[i] = 0x10
	}
}

// extract the PID from 2 bytes of the packet.
// Performs some binary magic to make it conform to the correct format.
// Example:
//     pid := extractPid(packet[1:3])
func ExtractPid(s []byte) uint16 {
	buf := bytes.NewBuffer(s)
	var pid uint16
	binary.Read(buf, binary.LittleEndian, &pid)
	pid = syscall.Ntohs(pid)
	pid = pid & 0x1fff
	return pid
}

func CheckPacketsCc(input chan []byte, output chan []byte, quit chan bool) {
	var pidCcTable CcTable
	InitCcTable(pidCcTable)
	InitCcTable(previousCcTable)
	for {
		packet := <-input
		pid := ExtractPid(packet[1:3])
		if pid < MaxPid {
			if pidCcTable[pid] == 0x10 {
				fmt.Fprintf(os.Stderr,
					"new pid entry %d\n", pid)
				// new stream to track cc
				pidCcTable[pid] = packet[3] & 0x0f
				previousCcTable[pid] = packet[3] & 0x0f
			} else {
				adaptationField := (packet[3] & 0x30) >> 4
				if adaptationField == 0x0 || adaptationField == 0x2 {
					// reserved, no increment
					packet[3] = (pidCcTable[pid] | (packet[3] & 0xf0))
				} else if (adaptationField == 0x1) &&
						((packet[3] & 0x0f) == previousCcTable[pid]) {
					// double packet accepted only once
					packet[3] = (pidCcTable[pid] | (packet[3] & 0xf0))
					previousCcTable[pid] = 0x10
				} else if (adaptationField == 0x3) &&
						((packet[3] & 0x0f) == previousCcTable[pid]) {
					// double packet accepted only once
					packet[3] = (pidCcTable[pid] | (packet[3] & 0xf0))
					previousCcTable[pid] = 0x10
				} else {
					// increase the cc and restamp
					previousCcTable[pid] = packet[3] & 0x0f
					pidCcTable[pid] = (pidCcTable[pid] + 1) % 0x10
					packet[3] = (pidCcTable[pid] | (packet[3] & 0xf0))
				}
			}
		}
		output <- packet
	}
	quit <- true
}

This library can be called from the main program below:

// Write packets to standard out
func WritePackets(c chan []byte, quit chan bool) {
	for {
		packet := <-c
		os.Stdout.Write(packet)
	}
	quit <- true
}

// src/tsloop/tsloop.go
package main

import (
	"flag"
	"fmt"
	"gomux/util"
	"os"
)

func main() {
	flag.Parse()
	if flag.NArg() >= 1 {
		if tsFile, err := os.Open(flag.Arg(0)); err == nil {
			input := make(chan []byte)
			output := make(chan []byte)
			quit := make(chan bool)
			nGoroutines := 3
			go util.ReadPackets(flag.Args(), 0, tsFile, input, quit)
			go util.CheckPacketsCc(input, output, quit)
			go util.WritePackets(output, quit)
			for i := 0; i < nGoroutines; i++ {
				<-quit
			}
		} else {
			fmt.Fprintf(os.Stderr,
				"tsloop: unable to open file %s\n", flag.Arg(1))
			os.Exit(1)
		}
	} else {
		fmt.Fprintf(os.Stderr,
			"Usage: 'tsloop filename1.ts filename2.ts ... filenameN.ts'\n")
		os.Exit(1)
	}
}

Replacing code with XML using Spring Integration

2013-07-20T00:00:00+00:00

A few weeks ago I had a go at implementing a familiar kind of application using some new tools. The basic structure should be well known to most developers:

Receive something to process
Do something with it
Send the result somewhere

This basic structure pervades the software industry and naturally, there is a vast swathe of reinvented wheels for managing the control flow, often multiple implementations in the same company.

I was building a new application with this structure in Java to consolidate a process that used two similar applications down to just one, reducing complexity along the way. The existing apps implement file system polling for input and FTP and SOAP calls for output. As I was doing exploratory work, I had the opportunity to look at different ways of doing things.

I started by bringing the existing code into our new application, rewriting the main method and control flow boilerplate code, and reusing the input and output code from previous projects. Once the basic structure was in place, I started looking for opportunities to replace some of the hard-to-test boilerplate code with library usage. Since Spring is used extensively on new projects at the office, we looked into what tools were available in the Spring framework that might meet our needs. That’s where we came across Spring Integration.

Using Spring Integration we were able to reduce the main method to 2 lines and completely remove all the boilerplate code that wired up the file system listener to a processing step and then to the output system. Ultimately, we had just a few lines of XML orchestrating the entire process.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xmlns:context="http://www.springframework.org/schema/context"
     xmlns:int="http://www.springframework.org/schema/integration"
     xmlns:int-file="http://www.springframework.org/schema/integration/file"
     xsi:schemaLocation="http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd
        http://www.springframework.org/schema/context
        http://www.springframework.org/schema/context/spring-context.xsd
        http://www.springframework.org/schema/integration
        http://www.springframework.org/schema/integration/spring-integration.xsd
        http://www.springframework.org/schema/integration/file
        http://www.springframework.org/schema/integration/file/spring-integration-file.xsd">

    <context:property-placeholder location="classpath*:config.properties"/>
    <!--Define channels for message-passing-->
    <int:channel id="input" datatype="java.io.File"/>
    <int:channel id="output" datatype="com.example.myapp.OutputType"/>

    <!--Feed the input from a file system monitor-->
    <int-file:inbound-channel-adapter channel="input"
                                      directory="file:${inputDirectory}"
                                      filename-pattern="*${updExtension}"
                                      auto-create-directory="false"
                                      auto-startup="true">
        <int:poller fixed-delay="${pollingFrequency}"/>
    </int-file:inbound-channel-adapter>

    <!--Transform input format to output format - the business logic-->
    <int:transformer method="transform" input-channel="input" output-channel="output">
        <bean class="com.example.myapp.Input2OutputTypeTransformer"/>
    </int:transformer>

    <!--Publish output over our own interface-->
    <int:outbound-channel-adapter id="out" method="publish">
        <bean class="com.example.myapp.WebServicePublisher"/>
    </int:outbound-channel-adapter>
</beans>

This Spring XML config defines an application which polls the file system for files with a particular extension (FTP’d in by an upstream process), performs some process with those files, then sends the resulting OutputType over a WebServicePublisher interface.

Since all the wiring is in Spring, we can expect it to be well tested. This allows us to focus on testing our business logic and delete our own boilerplate code, reducing the risk of it being rubbish.

To get the application to run, you just need a very simple main method:

public static void main(String[] args) {
    try {
        final ClassPathXmlApplicationContext context = new ClassPathXmlApplicationContext(
            new String[]{"beans.xml"});
        context.registerShutdownHook();
    } catch (BeansException e) {
        LOG.fatal("Beans Exception caught. Exiting...", e);
        System.exit(-1);
    }
    LOG.info("End of main method, other threads continue...");
}

The only gotcha here is that after main() exits, the file system poller will continue to run and keep the application alive. That being said, this is a rather nice way to make the simple things easy and avoid having to maintain a code to poll the file system and pass the objects between components in the chain.

Not shown here are the Input2OutputTypeTransformer which accepts a File and returns an OutputType, and the WebServicePublisher for sending the transformer’s output to a web service. All the glue for running the input file through the transformer and sending the output to where it needs to go are handled by the framework and a few lines of configuraiton.

What I really like about this approach is that not only do I not need to maintain code for passing messages around, I can also change the method of receiving and sending messages without doing any work on the remaining code. It is flexible and allows me to be avoid unnecessary work.

VS2003 compiler bug - 64 bit enum truncation

2012-09-22T00:00:00+00:00

I’ve written before about the compiler bug in managed C++ 7.1. From the information I had available, it looked like the compiler warning was merely a false-positive. Unfortunately, it looks like the problem goes deeper than that.

Consider this test application:

// Managed C++ legacy syntax for .Net 1.0

#include "stdafx.h"

#using <mscorlib.dll>

using namespace System;
using namespace System::Collections;

// commented the FlagsAttribute to simplify printout
//[FlagsAttribute]
__value enum MyEnum : Int64 {
    None = 0,
    Flag0 = 1i64 << 0,
    Flag1 = 1i64 << 1,
    Flag2 = 1i64 << 2,
    Flag3 = 1i64 << 3,
    Flag4 = 1i64 << 4,
    Flag5 = 1i64 << 5,
    Flag6 = 1i64 << 6,
    Flag7 = 1i64 << 7,
    Flag8 = 1i64 << 8,
    Flag9 = 1i64 << 9,
    Flag10 = 1i64 << 10,
    Flag11 = 1i64 << 11,
    Flag12 = 1i64 << 12,
    Flag13 = 1i64 << 13,
    Flag14 = 1i64 << 14,
    Flag15 = 1i64 << 15,
    Flag16 = 1i64 << 16,
    Flag17 = 1i64 << 17,
    Flag18 = 1i64 << 18,
    Flag19 = 1i64 << 19,
    Flag20 = 1i64 << 20,
    Flag21 = 1i64 << 21,
    Flag22 = 1i64 << 22,
    Flag23 = 1i64 << 23,
    Flag24 = 1i64 << 24,
    Flag25 = 1i64 << 25,
    Flag26 = 1i64 << 26,
    Flag27 = 1i64 << 27,
    Flag28 = 1i64 << 28,
    Flag29 = 1i64 << 29,
    Flag30 = 1i64 << 30,
    Flag31 = 1i64 << 31,
    Flag32 = 1i64 << 32,
    Flag33 = 1i64 << 33,
    Flag34 = 1i64 << 34,
    Flag35 = 1i64 << 35,
    Flag36 = 1i64 << 36,
    Flag37 = 1i64 << 37,
    Flag38 = 1i64 << 38,
    Flag39 = 1i64 << 39,
    Flag40 = 1i64 << 40,
    Flag41 = 1i64 << 41,
    Flag42 = 1i64 << 42,
    Flag43 = 1i64 << 43,
    Flag44 = 1i64 << 44,
    Flag45 = 1i64 << 45,
    Flag46 = 1i64 << 46,
    Flag47 = 1i64 << 47,
    Flag48 = 1i64 << 48,
    Flag49 = 1i64 << 49,
    Flag50 = 1i64 << 50,
    Flag51 = 1i64 << 51,
    Flag52 = 1i64 << 52,
    Flag53 = 1i64 << 53,
    Flag54 = 1i64 << 54,
    Flag55 = 1i64 << 55,
    Flag56 = 1i64 << 56,
    Flag57 = 1i64 << 57,
    Flag58 = 1i64 << 58,
    Flag59 = 1i64 << 59,
    Flag60 = 1i64 << 60,
    Flag61 = 1i64 << 61,
    Flag62 = 1i64 << 62,
    Flag63 = 1i64 << 63,
    All = Int64::MaxValue
};

int _tmain()
{
    Console::WriteLine(S"Testing enum range");
    Array* values = Enum::GetValues(__typeof(MyEnum));
    for(Int32 i = 0; i < values->Length; ++i) {
        Object* o = values->GetValue(i);
        Console::WriteLine("{0} = {1}",
            Enum::Format(__typeof(MyEnum), o, "G"),
            Enum::Format(__typeof(MyEnum), o, "D"));
    }
    return 0;
}

If you build that as a Managed C++ project in Visual Studio .Net 2003, you will see the telltale signs of compiler integer truncation:

Testing enum range
Flag33 = 0
Flag33 = 0
...
Flag0 = 1
Flag1 = 2
Flag2 = 4
...
Flag31 = 2147483648
All = 4294967295

This is what the dreaded compiler warning was telling us about in the first place, and sure enough, we got the warning while compiling this code: warning C4309: 'initializing' : truncation of constant value. We seem to have come full circle, verifying that despite being told to use Int64 to back the enum, the compiler decided to use an Int32 anyway and made a mess of things.

It gets stranger than that. I revisited this problem because I received a bug report where the enum has been opened up, and found to contain silly values like 0x102000000L as well as duplicates. The strange ones are None and All:

None = 0x100000000L,
All = 0x1ffffffffL,

A 64-bit integer is clearly being used for the storage backing of the enum, but the values are wrong and only the first 32 bits seem to be read when reading values in code. The only sane conclusion is that using a 64 bit integer to back a managed enum in Managed C++ invoked undefined behaviour in the Visual C++ 7.1 compiler.

This issue is fixed in VC8 using /clr:oldSyntax to compile the same code. I’d love to be able to tell all my clients to upgrade, but in order to keep everyone happy without requiring too much client code to change, I’ll have to settle for using static constants inside a little class:

public class EnumConstantFlags {
public:
    static const Int64 None = 0;
    static const Int64 Flag0 = 1i64 << 0;
    static const Int64 Flag1 = 1i64 << 1;
    // ...
};

These constants can still be used as a bitfield, so the damage is minimal and it will work on all versions of the compiler and CLR.

Nesting parentheses in a Windows batch script

2012-09-17T00:00:00+00:00

My project at work has historically been a small component in a much larger pile of APIs and applications all tied to the same build system, a Windows scheduled task and a pile of incoherent batch scripts. Since then, we have grown up into our own small pile of APIs and connected applications and the time to divorce the parent project has finally come.

In order to make the transition as clean as possible, I’ve been removing dependencies, reorganising the project structure and cleaning up our build scripts. Our Linux builds have always been isolated from the rest of the system so getting that to work in isolation was easy. The Windows builds on the other hand, took a lot more work.

I needed to write a new script to build the Visual Studio solutions on a new continuous integration server. This is largely because the server can’t really handle Visual Studio 2003 without a little help. The script started out simple enough, using the command line arguments to set a few variables, select a version of Visual Studio and a build configuration, then eventually calling devenv with the required parameters. The batch language is primitive and buggy so it took a few goes to get this right. Eventually, I got it working on my machine, so I commited the script and watched it die on the CI server.

The Visual Studio installation wasn’t found in Program Files. Well, this is a 64 bit server, so I queried %PROCESSOR_ARCHITECTURE% and had an alternate path on the 64 bit machine. Searching around the topic, you will sometimes see people checking specifically for %ProgramFiles(x86)% or even if the raw path C:\Program Files (x86) exists. I did something like this:

rem select program files directory
if /I "%PROCESSOR_ARCHITECTURE%" EQU "AMD64" (
  echo building on 64-bit Windows
  set _VS7DIR_="C:\Program Files (x86)\Microsoft Visual Studio .NET 7.1\Common7\IDE"
  set _VS8DIR_="C:\Program Files (x86)\Microsoft Visual Studio 8\Common7\IDE"
) else (
  echo building on 32-bit Windows
  set _VS7DIR_="C:\Program Files\Microsoft Visual Studio .NET 7.1\Common7\IDE"
  set _VS8DIR_="C:\Program Files\Microsoft Visual Studio 8\Common7\IDE"
)

This threw up errors in the script parser, saying things like '\Microsoft' not expected at this time. After a bit of searching around, I found that the parser was interpreting the closing parenthesis in Program Files (x86) as the end of the IF statement. This behaviour is obviously crazy, but nobody is ever going to update such an ancient and terrible scripting language, so we just have to accept that nesting parentheses are never going to work properly in a batch file and work around it.

In the end, the mighty goto solved the problem:

rem select program files directory
if /I "%PROCESSOR_ARCHITECTURE%" EQU "AMD64" goto progfiles_win64
:progfiles_win32
echo building on 32-bit Windows
set _VS7DIR_="C:\Program Files\Microsoft Visual Studio .NET 7.1\Common7\IDE"
set _VS8DIR_="C:\Program Files\Microsoft Visual Studio 8\Common7\IDE"
goto progfiles_done

:progfiles_win64
echo building on 64-bit Windows
set _VS7DIR_="C:\Program Files (x86)\Microsoft Visual Studio .NET 7.1\Common7\IDE"
set _VS8DIR_="C:\Program Files (x86)\Microsoft Visual Studio 8\Common7\IDE"
goto progfiles_done

:progfiles_done

Parsing command line options to correctly handle VC7 and VC8 was much the same, but more complicated:

rem establish paths and defaults
set _VS7DIR_=%_VS7DIR_:"="""%
set _VS8DIR_=%_VS8DIR_:"="""%
set _VSDIR_=%_VS7DIR_%
set _VC7SLN_=MyProject_71.sln
set _VC8SLN_=MyProject_80.sln
set _SLN_=%_VC7SLN_%
set _OPTS_=/Build
set _MODE_=Debug

rem command line option parsing
:begin_opts
if /I "%1" EQU "" goto done_opts
if /I "%1" EQU "clean" (set _OPTS_=/Clean) && goto next_opts
if /I "%1" EQU "build" (set _OPTS_=/Build) && goto next_opts
if /I "%1" EQU "rebuild" (set _OPTS_=/Rebuild) && goto next_opts
if /I "%1" EQU "release" (set _MODE_=Release) && goto next_opts
if /I "%1" EQU "debug" (set _MODE_=Debug) && goto next_opts

rem find the visual studio installation
rem forgive the spaghetti, Windows cannot parse parentheses in IF, FOR, etc.
rem when we might have (x86) in the path
if /I "%1" EQU "VC7" goto vc7_opts
if /I "%1" EQU "VC8" goto vc8_opts

rem catch-all for unknown options
echo unknown option '%1' - clean, build, release or debug expected >&2
goto :failed_opts

:next_opts
shift
goto begin_opts

:vc7_opts
echo VC7 selected
set _SLN_=%_VC7SLN_%
if exist %_VS7DIR_% goto vc7_exist_opts
goto vc7_failed_opts

:vc7_exist_opts
echo using Visual Studio from %_VS7DIR_:"""='%
set _VSDIR_=%_VS7DIR_%
goto next_opts

:vc8_opts
echo VC8 selected
set _SLN_=%_VC8SLN_%
if exist %_VS8DIR_% goto vc8_exist_opts
goto vc8_failed_opts

:vc8_exist_opts
echo using Visual Studio from %_VS8DIR_:"""='%
set _VSDIR_=%_VS8DIR_%
goto next_opts

:done_opts

:vc7_failed_opts
echo Visual Studio 7 not found >&2
goto failed_opts

:vc8_failed_opts
echo Visual Studio 8 not found >&2
goto failed_opts

:failed_opts
echo option parsing failed >&2
goto 2>nul:

I hope this is useful to anyone suffering under the tyranny of the Windows batch scripting language. Where possible it is probably better to use Powershell or a third party scripting language like Python, Ruby, Perl, etc. In my situation I had to hit the lowest common denominator across various editions of Windows.

Haiku - Lovers embrace

2012-09-08T00:00:00+00:00

I came up with this on the beach in Bournemouth. It breaks a few too many rules to be a proper haiku, but I like it.

Waves lapping, the star lit beach; Lovers embrace

Visual Studio 2003 compiler bug in Managed C++ - false positive warning

2012-07-08T00:00:00+00:00

I found a compiler bug at work yesterday while adding a large bitfield enum to the .Net interop layer of a project. The main project is a set of native C++ DLLs built primarily with Visual Studio 2003. We have other targets but as the lowest common denominator, everything has to work on the legacy compiler. The .Net layer is a mixed assembly built with Managed C++ (now known as /clr:oldSyntax).

There is a large enum used as a bitfield in the data API of this project. Unfortunately it has grown to 46 distinct values, so an enum based on a 32-bit integer is no longer appropriate. Adding the new values and changing the underlying type of the enum to Int64 causes the compiler to complain:

[FlagsAttribute]
__value enum eLongChangedFlags : Int64
{
    None = 0,
    Foo = 1 << 0,
    Bar = 1 << 1,
    // ...
    Scary = 1 << 44
    // warning C4309: 'initializing' : truncation of constant value
};

This is counter-intuitive at first, but it is essentially saying that a large number like 1 << 44 (17,592,186,044,416) is out of range of the possible values that the underlying number type can represent. This is incorrect, so we have to assume the compiler still thinks we are dealing with a 32-bit integer or worse.

There wasn’t a lot of information about this error on the internet but there are a few related known bugs in the old compiler. Tellingly, compiling this in Visual Studio 2005 did not give me the same warning. To verify if the enum values have in fact been truncated, we need a test application. I found some code on StackOverflow that suited my purpose and just debugged into it to see what value Scary ultimately represented. It was correct, not truncated and not overflowing into negative figures. The compiler warning is wrong.

To silence the warning I added a couple of pragmas to the original code:

#if _MSC_VER <= 1310
// Disable false positive warning on VS2003 compiler.
// The code has been verified to work, despite the scary warning.
// warning C4309: 'initializing' : truncation of constant value
#pragma warning(disable:4309)
#endif
[FlagsAttribute]
__value enum eLongChangedFlags : Int64
{
    None = 0,
    Foo = 1 << 0,
    Bar = 1 << 1,
    // ...
    Scary = 1 << 44,
};
#if _MSC_VER <= 1310
// re-enable compiler warning in case of further issues elsewhere
#pragma warning(default:4309)
#endif

Using video to improve your own performance

2012-07-08T00:00:00+00:00

I’ve been working on one of my kung fu forms consistently for the last few weeks. At first I was a little sloppy from not having practiced it in a long time. I worked on it a little every day and got used to the movements again, gradually making it flow better and with greater power.

After about a week of this, I received a challenge to make a video of myself performing. I recorded my form in a nearby field and had a quick look. It wasn’t pretty. I rushed sections of the form and lost my balance. My hands were all over the place in some of the techniques and the alignment of my stances made power delivery impossible in a few places.

Disappointed with myself, I had another go. This time I took care not to rush and to concentrate on the movements and applications of the techniques. The resulting video was much better. Everything was much more crisp and tidy. The remaining errors are indicative of flaws in my form rather than just bad habits and sloppy performance. They will take more considered practice to correct, but it is at least a lot easier to work on the problems when you are aware of what they are.

There are lessons to be learned here. Practising the same thing without feedback entrenches bad habits and allows improvement to slow to a halt. In the absense of an instructor to make corrections, video and self criticism can provide enough feedback to keep you honest and maintain your level of performance. It can also help to highlight any other flaws if you know what you are looking for. Video is good. I exspect it can be used in other sports and activities to hasten improvement and prevent bad habits from creeping in.

Fitness - mind and body awesomeness

2012-02-26T00:00:00+00:00

I’ve been in the martial arts for about 8 years now, through college, university, and employment. While I haven’t always been consistent, I did learn early on that the brain works much better when you give the body something difficult to do. This was especially obvious in my university years, when I had time for 4 classes per week, plus the occasional seminar. Despite my high level of physical activity, I found myself smarter, more energized and better rested during that time than ever before.

Now that I work full time and have other responsibilities it is impossible to train quite so much, but I still go to class twice a week and enjoy the many benefits. Since attending any more classes would prevent me from getting enough sleep at the moment, I’ve been finding other ways to get some decent exercise in on the days I don’t have kung fu. I’m working on balancing out my left and right sides to start with and will progress to lifting heavier things as time goes on.

I’m starting pretty slow, with simple bodyweight and kettlebell workouts, usually no more than 10 minutes a time, so I can focus on other things of an evening. I’m also running home after work when I don’t want to wait around for the bus. The recent sub-zero temperature was a great motivatior for not standing around outside for 20 minutes at a time. I’ve been enjoying the run, so I see no reason to stop doing it.

The other side of the coin is to clean up my diet. I’ve long been a lover of cheese and bread and a number of other unnatural foods, and I’m slowly reducing and eliminating the worst offenders. Since I like watching numbers change, I’m tracking a few basic measurements and statistics in a spreadsheet. The timeline chart in Google Spereadsheets is especially good for this.

I still have a long way to go but it’s a fun journey, inspired by websites like Nerd Fitness and authors like Tim Ferris. Good times.

Resolving Linux shared library dependencies

2012-01-22T00:00:00+00:00

Sometimes when you try to launch an application on Linux, typically one you have built yourself, you might get an error like this:

./myapp: error while loading shared libraries: libpq.so.5: cannot open shared object file: No such file or directory

When this happens, it is up to you to figure out what the problem is. Fortunately, you have a tool called ldd to help.

# using which to get the full path to the executable
ldd $(which sh)
    linux-vdso.so.1 => (0x00007fff44fff000)
    libtinfo.so.5 => /lib64/libtinfo.so.5
    ...

If any of the shared (dynamic) libraries are not found, you will have to locate them and add them to your LD_LIBRARY_PATH environment variable. something like the following should do the trick:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/missing/library

Haiku - Autumn on the wharf

2012-01-03T00:00:00+00:00

I wrote this little poem while sitting by the bridge on the South side of Canary Wharf in the Autumn.

The late sun gleams; wings beat on the surface, of glowing water

Fighting the holiday fat

2012-01-03T00:00:00+00:00

My priviledged position as an IT consultant brings with it the possibility that a client may decide to send me home for a couple of weeks to trim their end of year budget. That’s fair enough, but should I really be allowed to bum around at home with such massive quantities of food around? All the ~~good~~ bad things… I think not!

Feeling fat time for exercise- Pretty tree

I’ve got another week off work, so when better to clear out some of the bad things and get some proper exercise? I like to keep things simple, so I’m working on a few core areas only for now. Diet is another thing and pretty easy to keep a handle on when the house isn’t full of cake and other gorgeously evil delicacies.

Core stability/ lower back

My Achilles’ heel if ever there was one and the day job really doesn’t help. I have always had a weak lower back, and it atrophies quickly, so this is the most important thing I need to rebuild. I’m mainly doing simple body-weight exercises like push-ups and sit-ups, and a few swings and deadlifts with my 24kg (53lb) kettlebell.

Arse

Yes, I have a big one, as my girlfriend keeps reminding me. It has reasonably well developed muscles from years of martial arts, but it still carries a little more heft than I would like. The kettlebell swings seem to help tighten up its squishiness and a run through the local forest helps burn off the flab with side benefits to the heart, lungs and pretty much everything else. Death to the massive arse!

Displaying text on networked HP printers

2011-12-05T00:00:00+00:00

I stumbled across a fun little way to put arbirary text on the display screen of HP network printers today. The original code is in C# from the days of .NET 1.0–1.1, so I decided to bring it up to date to the current framework. The program uses HP’s Printer Job Language to communicate with the printer. This will probably work with some other printers as well as most manufacturers have some support for PJL, according to Wikipedia.

Why would I want to mess with the printer’s display? For fun. If a printer that ‘feels fat today’ can put a smile on someone’s face then it has all been worthwhile.

// hptalk.cs
using System;
using System.Text;
using System.Net;
using System.Net.Sockets;

namespace hptalk
{
    /// make HP network printers talk
    /// code taken from odetocode.com and updated to C# 4.0
    /// ref: http://odetocode.com/humor/68.aspx
    class HpTalk
    {
        static int Main(string[] args)
        {
            if (!ParseArgs(args))
            {
                return -1;
            }
            Console.WriteLine("Host: {0}, Message: {1}", args[0], message);
            try
            {
                var ipEndPoint = new IPEndPoint(
                    Dns.GetHostEntry(args[0]).AddressList[0],
                    PJL_PORT);
                Console.WriteLine("Host IP is {0}", ipEndPoint.ToString());
                using (var socket = new Socket(
                    AddressFamily.InterNetwork,
                    SocketType.Stream,
                    ProtocolType.Tcp))
                {
                    socket.Connect(ipEndPoint);
                    var sendString = String.Format(
                        "\x1B%-12345X@PJL RDYMSG DISPLAY = \"{0}\"\r\n\x1B%-12345X\r\n",
                        message);
                    var sendData = Encoding.ASCII.GetBytes(sendString);
                    var result = socket.Send(sendData, sendData.Length, 0);
                    if (result == 0)
                    {
                        Console.Error.WriteLine("hptalk: socjket error");
                    }
                }
            }
            catch (SocketException ex)
            {
                Console.Error.WriteLine(ex);
                return -1;
            }
            return 0;
        }

        protected static bool ParseArgs(string[] args)
        {
            if (args.Length == 2)
            {
                if (args[1].Length > 16)
                {
                    Console.Error.WriteLine(
                        "hptalk: message must be <= 16 characters");
                    return false;
                }
                else
                {
                    message = args[1];
                }
            }
            else if (args.Length == 1)
            {
                message = GetRandomMessage();
            }
            else
            {
                Console.Error.WriteLine("usage: hptalk printername [message]");
                return false;
            }
            return true;
        }


        public static string GetRandomMessage()
        {
            string[] Messages = {
                            "TOUCH ME",
                            "STEP AWAY",
                            "PAT EATS MICE",
                            "FEED ME",
                            "BE GENTLE",
                            "INSERT COIN",
                            "I FEEL FAT" // etc...
     };

            return Messages[random.Next() % Messages.Length];
        }

        private static Random random = new Random();

        protected const int PJL_PORT = 9100;
        protected static string message = GetRandomMessage();
    }
}

The comparison shows off a few of the changes to the language over the years. C# has obviously come a long way. Most of it is irrelevant to the tiny toy program, but even here, the line count has gone down, a using statement takes care of resource cleanup and the var keyword avoids excessive typing.