Alex Muller

Using Lambda to do bits and pieces

2017-09-20T11:30:00+01:00

I hate the idea of vendor lock-in, but I’ve found myself using AWS Lambda more and more often this year. And while neither of these uses are going to change the world, it’s lovely to have an easy, maintenance-free way of running code periodically.

I’m lucky that both of these are just little side projects which I don’t need to care too much about, so I haven’t had to think about setting up deployments for a team or monitoring or any of that supporting stuff. The web interface works well and is quick enough to get set up with.

The Lambda free tier is ludicrously generous for personal use, with 1 million invocations and about 74 hours of compute time per month (at the maximum 1.5GB memory). If you’re cynical then I guess they do that precisely so that people use it for things like this and then bring it to work.

Password protected podcast feeds

Earlier this year I switched to Overcast because my old podcast client stopped being updated. Overcast is great in lots of ways but it doesn’t support XML feeds behind a paywall, which was unfortunate because I’d just started paying for a subscription to LBC’s password-protected full show podcasts.

This is a great use-case for Lambda. I wanted a bit of code which would run periodically, access the XML feed with my password and put it somewhere secret that didn’t have a password.

So I wrote a script that does exactly that:

Use my username and password to get a session cookie from the LBC podcast service
Download the XML feed I’m interested in
Put it in an S3 bucket with a filename that only me and my podcast client know

CloudWatch Events schedules this script and runs it every 15 minutes for me and it’s worked flawlessly for the last 5 months.

The code’s available on GitHub.

Renaming Strava activities

This one’s a little more nerdy and a little less necessary. I cycle to work and use Strava to see how far I’ve ridden. But I like things being named properly and I’d rather not have a load of “Morning Ride” and “Evening Ride” every day on my profile.

I’d been renaming those things in the Strava interface and it took me too long to realise that it was a huge waste of my time and I should get a computer to do it for me. So I wrote a little script:

Get the latest activity from my Strava profile
Check if it has a boring name like “Morning Ride”
Use the coordinates of the start and end to figure out if it’s a commute
Rename it

As before CloudWatch Events makes this super easy to schedule with an expression like rate(1 hour).

Again, the code is on GitHub if you want to see it. As this was the second time I’ve used Lambda I had a quick look around for something that might make Lambda easier to develop against and settled on python-lambda for my simple needs.

Pensions: the basics

2015-02-16T20:30:00+00:00

Over the last couple of weeks our chatroom at work has revealed that nobody completely understands pensions. I find financial planning quite interesting, so this blog post is my attempt to gather what I think I know about the Civil Service pension scheme nuvos and hopefully be corrected by people who know more than me.

It should be pretty obvious from the fact this is written on my personal blog that it is absolutely not affiliated with the Civil Service and nobody there has reviewed it.

Defined contribution vs defined benefit

The first thing I had to get my head around was that there are at least two different types of pension scheme around.

The one I’m most used to is defined contribution. This is pretty straightforward because it’s how you expect these kind of things to work intuitively:

Each month you and your employer both purchase “shares” on your behalf
These shares are managed by a pension provider
Over time the value of the shares goes up and down
As you get closer to retirement, you’d generally expect your pension provider to balance your shares so that your pension becomes less volatile (ie more cash than stock), so that your entire pension doesn’t disappear just before you retire

In the double-entry bookkeeping software I use, defined contribution is simple:

Account: Income

                      Account                      Credit    Debit
August salary
    Gross salary      Income                       1000
    Net salary        Assets > Current account               600
    Tax               Expenses > Tax                         200
    Pension           Assets > Pension                       200

This means that my pension account has a balance which I can monitor over time.

Defined benefit schemes, on the other hand, are a bit weird.

As the name implies, they define the benefit you’ll receive when you retire and then the pension provider adjusts your contributions so that they have enough money in the bank.

Defined benefit plans confuse me because there doesn’t seem to be a way to reconcile the amount you put in with the amount you get out. It all looks a bit arbitrary.

In double-entry bookkeeping, I’ve set it up like this because I don’t know what else to do:

Account: Income

                      Account                      Credit    Debit
August salary
    Gross salary      Income                       1000
    Net salary        Assets > Current account               600
    Tax               Expenses > Tax                         200
    Pension           Expenses > Pension                     200

Instead of storing my pension contribution as an asset and being able to track its change in value, I write it off as an expense.

nuvos

nuvos is the default Civil Service pension at the moment. It’s a defined benefit scheme that comes with a 36 page instruction book which is an impenetrable FAQ.

The interesting parts can be distilled into a few bullet points:

You pay a contribution rate that depends on your “pensionable earnings”, which is pretty much the same thing as your salary
Each year that you work, your “defined benefit” increases by 2.3% of your pensionable earnings from that year
The benefit from previous years that you’ve built up increases in line with the cost of living
At pension age (65ish) you end up with a number which is the amount of money you’ll receive every year until you die (see, pensions are fun)

This is changing slightly, but not massively, in April with the introduction of the new pension scheme “alpha”.

GitHub Pages that don't depend on JavaScript

2014-11-29T13:30:00+00:00

There’s a blog post that explains how to use GitHub Pages metadata to display a list of repositories. It’s great because it’s one of the only uses of site.github.public_repositories that I can find online. But part way through it says:

I needed to get the metadata into JavaScript

Both the Shopify and Yahoo! GitHub Pages instances use that technique, and as a result if you load those pages without JavaScript you’ll get something that looks a bit broken.

We’re using the same metadata for GOV.UK’s GitHub Pages. I’d bet our repository hasn’t had as much work go into it as those other two, but our text is right in the HTML, so the page works if the JavaScript fails to execute.

At the Government Digital Service we try and build things that are as resilient to failure as possible. alphagov.github.io is quite possibly the least important thing we do, but it feels nice to create web things in the best way that we can all the time.

GDS has jobs currently available for developers and junior developers, so if this sounds like the way you work make sure you apply.

Performance Platform dashboard admin

2014-06-06T19:30:00+01:00

After releasing about a hundred dashboards in the last couple of weeks, members of the product team I’m part of at the Government Digital Service each spent a week prototyping standalone projects that we thought would be of use to the Performance Platform. Tom and I created a basic interface for administering dashboards.

We only had a week so we didn’t want to spend time thinking about where it would be hosted or how authentication could work, so we created a Node.js app that non-technical members of our team can download and run locally.

We ended up with an application that uses a GitHub repository to store our configuration and does file modification, committing, pushing, pulling and deploying through a browser. It’s hacky but it does exactly what we need it to do. SSH handles authentication in the background and GitHub provides us with a lovely interface for a history of who, what and when things changed.

Hopefully this marks the end of Matt having to write out text changes into Pivotal Tracker stories just to wait for a developer to copy and paste them into an editor and push a commit.

I’m really happy with what we achieved in 5 days. The Performance Platform dashboard admin app is on GitHub, as is the config repository. I’m looking forward to approaching this project slowly and carefully now that we’ve got a good idea of the direction it should go in.

Starting at the Government Digital Service

2013-10-27T12:30:00+00:00

Aside from the fact that the day-to-day work is interesting and challenging and the people are fantastic, there are a few things that particularly stick out to me as brilliant about GDS.

Second line support

This seems quite polarising for developers, but I love it. Each week a different developer pairs with somebody from the operations team to spend a week improving infrastructure and responding to any production alerts and issues. I finished my first rotation on second line as part of the operations team about a month ago.

It’s such a great learning experience. You learn a load about the GOV.UK infrastructure, as well as an awful lot about hosting, monitoring and deployment. It’s also a great way to work with people outside your normal team on a short-term basis.

It’s all out in the open

Almost every line of code that technically can be is hosted at www.github.com/alphagov (obviously the passwords, keys and other fun stuff is elsewhere). British taxes pay for the lights to stay on, so it’s only right that the public can see what GDS is up to as much as possible. As with everything on this site, that’s my opinion and it may or may not be shared by GDS. You’d have to ask someone who’s allowed to speak on behalf of the Cabinet Office.

In my team (I work on the Performance Platform), we’re making data about transactional services available for service managers (the people who run, for example, the passport application process as part of HM Passport Office). But it’s a nice side effect that the data is there, online, for everyone to see. The same thought process as for the code applies: you’re paying for those transactions to take place. You should be able to see how they’re performing.

Informally we work off the idea that for every page we build, you should be able to pull the data out yourself. For the Licensing “Form submissions” graph, the JSON’s available. It’s not documented (yet), but you can use developer tools in the browser to view the AJAX requests.

Once more: all my opinion, possibly (but definitely not necessarily) shared by GDS. I might turn up to work on Monday and find out it’s all changing.

Fixing broken windows

James mentioned the broken windows theory after I made a change to one of our internal apps recently. When I wrote about a company’s tools in April, that theory was exactly what I was getting at - I just didn’t know it had a name. If you make it easy to fix problems, as GDS does by giving all staff the ability to push changes to almost every repository, the state of the code and the community constantly improves. I love tinkering with little bits and pieces that keep things running along smoothly, and internal apps are a great place to do that.

The Service Manual is another great example. Because it’s so easy to fix typos and reword badly written sentences, I’ve committed to it far more than I would have expected. And that’s mostly because I can do it in a couple of minutes when I’m bored. And even better than that, making the repository public means that a load of people from outside government have submitted fixes too.

GDS isn’t perfect by any means, but the good outweighs the not-so-good by an absolutely huge margin.

The tenth design principle says it all:

Make things open: it makes things better

Internal Tools and Infrastructure

2013-04-30T18:30:00+01:00

At everywhere I’ve worked so far I’ve been disappointed by the tools available to employees to help them do their jobs better; put simply, I think every company I’ve been a part of could have benefitted immensely from a simple MediaWiki install.

Obviously I’m aware that the lack of such a setup stems from a people problem rather than a technical one, but it needn’t be a people problem. If there’s an argument that can be made against the following setup, I would love to hear it:

Install a wiki with text-based markup and an obvious, attributable change log. Everyone who gets an email address with your company gets an account.
The front page can only be edited by people who manage internal company communications. It should link to other important parts of the wiki.
Official policies and company documents can only be edited by a very limited number of people.
Everything else is a free-for-all, open to edit.

To pick on MediaWiki specifically: it is so good at handling all this kind of stuff, with a visible history with each change attributed to users. I’ve never come across an employee both malicious and stupid enough to bother messing with this theoretical open wiki and doubt I ever will. And if such a person did exist, they should be handled exactly as though they’d scribbled on the walls.

This is so appealing to me because I’m the kind of person who loves to read things. I really enjoy poking through obscure documentation, and if I have the ability to fix a typo easily, I’ll do it. On Wikipedia you might describe that kind of person as a WikiGnome. People should be encouraged to edit and to make a mess, because for every well meaning employee who adds something useful with bad syntax, there’s somebody like me who will make it neat. Win-win.

This isn’t a technical problem. The software exists, and it’s really good. And it shouldn’t be a people problem either.

At this point, I’m probably just rehashing things that people smarter than me have said before. If I was creating a company I intended to grow from scratch, here’s how I’d start:

A MediaWiki install as detailed above.
Internal services for employees have sane domain names with one internal TLD: http://calendar.muller/, http://wiki.muller/, http://mail.muller/, etc
Have an external domain that points to the same place, but only for services you want available externally: http://wiki.muller-network.com/.
No, your department doesn’t get to buy a domain name, hire a designer/developer and host a WordPress site just to show the rest of the company how great you are. Use the wiki.
Have well publicised places for employees to help you run your company better (Running a news site? Can Finance employees easily report typos? Why not?!).

A more concrete example

When I’ve got a few minutes in between work here I might pop on to The Times site, see what’s going on and find an article that takes my fancy to skim.

I did that the other day and found a pretty bad typo in an article, a remnant of an editor suggesting a text change that somehow made it through the CMS (part of the article text read “<NO1>10<NO>ten-month [sic]”). What were my options at that point?

Email it to… someone. The editor? The digital editor? The article author? Seems like a little bit of overkill to compose an email for a typo and send it to people I don’t know.
Leave a comment on the article, or do something else that any “normal” user could.

Why is there no third option? Why is there no issue tracker for employees to report problems, especially public-facing problems that make the company look less impressive at its core job?

I’m not picking on The Times or The Sun here, just using them as an example I’m familiar with: I’ve never seen this done well anywhere, but would love to hear about examples.

And if there are any good resources about internal tools and infrastructure (perhaps even long-form articles or books?), I’d love to hear about them.

On The Lab's blog: Designing for Receipt Printers

2012-12-03T19:20:00+00:00

I wrote a bit for the office blog about our work writing HTML and CSS that ends up printed on a bit of paper a couple of inches wide:

For web developers, there’s no real technical challenge in the front-end side of creating a publication; it’s all HTML and CSS with nothing to think about other than how WebKit will render your page in black and white. But this lack of difficulty is more than made up for with the tough constraints imposed by the media and the size it’ll be printed at. The printable area is less than 400 pixels wide and just 800 tall to avoid the unpleasant surprise of waking up to a publication five metres long trailing across the floor. Berg’s guidelines recommend that text should be sized no smaller than 20px (and they’re dead on), but with letters so large and paper so small there isn’t room to say much.

Designing for Receipt Printers (December 2012, News International R&D)

I love the way that our blog looks and works. It’s an Octopress-based static site. Bel created designs that look awesome, and David and I set the thing up.

Pirate Party UK

2012-11-01T14:50:00+00:00

(I originally wrote this article in about May of this year, when the news stories mentioned were current.)

2012 is the year I joined my first political party. After a bit of back-and-forth with my elected Member of Parliament (Conservative Mary Macleod) in which we discussed the CCDP, the High Court ruling against The Pirate Bay and Claire Perry’s dangerous crusade to censor the web, I wrote this to her:

The Internet is being legislated by people who do not understand it. I have not seen a single elected figure take any of the issues of digital civil liberties seriously, and as a result I am incredibly disheartened with the state of UK politics. As a twenty-one year old I find that quite sad and I hope you do too.

I remember being on holiday in April two years ago, in a game camp in the middle of South Africa, using a phone and a local SIM card to read about the Digital Economy Bill vote on Twitter. I can still remember feeling entirely powerless, and the same’s happening now with the three topics I mentioned to my MP – different Government elected, same mistakes being made.

I wrote her that message two days before the local elections on the 4th of May and was hoping for a little more of a reply than I actually ended up receiving; her assistant simply noted that my concerns would be forwarded to Ed Vaizey, who holds the title of “Minister for Culture, Communications and Creative Industries”. While waiting for London Elects to announce the results of the election for Mayor of London I (re)stumbled across the Pirate Party’s site (led there, I think, by their Pirate Bay mirror). For whatever reason I had never before realised that their thinking was so aligned with mine.

I wish they weren’t called the Pirate Party. It’s not going to do them a single damn favour as awareness of them becomes more widespread. But having just one more MP who can talk sense about all things digital in the House of Commons (alongside Tom Watson) would be hugely beneficial for the UK.

Politics has made me feel nothing but frustration and disappointment for the last couple of years. I’ve finally realised that (much like every aspect of life) it’ll be far more productive to stop thinking negatively and put that energy into contributing to a political party that has sensible policies relating to issues I feel are the most important. Call me naïve, but I hope that the next general election will see the vote spread far more evenly across large and small parties, and the Pirate Party should be a piece of that.

That paragraph contains a phrase it feels like it’s taken me too long to realise. Sensible policies about issues I think are important. Bad legislation will continue to magically appear unless people who understand the issues involved make sure they’re vocal about the politicians who are doing it right.

I’ve mentioned to a few friends recently that I’m constantly surprised by how few young people are interested in the technical side of creating things for the web; a tiny minority of the population want to learn what HTML is, let alone anything more involved than that. Given how widespread the web is today and that it’s only going to grow, I’d expect far more enthusiasm for web development. That same idea applies equally to digital legislation, but perhaps attitudes will change as those who have grown up with access to a computer start being elected.

If you’re interested in The Pirate Party, have a flick through their 2012 manifesto which has recently been approved by members.

Twitter, Tweet Nest and the Data Protection Act

2012-08-17T10:50:00+01:00

I’ve had a copy of Tweet Nest running since the end of 2010, pulling in a copy of my latest tweets twice a day.

Unfortunately, as the nerds out there might be aware, Twitter allow access to just your most recent 3200 tweets online. The only way to get a copy of your older tweets is to send them a request under the Data Protection Act 1998. Folks elsewhere in the EU should have similar legislation to help them, but I’m not sure about other countries. To their credit, Twitter complied with the request fully (as far as I can tell) and without making it too painful for me (depending on how painful you consider having to send a fax).

Here’s the first twenty lines of the alexmuller-tweets.txt file they sent:

While it was good of them to supply so much information, there was really only one thing I was interested in: the list of status_ids of all my tweets. Twitter provide the tweet text as part of this request, but strip other interesting metadata such as location and source.

With that, I used a rubbish bit of Python (with tweepy) to pull every bit of JSON I could out of the Twitter API and save it to a file:

import tweepy
import json
import time
import datetime

# I found this somewhere online. Add a .json attribute to just get the raw JSON.
@classmethod
def parse(cls, api, raw):
  status = cls.first_parse(api, raw)
  setattr(status, 'json', json.dumps(raw))
  return status

tweepy.models.Status.first_parse = tweepy.models.Status.parse
tweepy.models.Status.parse = parse

username = "YOUR_USERNAME"
password = "YOUR_PASSWORD"
auth = tweepy.auth.BasicAuthHandler(username, password)
api = tweepy.API(auth)

myfile = open('list_of_ids_to_grab.txt', 'r') # this is a list of tweet IDs
jsoncontents = open('tweets.json', 'a') # blank file

i = 1

for line in myfile.readlines():
  print i
  print datetime.datetime.now()
  line = line.strip()
  print line
  status = tweepy.api.get_status(line)
  print status.json
  time.sleep(25) # 150 an hour
  donetweets = open('done_tweets.txt', 'a') # blank file
  donetweets.write(line + "\n")
  donetweets.close()
  jsoncontents.write(status.json + "\n")
  i += 1

WARNING: this code is so wonderfully breakable that it will probably set your machine on fire. You’ve been warned.

Basic auth calls are limited to 150 an hour, so I left this running overnight to complete. With the result, you can easily turn it into an array (wrap it in [] and add commas to each line, also known as the poor man’s way to code) and then use Bryan Veloso’s beautiful script to import it into Tweet Nest. It’s worth importing your older tweets first before setting up a repetitive job to pull in new ones, or else you’ll have to do some MySQL funkery that I can explain in more detail if you need (yell on Twitter).

This script highlights a problem that Twitter mentioned again last night: basic auth will not work for much longer. This is a huge issue for Tweet Nest too, as it doesn’t use OAuth. I’m going to (attempt to) add OAuth support to it in the near future.

While you’re here, let’s have a quick look at what else they returned from the DPA request. Apart from the normal stuff you shouldn’t be shocked to know Twitter have access to (direct messages, favourites, followers and following), these two things were surprising, though not massively so, to me:

A list of all the phone numbers and email addresses that were stored on my phone when I first ran the official Twitter iPhone app (1222 in total).
A list of every IP address that has accessed my account for the last four months (it’s fairly standard practice to store this).

For what it’s worth, I agree entirely with David Singleton, who tweeted:

In case it wasn’t clear, I’m pretty worried about the twitter changes too. I suspect many of them won’t be enforced, but still, dangerous.

No matter how much you trust Twitter, it would be prudent to store a copy of your own data.

Mid-2012 tech priorities

2012-08-05T08:05:00+01:00

A quick update on how I’m feeling about technology and computer stuff, and where my priorities are at the moment.

Security and ownership of data

Pretty much every password I use now is unique, and the ones that aren’t will be soon. I only know a couple of them from memory, and most of the important ones are over twenty characters. 1Password is still a dream come true. Password leak after leak has meant I don’t trust anyone online any more (and probably never should have).

I’ve changed the MX records (email provider) for this domain away from Google Apps and over to FastMail.FM. I’ll register new accounts with this domain rather than my @gmail.com address, and will move important services over gradually. I’m no longer comfortable with Google providing such an essential service for free. I pay O2 in excess of £30 monthly for a phone; why on earth was I holding out over £2.50 a month for email, which is arguably way more important?

Dropbox is no longer my go-to place to shove everything into. Instead, I’ll be even more anal about backup, soon setting up Crashplan to backup in an encrypted fashion to a drive at (my parents') home. This’ll be more useful when it’s actually an off-site backup.

For services that store data on my behalf, I’ve become a bit crazy about maintaining my own copy of that data. This includes keeping a copy of my Oyster card history using Mollusc, and sending Twitter a request under the Data Protection Act so that I can get a copy of my first three thousand tweets (they’ll be imported into Tweetnest soon).

Portability of data and striving to be lightweight

Much to Adam’s amusement, I fear, I’ve become a bit militant about the idea that web servers should serve HTML files whenever possible. To that end, this blog is now using Jekyll to generate HTML files that are transferred from my computer using rsync. I was rather shocked to see just how slow WordPress is:

Nothing’s changed with regards to what I wrote about links; I’ve tried to keep everything intact, and I think I’ve done pretty well. Old posts look old. I still like that a lot.

Reading

The iPad’s no good for long reading for me, even with the Instapaper or Kindle apps. Instead, I bought a (non-touch) Kindle from Amazon for my new commute and set up Instapaper articles to be delivered wirelessly each week. It’s incredible. The weight and the screen together make it a great reading experience on the Tube.

Money and graphs

I finally got serious about boring grown-up stuff like dealing with money. GnuCash is an excellent app, and I wish I’d been using it for years. I spend way too much on food. This is yet another example of how right Etsy are when they get shouty about measuring everything. Graphing expenses isn’t the most exciting thing in the world, but is so, so useful. For me personally, I find it so much easier to improve something if I can see it changing over time. A single GnuCash database that I can update both at my laptop and on an iPhone doesn’t seem possible today, but that’s the ideal end goal.

Later this year, I want to start graphing some of the data that’s generated by the house I live in. Andy Stanford-Clark gave a talk in York about some of his home monitoring kit and it’s got me a bit excited (his TEDx talk is very similar). There are hugely practical uses, like figuring out where in your house the majority of your electricity bill comes from so that you can do something about it.

And that’s where I am right now

A bit of a mishmash of stuff. This is just what’s important to me this month, a bit of a State of the Union brain fart.