Casey Banner

Effectively enabling gzip in nginx

Wed, 12 Oct 2011 21:12:40 +0000

Google released their [Page Speed Online](https://developers.google.com/pagespeed/) tool today. Originally available as a [Chrome extension](http://code.google.com/speed/page-speed/), Pagespeed lets you analyze the performance of your page and how it adheres to web best practices. It is a pretty neat tool which developers should be running on their sites before they go live to make sure everything is in order. One of the thing I was missing was having gzip compression enabled on all my resources. By default, nginx has gzip compression enabled, but only for plain text MIME types. With all the HTML5 sites sporting fancy web fonts these days, compression becomes especially important. The `gzip_types` directive inside the `html {}` configutation block lets you specify what MIME types you want to gzip. Check out the `mime.types` file in your nginx configuration directory to see the mappings between MIME types and file extensions. Here is how you get gzip compression going for more of the commonly used MIME types for a typical HTML5 site: gzip on; gzip_types text/plain text/css application/x-javascript application/xml application/x-font-woff image/svg+xml; gzip_vary on; Javascript, CSS, and web fonts should all be compressed with gzip. Woohoo!

pivotal2trac - Pivotal Tracker to Trac importer

Fri, 16 Sep 2011 19:08:30 +0000

I recently need to import a bunch of stories from [Pivotal Tracker](https://www.pivotaltracker.com/) into [Trac](http://trac.edgewall.org/). I had quite a few and I didn't want to enter them all by hand, so this is what I came up with: [pivotal2trac](https://github.com/kcbanner/pivotal2trac). Hopefully this is useful to someone else!

botLog - microblogging with an xmpp frontend

Mon, 11 Jul 2011 18:51:53 +0000

I've just published [botlog](https://github.com/kcbanner/botlog), the microblogging framework that powers my [awesomeshit](http://awesomeshit.caseybanner.ca/) page. Posts are stored in MongoDb and are managed by sending messages to an XMPP chat bot which generates the html and RSS feed. botLog has evolved substantially since I started that page last November, and I'm pretty happy with where it has ended up. Initially, I had a CSV file containing all the posts, and a python script that would generate the page and RSS feed. To add a post, I would ssh to my server, add the post to to the CSV file, and re-run the script. This worked fine, as long as I was on a computer I could ssh from; editing files in emacs from an Android phone is not as fun as it sounds. When I had initally written the generation script, node.js was not what it was today and the phrase 'server-side javascript' would have made me chuckle. However, we have come far! I write almost everything I can in Javascript these days, and I realized that what I wanted to do with botLog would be the perfect application. The plan was to write an XMPP bot which could be used to manage posts. I've always liked the idea of chat bots as interfaces. They have been used in IRC since the dawn of time to provide users with tools and information. If you've connected to freenode, you've most likely talked to nickserv to register your nick. The #archlinux channel has a bot which provides access to package information. The idea is that as the user you don't need to switch contexts to perform a task; I don't need to go to my browser and google for a link to a package, I can just message the bot. Writing an XMPP bot in node.js is fairly simple, thanks to [node-xmpp](https://github.com/astro/node-xmpp). I spent an afternoon writing up a bot that responded to a few simple commands, `list`, `post`, and `remove`. Then, all that was needed was to import all my old posts from the CSV into Mongo. When a post is added, the bot uses [jade](http://jade-lang.com/) to render the single html page and the XML for the RSS feed. Thats the story of botLog! I'm hoping to add some more features in the future, such as template management and perhaps pagination, but so far I'm pretty happy with it.

UW Schedule is live!

Sat, 23 Apr 2011 23:56:18 +0000

For the last while I've been working on a social class scheduling site for Waterloo students. Basically, I was getting frustrated with the long (5+ clicks) process of logging in to Quest to see my weekly schedule, so I decided to do something about it.

I was also motivated by my desire to learn Node.js and the goodness of server-side Javascript. The server is written using Node.js. The libraries I'm using for all the HTTP goodness are Express (routing, middleware), Connect (middleware), Jade (template rendering), and Less (css rendering).

In the process of writing UW Schedule, I pulled out several components that I had to build into libraries of their own: node-cas (Node.js Central Authentication System client library), facebook-test-users (command line utility for managing Facebook test users), and connect-mongo (MongoDB session store for Connect). Node seems to have a great community and I'm happy to contribute back to it.

More features are in the works, but currently you can sign in (with WatIAM credentials or a normal account), add courses to your schedule, and share your schedule with friends via Facebook.

If your a Waterloo student, check it out! Feedback is welcome and appreciated.

-Casey

Safe Javascript/Asset caching with Node.js and nginx

Sun, 17 Apr 2011 21:24:17 +0000

Most web applications will have a large chunk of Javascript that needs to be available every pageload, one of my applications clocks in at 134k (minified, not gzipped).

We can easily tell the browser to cache this by setting Expires headers, however, this leads to problems when you roll out a new version: the user has to refresh the page (not simply load it) to get the new version. If you have some inline Javascript, and it gets out of sync with the user's cached version of your static Javascript, then you have problems.

Fortunetly this problem has been solved elsewhere, and all we have to do is adapt it to Node. Rails does this by appending `?file_mtime` to the url of all assets. When a new version of an asset is deployed, the file modification time changes, thus changing the URL, which in turn causes the browser to request the new version.

Configuring nginx to set the expires header for the url?timestamp style is discussed in a great article here.

Now that nginx is setting a maximum Expires header as appropriate, all we have to do is append the timestamp to all of our URLs:

var findit = require('findit');
var fs = require('fs');

function asset_hrefs(dir) {
  var asset_hrefs = {};
  var files = findit.findSync(dir);
  files.forEach(function(file, i) {
    var href = file.substring(dir.length);
    asset_hrefs[href] = [href, '?', fs.statSync(file).mtime.getTime()].join('');
  });
  return asset_hrefs;
};

// Run this once, when your application starts
var hrefs = asset_hrefs('public');

Pass in the directory where all your static assets live (in my case 'public'), and that function will return an Object that looks like this:

{
  '/js/all.js': '/js/all.js?1303065298000',
   ...
  '/css/style.css': '/css/style.css?1303065297456'
}

So, all you have to do is pass this structure to your view renderer, and use it like so (here I'm using Jade, and I've called the structure hrefs):

script(src=hrefs['/js/all.js'], type="text/javascript")

hrefs['/js/index.js'] of course maps to '/js/all.js?1303065298000'. Now the actual url for your Javascript file will change whenever that file is modified, so you never run the risk of the using running old code.

Start using ZSH

Tue, 12 Apr 2011 02:49:23 +0000

ZSH is a pretty cool shell, and I've been using it for a while. The main selling point is the really extensive completion support. Check it out:

As you can see, some pretty neat stuff is possible. The only real problem is that zsh has a bit of a learning curve when it comes to configuration; to tell the truth most of my config was pieced together from various blog posts and snippets that I've accumulated since I switched to zsh. The solution here is to of course perpetuate the tradition!

Check out my .zshrc here.

Minifying Javascript with Jake and UglifyJS

Mon, 28 Mar 2011 03:05:33 +0000

One of the first things you'll want to do when deploying your shiny new HTML5 app is minify all your Javascript. It saves space, and depending on the minifier, will perform some neat optimizations on your code.

Recently I had to do this on one of my apps, and I didn't feel like running my files through a minifier by hand; that's what build scripts are for! So, here is how I did it.

The minifier I've chosen to use is UglifyJS, and the build tool is Jake. You can install both via npm ('uglify-js' and 'jake' are the module names).

This is the Jakefile that I use to minify all my client-side Javascript into a file called all.js:

var fs = require('fs');
var sys = require('sys');
var uglify = require('uglify-js');

desc('Uglify JS');
task('minify', [], function(params) {
  var files = ['jquery-1.5.min.js',
               'jquery.easing.1.3.js',
               'jquery.boxy.js',
               'jquery.form.js',
               'date.js',
               'util.js',
               'layout.js'];
  
  var all = '';
  files.forEach(function(file, i) {
    if (file.match(/^.*js$/)) {
      all += fs.readFileSync('public/js/'+file).toString();
    }
  });
  
  var ast = uglify.parser.parse(all);
  var out = fs.openSync('public/js/all.js', 'w+');
  ast = uglify.uglify.ast_mangle(ast);
  ast = uglify.uglify.ast_squeeze(ast);
  fs.writeSync(out, uglify.uglify.gen_code(ast));
});

Use Jake the same way you would use make, just run `jake minify`.

CLI Interface to Manage Facebook Test Users

Sat, 26 Mar 2011 20:32:21 +0000

Facebook released a new APIto create and manage test users, however there is currently no nice interface to it (besides curl!). I decided to come up with a fdisk-style management script to handle this problem.

Check it out here:https://github.com/kcbanner/facebook-test-users

Hopefully some other devs can find this useful!

MongoDB Session Store for Connect

Thu, 17 Mar 2011 02:37:20 +0000

Recently, several updates were made to Connect, which is a middleware framework for Node applications. One of the great features of Connect is its ability to manage sessions. However, the recent updates broke the old MongoDB session store, connect-mongodb.

It was failing with the error:

TypeError: Cannot set property data of #Object which has only a getter

It seemed like it wasn't going to be updated, so I decided to try my hand at writing my own Connect 1.0+ compatible MongoDB session store. It was a fun exercise and I learned a bunch about the internal workings of Connect while I was at it.

Here is the result: connect-mongo. It's also on npm.

Proxying node.js with nginx

Sat, 12 Mar 2011 19:39:14 +0000

Sometimes you might want to run nginx in front of node.js in order to serve static content or proxy requests to several node instances.

Here's how you do it (/etc/nginx/conf.d/virtual.conf):

upstream node_blog {
    server 127.0.0.1:12000;
}

server {
    listen      80;
    server_name example.com www.example.com;

    access_log /var/log/nginx/example.access.log;

    root /path/to/docroot/public;

    location ~ /(images|stylesheets|javascripts) {        
        expires max;
    }

    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header X-NginX-Proxy true;

        proxy_pass http://node_blog/;
        proxy_redirect off;
    }
}

In this example, the node server is running on port 12000 of the localhost. I'm serving the static content with nginx and everything else is proxied to node.

You can add several upstream blocks, one for each node instance. The main idea is using proxy_pass to proxy the request to node.

Node.js blogging software

Thu, 10 Mar 2011 05:26:18 +0000

I recently finished the first usable iteration of some node.js blogging software, which I'm calling nodelog. You're seeing it in action right now!

The reasons I created this in the first place are are slightly bizarre. Firstly, I was running out of resources on my VPS, mostly because PHP and WordPress are rediculous memory hogs, but also because I was simply running too many subdomains/sideprojects/etc. Something had to be done to free up some resources.

Secondly, there has been a bunch of hype around node.js recently, and I'd been waiting for an opportunity to try it out. What better way than to write some blogging software. Two birds with one stone!

As it turns out, node.js is really great, has a huge whack of libraries for things like routing, middleware, and template rendering, and is fairly lightweight.

So far I'm really enjoying working with server side javascript. More on this topic soon!

Check out the project on github.

-Casey

Another Calculus Timelapse

Tue, 13 Apr 2010 04:00:00 +0000

Huzzah! Its exam time again and I'm back at my usual antics. This is what I did today:

Calculus Timelapse from Casey Banner on Vimeo.

In-terminal tabbing with urxvt

Fri, 02 Apr 2010 04:00:00 +0000

Not many people know about the built in perl extensions to urxvt. There are a bunch of them, but the one I find most useful is tabbing. All you have to do is add:

 URxvt.perl-ext-common: default,tabbed

to your ~/.Xdefaults file, and restart your terminal.You'll get something like this:

The keyboard shortcuts are what make this awesome. Shift-down to create a new tab (or click 'new'), and Shift-left/right to cycle between tabs.

Minimal tracks

Mon, 29 Mar 2010 04:00:00 +0000

I've been really getting into ambient and minimal stuff. Here are a couple tracks that I've been working on over the past few days.

Depth Perception by kcbanner

Cleanroom by kcbanner

Calculated Scribblings by kcbanner

Enjoy. If you wanted higher bitrate versions, I would be happy to provide them.

Calculus Timelapse

Tue, 16 Feb 2010 05:00:00 +0000

I spent the day studying for my MATH211 midterm, extreme fun was had by all involved. Seriously, things got dangerous.

I don't know why but I really really like timelapses.

Man + computer

Sun, 24 Jan 2010 05:00:00 +0000

A little experiment.

Insight from Casey Banner on Vimeo.

Nouveau incompatible with nVidia binary driver.

Mon, 21 Dec 2009 05:00:00 +0000

In case your using the nVidia binary driver and are upgrading your kernel to 2.6.32 any time soon, make sure to blacklist the nouveau module in whatever form your distro uses. It seems that the nouveau module is including by default in 2.6.32, which I was unaware of at first.I was getting a strange error when starting X:

Failed to initialize the NVIDIA graphics device PCI:1:0:0.
Error opening /dev/nvidia0, Input/Output error.

A look into dmesg showed:

NVRM: loading NVIDIA UNIX x86 Kernel Module  195.22  Sun Nov 22 16:55:48 PST 2009
vmap allocation for size 16781312 failed: use vmalloc= to increase size.
NVRM: RmInitAdapter failed! (0x26:0xffffffff:1138)
NVRM: rm_init_adapter(0) failed

Hm. So that was strange. As it turns out, if the nouveau module is loaded, it will cause loading the binary nVidia to fail in exactly this way. A quick 'rmmod nouveau' confirmed this was the case.Hopefully this helps anyone who runs into this problem.

Connecting to a WPA WiFi network on Linux (CLI)

Fri, 18 Dec 2009 05:00:00 +0000

Most articles make this out to be much more complicated than it needs to be. First, make sure you have wpa_supplicant. On Archlinux:

$ sudo pacman -S wpa_supplicant

You will need to add your network definition to wpa_supplicant.conf:

# wpa_passphrase YOUR_SSID YOUR_PASSPHRASE >> /etc/wpa_supplicant.conf

Finally, create a script to bring the interface up. I put mine in ~/scripts/wireless.sh:

#!/bin/sh
sudo pkill wpa_supplicant
sudo pkill dhcpcd
sudo ifconfig wlan0 up
sudo iwconfig wlan0 essid YOUR_SSID
sudo wpa_supplicant -B -Dwext -i wlan0 -c /etc/wpa_supplicant.conf
sudo dhcpcd wlan0

Change your ssid and interface accordingly. Run that script to connect!

Getting Apple keyboard F keys to work on Arch Linux

Fri, 13 Nov 2009 05:00:00 +0000

You need to do add the following line to /etc/rc.local:

echo 2 > /sys/module/hid_apple/parameters/fnmode

Via: http://bit.ly/dbOq

How to get nice, smooth, X11 fonts in your terminal and editors

Thu, 02 Apr 2009 04:00:00 +0000

We all know that bitmap fonts are great, but sometimes its nice to have some smoothing and anti aliasing to make things easier on the eyes. How is this accomplished? By adding this to your ~/.Xdefaults file:

Xft.hinting: true
Xft.hintstyle: hintslight
Xft.antialias: true
Xft.rgba: none
Xft.dpi: 96

If your using an LCD you can set Xft.rgba to rgba to do sub-pixel smoothing. Personally I don't like it, but it could just be the LCDs I'm using. Also, if the font your using supports different levels of hinting, you can change Xft.hintstyle to 'hintslight', 'hintsfull', 'hintsmedium', or 'hintsnone'. All fonts seem to do these differently, so you have to experiment a little bit.

After you change your ~/.Xdefaults, you have to merge it to your current running settings (or restart X, but who wants to do that). Merge your changes with 'xrdb -merge ~/.Xdefaults'. To set different fonts for individual programs, do something like:

Emacs.font: Inconsolata-10

urxvt*font: xft:Inconsolata:size=10

You may have to consult your program's man page or docs to see what style it wants the font string in. It seems that every program does it a little bit differently. Hope this was helpful, enjoy easy reading with smooth fonts! I'm using the awesome Envy Code R font in my terminal.