deferred until inspiration hits

Down for Everyone or Just Me Greasemonkey Script

Chris Roos — Wed, 24 Jun 2009 21:07:18 +0100

I hacked on this simple greasemonkey script a while ago but never got around to writing about it. It’s probably easiest explained with a screenshot.

If the webpage you’re trying to access can’t be loaded then you’ll see that the standard ‘page not found’ firefox error page but with an additional link to the useful Down for everyone or just me service.

Get the script from userscripts.

Apacherb Is Now Hostess

Chris Roos — Tue, 23 Jun 2009 21:41:27 +0100

A little while ago I wrote about a utility called apacherb. It took the concept of the passenger preference pane and applied it to any local directory (rather than specifically rails apps). I’d always meant to turn the utility into a gem but never quite got round to it. Luckily, James Adam moved the code to github, tidied it a little (notably making it restart apache after a change), renamed it and gemifimibulated it.

Install it with:

$ gem install hostess

Executing it without any arguments will display some simple instructions.

$ hostess
  Usage:
    hostess create domain directory - create a new virtual host
    hostess delete domain           - delete a virtual host
    hostess list                    - list hostess virtual hosts
    hostess help                    - this info

You’ll need to run it as sudo as it writes the configuration into /etc/apache2 and restarts apache.

To create a domain pointing at a local directory, you’d do something like:

$ sudo hostess create example.local ~/Sites/example.local/

This will register the local domain (the .local suffix is a convention rather than a rule) using directory services and add the apache virtual host. Open your browser and visit example.local to see your local site. Good huh.

Introducing Deliciolytics

Chris Roos — Tue, 23 Jun 2009 21:03:47 +0100

I’ve been playing around with this for a little while and although it doesn’t do most of the stuff I’d like I figure it’s still worth allowing other people to use it: deliciolytics

Its main purpose in life at the moment is to provide an aggregated feed of every bookmark in delicious for pages within your domain. Feel free to go ahead and add your own domain. You’ll need to have a sitemap and that sitemap will currently need to be at /sitemap.xml (it can be an index to actual sitemaps though). After you add a domain it’ll be in a pending state for up to five minutes before a task runs to ‘normalise’ it. After that you just have to wait until the task runs to download your urls (from the sitemap) and query delicious for any bookmarks against each of them. This currently happens once every 24 hours. Your best bet would be to subscribe to the bookmarks feed (click through to your domain and then through to bookmarks) and wait for the task to populate it with bookmarking goodness.

You could easily launch a DoS attack by automating the process of adding domains but it be great if you didn’t. Thanks.

A Greasemonkey Script that inserts a Google Site Search form into every page

Chris Roos — Fri, 24 Apr 2009 14:09:30 +0100

I often find myself wanting to use google to search within a site I’m visiting. My current method is to navigate to the search box in firefox (cmd-k) and type “site:domain-of-site search-term”. That’s a bit painful so I finally got around to creating a greasemonkey script to make it easier. Install the script from the userscripts site.

Once installed, you can hit ctrl-9 on any page (except google, that’d be crazy) and a search form will appear at the top of the page ready for you to type into.

The simplest thing is to type your search term and tap enter to view the google results. However, it’s often useful to restrict your search to more than just the domain of the site (I might want to search for things I’ve said in twitter, site:twitter.com/chrisroos, for example) so there’s a drop down list of url components that you can use to restrict which areas of the site you want to search. Pressing escape within any of the fields in the search form will hide it again.

A previous version of this script would display the search form at the top of each page all the time. That didn’t play nice with absolutely positioned elements already in the page though, hence my decision to only show it when you actually need it. There’s still work to do (one of the main problems is that the search form will currently be inserted into every frame on a page, including iframes) but I think it’s useful enough to get out there.

Rails 2.3 and the ability to update created_at, created_on, updated_at and updated_on timestamps

Chris Roos — Sun, 19 Apr 2009 08:39:23 +0100

The timing of the automatically generated timestamps and attribute assignment in rails 2.3 security note ¹ was interesting to me because I’d just spent a while really trying to get my head around attr_accessible and attr_protected and the best way to test their effects in an app I was working on. Although I intend to write about the testing separately, the conclusion I came to was that I wanted to test the behaviour (more specifically, I was interested in what I could/couldn’t assign) of my objects and not that I was specifically using attr_accessible or attr_protected.

Reading Alex’s post made me wonder whether the rails API ever guaranteed that the timestamps would be readonly: If that was the case then I can see arguments for not wanting to test the behaviour of your objects (because you’re essentially testing the framework itself). I couldn’t find any mention of whether they’d specifically be readonly so chose to do a bit of digging. I created a script that allowed me to, with relative ease, run tests against rails apps created with different versions of rails. Using this I was able to test the mass assignment of timestamps (created_at, created_on, updated_at and updated_on) against multiple versions of rails. The conclusion I came to (and I’d love for other people to replicate my experiment and prove/disprove my results) was that the created_at and created_on pair of attributes have always (at least as far back as rails 1.0.0) been assignable, while the updated_at and updated_on attributes have only recently become assignable. While digging I managed to find the lighthouse ticket (#1612) that requested the ability to set the updated_* timestamps and the rails commit that closed that ticket.

Replicating my experiment

Get a copy of rails

  $ cd /path/to/code
  $ git clone git://github.com/rails/rails.git

Create a mysql database called rails_timestamps_test, and create a table within that database called people.

  mysql> CREATE DATABASE rails_timestamps_test;
  mysql> USE rails_timestamps_test;
  mysql> CREATE TABLE people (id INTEGER AUTO_INCREMENT, created_on DATE, created_at DATETIME, updated_on DATE, updated_at DATETIME, PRIMARY KEY (id));

Get a copy of my test script

  $ cd /path/to/code
  $ svn co http://chrisroos.googlecode.com/svn/trunk/scratch/rails_timestamps_test

Run my script against rails 2.3 (run setup_rails_project.rb without any arguments to see the options) and hopefully see output similar to below (which means that the tests passed as expected).

  $ cd /path/to/code/rails_timestamps_test
  $ ruby setup_rails_project.rb 2.3.0 /path/to/code/rails
  
  *** Removing the rails app at /path/to/code/rails_timestamps_test/projects/rails-app-2-3-0
  *** Checking out the rails version tagged v2.3.0
  HEAD is now at beca1f2... Template#mime_type should not use Mime::Type when Action Controller is not included
  *** Creating the rails app at /path/to/code/rails_timestamps_test/projects/rails-app-2-3-0
  *** Vendorising rails from /path/to/code/rails to /path/to/code/rails_timestamps_test/projects/rails-app-2-3-0
  *** Generating rails_version_test.rb in /path/to/code/rails_timestamps_test/projects/rails-app-2-3-0/test/unit to ensure we are testing against the correct version of rails
  *** Copying assets to the new rails app
  ****** Linking /path/to/code/rails_timestamps_test/assets/person_test.rb to /path/to/code/rails_timestamps_test/projects/rails-app-2-3-0/test/unit/person_test.rb
  ****** Linking /path/to/code/rails_timestamps_test/assets/person.rb to /path/to/code/rails_timestamps_test/projects/rails-app-2-3-0/app/models/person.rb
  ****** Linking /path/to/code/rails_timestamps_test/assets/database.yml to /path/to/code/rails_timestamps_test/projects/rails-app-2-3-0/config/database.yml
  *** Running the rails version test to ensure that we're testing against the correct version of rails
  Loaded suite test/unit/rails_version_test
  Started
  .
  Finished in 0.000327 seconds.

  1 tests, 1 assertions, 0 failures, 0 errors
  *** Running the timestamps test
  Loaded suite test/unit/person_test
  Started
  ....
  Finished in 0.078819 seconds.

  4 tests, 8 assertions, 0 failures, 0 errors

Re-run the script against rails 2.2.2 (the last tag before 2.3) and you should see the same as above except for the last test which should now contain two failures.

  $ cd /path/to/code/rails_timestamps_test
  $ ruby setup_rails_project.rb 2.2.2 /path/to/code/rails
  
  ... some lines snipped ...
  *** Running the timestamps test
  Loaded suite test/unit/person_test
  Started
  ..FF
  Finished in 0.069429 seconds.

    1) Failure:
  test_should_be_able_to_set_updated_at(PersonTest)
      [test/unit/person_test.rb:35:in `test_should_be_able_to_set_updated_at'
       /path/to/code/rails_timestamps_test/projects/rails-app-2-2-2/vendor/rails/activesupport/lib/active_support/testing/setup_and_teardown.rb:94:in `__send__'
       /path/to/code/rails_timestamps_test/projects/rails-app-2-2-2/vendor/rails/activesupport/lib/active_support/testing/setup_and_teardown.rb:94:in `run']:
  FAIL: Couldn't persist the mass assigned updated_at attribute.
  <Thu Jan 01 00:00:00 +0000 2009> expected but was
  <Sun, 19 Apr 2009 09:23:46 UTC +00:00>.

    2) Failure:
  test_should_be_able_to_set_updated_on(PersonTest)
      [test/unit/person_test.rb:28:in `test_should_be_able_to_set_updated_on'
       /path/to/code/rails_timestamps_test/projects/rails-app-2-2-2/vendor/rails/activesupport/lib/active_support/testing/setup_and_teardown.rb:94:in `__send__'
       /path/to/code/rails_timestamps_test/projects/rails-app-2-2-2/vendor/rails/activesupport/lib/active_support/testing/setup_and_teardown.rb:94:in `run']:
  FAIL: Couldn't persist the mass assigned updated_on attribute.
  <Thu, 01 Jan 2009> expected but was
  <Sun Apr 19 09:23:46 UTC 2009>.

  4 tests, 8 assertions, 2 failures, 0 errors

Re-run the script against any other tagged version of rails you wish.

A couple of notes about the script

If you specify an invalid tag (i.e. a tag that doesn’t exist in the rails repository) then the rails_version_test.rb will fail.

  $ cd /path/to/code/rails_timestamps_test
  $ ruby setup_rails_project.rb made.up.tag /path/to/code/rails
  
  ... some lines snipped ...
  Loaded suite test/unit/rails_version_test
  Started
  F
  Finished in 0.045543 seconds.

    1) Failure:
  test_should_be_using_rails_made_up_tag(RailsVersionTest)
      [test/unit/rails_version_test.rb:6:in `test_should_be_using_rails_made_up_tag'
       /path/to/code/rails_timestamps_test/projects/rails-app-made-up-tag/vendor/rails/activesupport/lib/active_support/testing/setup_and_teardown.rb:94:in `__send__'
       /path/to/code/rails_timestamps_test/projects/rails-app-made-up-tag/vendor/rails/activesupport/lib/active_support/testing/setup_and_teardown.rb:94:in `run']:
  <"made.up.tag"> expected but was
  <"2.2.2">.

  1 tests, 1 assertions, 1 failures, 0 errors

If you specify a tag of rails that is anything other than a 3 part version number (e.g. the four part version 2.3.2.1, or the release candidate 2.1.0_RC1) then the version test will fail because the Rails::VERSION constant only contains the MAJOR, MINOR and TINY components and that’s what we use to check the version our app uses.

The supplied database.yml file assumes a mysql database running on localhost that has a root user with no password. You can update this to reflect your actual environment.

The projects are created in /path/to/code/rails_timestamps_test/projects/ with a name like rails-app-x-x-x (major, minor, tiny).

In order to ensure we are testing against the correct version of rails, I symlink rails from /path/to/code/rails (after changing to the requested tag) into rails-app-x-x-x/vendor/rails. The problem here is that if you re-run the script with a different rails version and then attempt to run a test from your first project that it will be using the recently specified version of rails. Luckily, the rails_version_test will fail fast which means you shouldn’t have to waste time wondering why something is not working as expected.

I wonder whether it’d be useful to extract the ‘create a rails app from a specific version of rails’ functionality out of this script?

¹ Alex MacCaw raised the issue in reference to number eight in the list of 10 cools things in Rails 2.3.

Generating and Inserting a Rel=Canonical Link Into Pages With Firefox and Greasemonkey

Chris Roos — Tue, 14 Apr 2009 14:14:00 +0100

I’ve been thinking about this for over a year but have only recently got around to doing something about it. I’ve also chosen to use rel=canonical rather than rel=bookmark.

The magic is all in this greasemonkey script. It uses little rule objects to generate the canonical urls for, at the moment, ebay, google and theyworkforyou and insert those urls into the page as a link element with rel=canonical. Combining my script with the show canonical greasemonkey script (found on the tripwolf blog) allows me to easily visit and use the canonical url in emails and the like.

You should just about be able to see the relatively short canonical url compared to the large url generated by my searching from the firefox search box.

I hope to expand the set of rules (if you wanted to create one right now I’d suggest looking at the source of the script, the rules are defined towards the end of the script, and the rule tests) and possible have some central repository of rules that the script could use. If you know of some sites that you’d like to see canonical urls for, but don’t want to create the rule, then let me know and I’ll see if I can create it for you.

CruiseControlrb With Passenger and Launchd on a Mac

Chris Roos — Tue, 31 Mar 2009 18:27:25 +0100

This has taken me ages, is quite fiddly and I’m still not 100% happy with it but it seems to work well enough to document here.

Configuring the server to run under Passenger

Buy a Mac. Hmm, maybe that’s starting a little too early in the process…

Install Passenger.

Download cruisecontrol.rb (from github or rubyforge). We’re using the version from github as it has git support built in.

Edit the cruisecontrol.rb production environment (cc.rb/config/environments/production.rb) so that we can disable the builders by setting an environment variable. By default, starting the server (even through Passenger) starts the builders too. This doesn’t work by default because the PATH environment variable isn’t set which means the builders can’t find git. Although it doesn’t feel like entirely the right approach I actually wonder if I could use the ruby-wrapper-approach (detailed below) to have these builders work correctly.

   config.after_initialize do
     require CRUISE_DATA_ROOT + '/site_config' if File.exists?(CRUISE_DATA_ROOT + "/site_config.rb")
     require RAILS_ROOT + '/config/dashboard_initialize' 
  -  BuilderStarter.start_builders 
  +  BuilderStarter.start_builders unless ENV['CCRB_WITHOUT_BUILDERS']
   end

Create a wrapper around ruby that allows us to pass environment variables to our Passenger apps. I actually made the amendment to the code above before realising that it’s not entirely trivial to pass environment variables from Apache to a Passenger app (this ticket details the problem). I wasn’t keen on the idea of this wrapper (explanation on the phusion blog) at first but I ended up thinking it was the neatest solution.

  $ cat > ~/ruby_for_passenger
  #!/bin/sh

  export CCRB_WITHOUT_BUILDERS=true

  exec "/usr/bin/ruby" "$@"

  $ sudo mv ~/ruby_for_passenger /usr/local/bin/
  $ sudo chmod +x /usr/local/bin/ruby_for_passenger

Change the value of the PassengerRuby configuration option in your apache httpd config. We need to tell Passenger to use our wrapper script instead of the default ruby binary it found during installation.

  $ sudo vi /etc/apache2/httpd.conf
  #PassengerRuby /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby
  PassengerRuby /usr/local/bin/ruby_for_passenger

Configure the passenger virtual host for your cruise instance. I strongly recommend using the Passenger preference pane to do this (remember to set it to use the production environment).

Navigate to your chosen local domain and you should be presented with the default cruise web interface.

Configuring the builders to start under launchd

Edit the cc.rb/script/builder file to set the default log directory to a path relative to the project root. This is so that we can run the builders from a location other than within cc.rb and still have them log correctly (the default behaviour is that we receive a warning about a missing log directory and the log output is redirected to stdout).

  -  CRUISE_OPTIONS[:log_file_name] = "log/#{CRUISE_OPTIONS[:project_name]}_builder.log"
  +  CRUISE_OPTIONS[:log_file_name] = File.join(RAILS_ROOT, 'log', "#{CRUISE_OPTIONS[:project_name]}_builder.log")

Create a launchd property list for each task. I’ve got mine in ~/Library/LaunchAgents/label-from-plist in the home directory of a user that has public key access to the git repository. This is my one annoyance with this current solution: It requires the user mentioned above to be logged in. I spent a very long time trying to get the builders working as launch daemons, rather than agents, so that they would build without a user having to be logged in. I was hoping to find an environment variable I could set so that git or SSH knew to use the public key from a user other than root (who the launch daemons run as) but didn’t have any such luck. If anyone has any ideas about this I’d love to hear them. Note Although I haven’t tried it, I suspect that it’d be trivial to get non-public-key subversion builders to run as daemons.

  <?xml version="1.0" encoding="UTF-8"?>
  <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">
    <dict>
      <key>EnvironmentVariables</key>
      <dict>
        <key>PATH</key>
        <string>/usr/bin:/opt/local/bin</string>
      </dict>
      <key>Label</key>
      <string>label-describing-your-builder</string>
      <key>ProgramArguments</key>
      <array>
        <string>/Users/chrisroos/cruisecontrol.rb/cruise</string>
        <string>build</string>
        <string>name-of-project-to-build</string>
      </array>
      <key>RunAtLoad</key>
      <true/>
    </dict>
  </plist>
  
  # Notes:
  # We set the PATH environment variable so that both git and ruby can be found.
  # We set RunAtLoad so that as soon as the plist is loaded by launchctl it will start to run.

Load the agent using launchctl as the user named above (i.e. not root).

  $ launchctl load ~/Library/LaunchAgents/label-from-plist

And that should be about it. I’d appreciate any feedback anyone has about this approach and to hear about alternative approaches of getting cruise running automatically on system boot.

Installing Passenger 2.1.2 on CentOS 5.2

Chris Roos — Mon, 30 Mar 2009 08:24:00 +0100

I needed to upgrade Passenger on our web server last week (so that we could run Rails 2.3 apps) but ran into a little problem getting it installed.

We’re running CentOS 5.2 on our staging server.

  [root@rails-staging0 ~]# cat /etc/redhat-release 
  CentOS release 5.2 (Final)

The passenger gem installed OK but when I ran passenger-install-apache2-module I got the following error.

  ... other lines snipped ...
  
  /usr/bin/ld: skipping incompatible /usr/lib/libaprutil-1.so when searching for -laprutil-1
  /usr/bin/ld: cannot find -laprutil-1
  collect2: ld returned 1 exit status
  rake aborted!
  Command failed with status (1): [g++ -shared SystemTime.o Utils.o Bucket.o ...]
  /var/ruby-enterprise-1.8.6-20080810/lib/ruby/gems/1.8/gems/passenger-2.1.2/Rakefile:157
  (See full trace by running task with --trace)

  --------------------------------------------
  It looks like something went wrong

Searching for this error didn’t yield any definite fixes but a couple of posts mentioned re-installing the apr-devel packages. I tried apr-devel to begin with but was told that it was already up-to-date.

  [root@rails-staging0 ~]# yum install apr-devel
  ... other lines snipped ...
  Nothing to do

I can’t remember why but I tried apr-util-devel next which did result in the package being installed.

  [root@rails-staging0 ~]# yum install apr-util-devel
  ... other lines snipped ...
  Installed: apr-util-devel.x86_64 0:1.2.7-7.el5
  Complete!

With this package re-installed I was able to go ahead and install the passenger apache module (i.e. passenger-install-apache2-module succeeded).

One slightly interesting/odd thing was that yum reported that the same version (1.2.7-7.el5) of apr-util-dev was already installed. The only difference after the re-install was that its status changed from ‘base’ to ‘installed’.

  # before the re-install
  [root@rails-staging0 ~]# yum list | grep apr-util-devel
  apr-util-devel.x86_64                    1.2.7-7.el5            base
  
  # after the re-install
  [root@rails-staging0 ~]# yum list | grep apr-util-devel
  apr-util-devel.x86_64                    1.2.7-7.el5            installed

Hack the Government Day (Rewired State)

Chris Roos — Mon, 09 Mar 2009 22:14:51 +0000

James Darling, Richard Pope and Emma Mulqueeny organised the excellent Rewired State on Saturday (7th March) at the new Guardian offices in Kings Cross. The idea behind rewiredstate was pretty simple (from rewiredstate.org):

Government isn’t very good at computers.
They spend millions to produce mediocre websites, hide away really useful public information and generally get it wrong. Which is a shame.

I had a plan for the day. It’s a plan I’d had for about a year or so and it involved the atrocity that is Active Places (a Sport England funded project). It’s really hard to find information about sports facilities (I was specifically interested in public swimming pools) in the UK. I know this because I’ve tried to find public swimming pools that are either near where I stay (when in London), easily accessible from a station on the commute, or near work. At the moment that involves finding and searching the websites of each of the councils whose boundaries I cross or am near. This is certainly not ideal. It’s more frustrating because the government have this information locked up (the entire site requires cookies so there’s no chance of it appearing in google) in activeplaces. Oh, and there’s another little gem of information: activeplaces cost £5.2 million over three years.

Before the event on Saturday, I spent quite a while scraping the data out of activeplaces. This involved issuing a search for each of the postcode outcodes in England (I thought I had this postcode data for the UK but apparently didn’t) and parsing the search results to get a unique list of sports centres. I could then download the relevant site, and facility (which is in separate pages and displayed through an iframe on the main site page), information for each centre.

On the day I was lucky enough to get help from James Adam, James Andrews and Tom Taylor to work on the active places project. I had a rough plan to geocode the sports facility data we had, parse the downloaded site data to extract the access type of the facility (i.e. private membership or publicly accessible) and create a simple alternative to activeplaces. Tom helped me geocode the data (using graticule and the multimap api), while James Adam parsed the access type and James Andrews worked on the site. The geocoded data made it incredibly easy to produce a KML file of sites that Tom could consume in his excellent I am near service (swimming pools I am near). At the last minute I put together a very simple sinatra webservice that allows for the querying of the activeplaces data (by facility and access type) and returns the data in kml (source). You can plug the url of this kml file directly into google maps or download it and plug it into google earth. All that was left was to present (‘slides’ are on flickr) what we’d done (in just two minutes) and go to the pub.

There was a chap from DirectGov at the event and he mentioned that he was interested in us continuing the work we’d started. I’ve also spoken to someone from London 2012 as it sounds like they’re aiming to do something similar. So, not only was it a great day but there’s a chance that we might be able to help shape some of the stuff that the government plans to work on.

Posting Bookmarks to Delicious From My Android G1

Chris Roos — Wed, 04 Mar 2009 23:09:41 +0000

For a while I’ve wanted the ability to bookmark pages in delicious from my Android G1. After a few weeks of occasionally looking to see if anyone had done the work for me I started to investigate building an Android app. I got about as far as reading through some of the tutorials but never seemed to find the time to really get stuck in. In the meantime, to get around the limitations of the client side only bookmarks on the G1, I would save a draft email containing the URL of the page I was interested in and later open it on a desktop machine and post to delicious. Saving a draft is fairly trivial in that you visit a page in the browser, hit the share menu item, choose gmail and hit save draft (sharing via gmail pastes the URL of the current page into the body of the email) but the whole process was a bit cumbersome. I think it was about a week ago when I realised that I should just be able to email my bookmarks to delicious directly. A little searching appeared to suggest that there was nothing to let me do this¹ so I went about hooking a few things together in order to get the functionality I was after.

I have recently discovered just how simple and powerful rack is, especially when combined with Phusion Passenger (which makes it stupidly easy to deploy rack apps). It seemed ideal for the job. Combine that with smtp2web (I recently rediscovered this after catching up on all the webhook buzz that seems to be doing the rounds) and a little bit of ruby to talk to delicious and I had everything I needed. So, I spent an hour or so last night getting a little end to end proof of concept up and running. Happy that I could hook all the components together I spent a further two or three hours tidying everything up and getting an app together (source in google code) that I could deploy to my exonetric VPS. I was even able to use the service for real on the train this morning.

Feel free to deploy the code to your own server (or the awesome heroku) if you wish to post bookmarks to your own account. You’ll need to copy config/delicious.yml.example to config/delicious.yml and enter your delicious credentials. The email parser expects to get the bookmark title in the subject line of the email (required), the URL on the first line of the email body (required) and tags on a line in the body beginning with T (i.e. /^T (.*)/). The rest of the body is treated as the notes. I was originally going to see if there was any interest in a hosted service whereby you register your delicious credentials and receive a secret email address to use in order to post to delicious. It seems that ping.fm already has this covered so I’m not going to worry about hosting it for anyone but me.

I love how relatively easy it is to hook these things together, in a short period of time, to produce something useful.

Note. I searched again for android delicious clients just before I started writing this post and was slightly surprised to see that there are now two apps that offer this functionality: Android delicious bookmarks and Beelicious (for which I couldn’t find a link).

¹ Having just repeated the search I found this post that suggests using ping.fm to do exactly what I’m after. Oh well.