Ronin on Rails

Madame Pain's Boudoir Circus

noreply@blogger.com (Anonymous) — Thu, 14 Oct 2010 11:00:00 +0000

Nothing at all to do with computers, this time.

Some people at LRUG asked about the circus show I'm in. So, here are the details.

MADAME PAIN’S BOUDOIR CIRCUS

Alice Cooper’s Favourite Freak Chrisalys is to wreak madness as ‘Madame Pain’ in AirCraft Circus’ critically acclaimed side-show spectacular ‘Madame Pain's Boudoir Circus.’

www.madamepainsboudoircircus.com

Chrisalys - TimeOut’s favourite circus freak and ‘stand out’ star of Alice Cooper’s upcoming Roundhouse show Halloween Night of Fear (as featured in Bizarre Magazine) is to revisit the shadows of the cult hit spectacular Madame Pain’s Boudoir Circus.

Previously only available as an underground sneak peak to the elite few, the doors are unbolting and the lights are lifting on London’s most diverse troupe of aerial acrobats and sideshow miscreants. Telling the tale of the world’s greatest serial killer, shrouded in an underworld of forgotten freaks and Parisian burlesque dancers.

“Despite having seen some of the jaw dropping spectacles of the Moscow state circus and the Berlin circus, I still have not seen a performance that matches the fusion of ideas, spectacle, visual image and musical extravagance of Madame Pain’s Boudoir Circus” – James Underwood

Witness an outrageous aerial display from a cast of 35 swinging trapeze, pole dancing, fire eating, extreme sports and shocking sideshow acts with live music and DJ sets, whilst raising funds for the ‘Heat the Hangar’ project, in aid of the Hangar Arts Trust, home of Aircraft Circus.

See the Showreel on YouTube!

For press enquiries contact:

Moira Campbell Press Manager on 07976 618163 moira@aircraftcircus.com

Mike Wells Press and Marketing Assistant mike@aircraftcircus.com

Listing Information

Tickets and information www.madamepainsboudoircircus.com 020 8317 8401

The Hangar Arts Trust

Unit 7A, Mellish House

Harrington Way

Charlton

SE18 5NR

Performances:

Friday 19th @ 8pm + opening night party with guest DJ’s

Saturday 20th @ 8pm

Sunday 21st @ 7pm

Friday 26th @ 8pm

Saturday 27th @ 8pm

Sunday 28th @ 7pm + after show party

This event is for over 16's only, due to adult themes.

Tickets are £10.00 early bird and £12.50 from 31st October and on the door.

Running time 1hr 15min (no interval)

Early booking is advised.

Hope to see you there.

Using Ipredator on OS X

noreply@blogger.com (Anonymous) — Wed, 16 Jun 2010 19:36:00 +0000

[UPDATE] There seems to be a bug in Snow Leopard where changes made to the Airport interface's MTU do not persist - it just jumps back to 1500 immediately. To make the change stick, you need to do this in a terminal window;

$ sudo ifconfig en1 mtu 576

(Assuming en1 is your airport interface, and 576 is the MTU value you want)

Sadly, the change will not persist through a reboot, so you'll need to run the command whenever you boot up your mac.

______________________________

I'm not a big fan of being spied on, so I've signed up to IPredator, the VPN service run by the people who brought us The Pirate Bay.

Setting it up on OS X is pretty trivial, and I haven't noticed much, if any, difference in network speeds. But, I did find myself unable to access certain websites. Notably my Basecamp site, which is quite inconvenient. I've got HTTPS turned on, and when using my VPN connection, I couldn't load the login page - it fails as soon as it tries to do the TLS handshake. Turn off the VPN, and everything is just fine. Some other sites behaved in a similar way - even some sites that weren't using HTTPS (although digging into the page content suggests that it is https elements on the page that were causing the browser to hang).

37Signals support, sadly, were no help in this case. But, I did find this page which suggested changing the MTU (Maximum Transmission Unit) setting of my TCP/IP connection.

That page talks about how to do this on Windo$e, but I found that the problem went away if I made the corresponding change on OS X. So, I'm posting this here in case anyone else finds it helpful.

First of all, in "Network Preferences", select your main network connection (I'm using wired ethernet), and click "Advanced".

In the advanced options, click "Ethernet" and look for the MTU setting. Change it from "Standard (1500)" to "Custom", and enter a lower value. In my case, I'm using 576.

Click "OK", click "Apply" and then disconnect and reconnect your VPN connection.

That's it. I can now access my Basecamp site via my IPredator VPN connection. I hope this helps someone else.

Have a happy & private Internet!

Book review: Zabbix 1.8 Network Monitoring

noreply@blogger.com (Anonymous) — Sat, 08 May 2010 10:01:00 +0000

Packt publishing asked me to review "Zabbix 1.8 Network Monitoring" (Disclosure: They gave me a copy of the book. Other than that, I have not been compensated for this review in any way).

I've been using Zabbix for a few years to monitor a variety of systems. I like it for its small (initial) learning curve and the out-of-the-box graphs and reports. But, the kindest word I can use to describe the provided documentation is "minimal". So, I was pleased to see a book that tries to address this problem.

At just under 400 pages, there's a lot of material. The book takes a tutorial approach, going from installation through to fairly advanced monitoring ideas including distributed monitoring with Zabbix proxies.

The book's target audience is sysadmins (aka "devops") and developers who have to do their own monitoring. So, the level of assumed technical ability is quite high. As usual with technical books, this can lead to over-explanation of some steps and under-explanation in others. But, I found the tone and level of detail struck quite a nice balance. YMMV, of course.

As well as covering the detail of the steps you need to take to setup Zabbix and start monitoring, there are good explanations of the high-level architecture of how the various Zabbix components fit together and communicate, which I found very helpful.

The author, Richards Olups, works with Zabbix SIA (who develop Zabbix), and his in-depth knowledge of the system is obvious throughout the book.

For example, I was pleased to see a section about digging into the underlying database Zabbix uses to store configuration and historical values. Zabbix's PHP front-end is functional, but can be a bit quirky. IMO, it's ripe for replacement/augmentation with something a bit slicker - maybe a Rack application using some Flash-based graphing tools. Having a guide through the database structure will make it a lot easier to create something like that.

Also nice are some tips about using the unix command-line tools to help troubleshoot monitoring problems. e.g. using telnet to connect to the zabbix-agent process to confirm it's listening correctly for connections from the server. This is basic stuff, but a worrying number of developers (and even some sysadmins) I've met don't seem to have much awareness of the toolbox unix provides.

The section on how to upgrade Zabbix and patch its database is also a nice inclusion. A quick glance at the Zabbix support forums shows that that is often a problem area, so I like the fact that this has been addressed upfront. Using Zabbix to monitor its own health is also covered, and the section contains lots of advice that would have saved me at least one weekend spent rebuilding my monitoring from scratch when an earlier version of Zabbix decided to eat its database, one day!

As you can probably tell, I like this book a lot, which is just as well since it appears to be the only book on Zabbix available right now.

There are a few nitpicks, perhaps. The step-by-step tutorial approach makes the book far more suited to being read through rather than as a reference, but since this is the first book available (AFAIK), I think that's probably a good choice. I would have liked to see Ubuntu covered in the installation section (the author covers compilation/installation on SuSe and Slackware), although the book's target audience shouldn't have any trouble adapting the instructions for their distro of choice. The writing and grammar are a little quirky, and a glossary would have been useful. But, these are really minor points.

Overall, I think the book has a lot to offer anyone who is using, or thinking of using Zabbix as their monitoring solution. If I hadn't been given a copy, I'd probably buy it.

Incremental dump of a mysql database

noreply@blogger.com (Anonymous) — Mon, 21 Dec 2009 15:59:00 +0000

I've just written a couple of simple ruby classes to help dump a mysql database without locking tables for extended periods.

In particular, some of my tables contain lots of date-based data which is not changed after the day it was created. The classes can be told to dump such data one day at a time, and won't dump a day that has already been dumped.

Check it out on github;

http://github.com/digitalronin/DbDumper

I hope people find it useful.

David

Introduction to Hive on Amazon Elastic Map Reduce

noreply@blogger.com (Anonymous) — Thu, 12 Nov 2009 14:33:00 +0000

Background

Scaling an application usually involves adding processing nodes. This means you end up with valuable data (e.g. server logs) existing on multiple different machines. Very often, we want to mine those logs for useful information.

One way to do this would be to put all the logs in one place and run some kind of computation over all the data. This is relatively simple, but it really doesn't scale. Pretty soon, you reach the point where the machine which is analysing a day's worth of log data is taking more than a day to do it.

Another way is to let each processing node analyse its own logs, and then have some way to collate the analysis from each node to find the overall answer you're looking for. This is a more practical solution for the long term, but it still has a couple of problems;

1. A processing node should be processing - i.e. doing the task for which it was created. Ideally, you wouldn't want to bog it down with other responsibilities. If you do, you probably need to make the node more powerful than it needs to be in order to carry out its primary task, so you're paying for extra capacity just in case you want to run a log analysis task.

2. In the case of log data in particular, keeping the logs on the node which created them generally means you have to keep the node around too. This makes it awkward to remove nodes if you want to scale down, or replace nodes with better nodes, because you have to worry about copying the logs off the node and keeping them somewhere.

It would be nice if we could have each node push its logs into something like Amazon S3 for storage, and spin up a distributed computing task whenever we want to run some analysis. Amazon Elastic Map Reduce (EMR) is designed to work in exactly this way, but the learning curve for writing map/reduce job flows is pretty steep - particularly if you're used to writing simple scripts to get useful information out of log data.

As of October 1st 2009, Amazon EMR supports Apache Hive, which makes things a lot easier.

What is Hive?

The proper answer is here.

The way I think of Hive is that it lets you pretend that a whole mess of semi-structured log files are actually big database tables, and then helps you run SQL-like queries over those tables. All this without having to actually insert the data into any kind of table, and without having to know how to write distrubuted map/reduce tasks.

Using Hive with Amazon EMR

This is a very basic introduction to working with Hive on Amazon EMR. Very basic because I've only just started looking into this myself.

You will need to be signed up for Amazon Web Services, including S3 and Elastic Map Reduce.

I'm going to go through part of an exercise from the Cloudera Introduction to Hive, which I strongly recommend working through. That training exercise uses a Cloudera VMWare virtual appliance running Hive. Here is how to I did a similar task using Hive on Amazon EMR.

For this exercise, we're going to take a data file consisting of words and the frequency of occurrence of those words within the complete works of William Shakespeare. The file consists of a number of lines like this;

25848   the
23031   I
19671   and
18038   to
16700   of
14170   a
12702   you
11297   my
10797   in

The first value is an integer saying how many times the word occurs, then a tab character, then the word. This file is generated by an earlier exercise in one of the Cloudera Hadoop tutorials. If you don't feel like running through those exercises, just generate a file containing a bunch of numbers and words, separated by a tab character, and use that.

Upload the data to S3

Before we can analyse the data, we need it to be available in S3. Create a bucket called "hive-emr", and upload your data file into it using the key "/tables/wordfreqs/whatever". In my case, I have the tab-delimited text file in;

s3://hive-emr/tables/wordfreqs/shakespeare.txt

NB: The S3 path "hive-emr/tables/wordfreqs" is going to be our Hive table. If you're unfamiliar with S3, "hive-emr" is the name of our bucket, and 'tables/wordfreqs/shakespeare.txt' is the key whose value is the contents of our "shakespeare.txt" file.

Everything in the 'directory' "tables/wordfreqs/" (which isn't really a directory, but we can pretend it is) must be parseable as data for our table, so don't put any other types of file in there. You could, if you wanted, have more than one tab-delimited text file though, and all of the data in all of those files would become records in the same Hive table.

It's also important not to have any underscores in the S3 bucket or key. S3 will happily let you create and upload files to buckets/keys with underscores, but you'll get an S3 URI error when you try to create the table in Hive.

I'm using s3sync to upload the data files, but you can use anything you want provided you get the data into S3 with the correct bucket and key name.

Generating an EC2 Key Pair

We need a key pair to enable us to SSH onto our Hive cluster, when we've started it. If you don't have a suitable key pair already, sign in to the Amazon Web Services console and go to the Amazon EC2 tab. Near the bottom of the left-hand column, use the "Key Pairs" function to generate a key pair and save the secret key to your local machine.

Be aware of the "Region" you're using - key pairs will only work for servers of the same region. I'm using "EU-West", but it doesn't matter which you use, as long as you're consistent.

Starting Hive

Sign in to the Amazon Web Services console and go to the Amazon Elastic MapReduce tab (you won't see the tab if you haven't signed up to the service, so make sure you do that first).

Click "Create New Job Flow". Make sure you're using the same region you used when you generated your key pair.

Give the job flow a name, and select "Hive Program" for the job type.

On the next screen, choose "Start an Interactive Hive Session".

On the next screen, we choose the number and size of the machines we want to comprise our cluster. In real life use, using a lot of big machines will make things go faster. For the purpose of this exercise, one small instance will do. We're not doing anything heavyweight here, and we only have one data file, so there isn't much point spending the extra money to run lots of large machines.

Select the key pair you generated earlier, and start the job flow. Don't forget to terminate the job flow when you've finished, otherwise you'll be paying to keep an idle cluster going.

Now we have to wait for the cluster to start up and reach the point where it's ready to do some work. This usually takes a few minutes.

When the job flow status is "WAITING", click on the job flow and scroll down in the lower pane to get the "Master Public DNS Name" assigned to your cluster so that we can SSH to it.

From a terminal window, ssh onto your cluster like this;

ssh -i key/hive.pem hadoop@ec2-79-125-30-42.eu-west-1.compute.amazonaws.com

Replace key/hive.pem with the location and filename of the secret key you created and saved earlier.

Replace "ec2-79-125-30-42.eu-west-1.compute.amazonaws.com" with the Master Public DNS Name of your cluster. The username 'hadoop' is required.

You should now have a terminal prompt like this;

Type "hive" to get to the hive console. This is an interactive shell that works in a similar way to the "mysql" command-line client.

Creating a table

We're almost ready to start querying our data. First, we have to tell Hive where it is, and what kind of data is contained in our file.

Type these lines into the hive shell;

hive> create external table wordfreqs (freq int, word string)
  > row format delimited fields terminated by '\t'
  > stored as textfile
  > location 's3://hive-emr/tables/wordfreqs';
OK
Time taken: 1.29 seconds

Note that we didn't need to put "shakespeare.txt" as part of the location. Hive will look at the location we gave it and, provided all the "files" in that "directory" have the right kind of contents (lines consisting of an integer, a tab character and a string), all of their contents will be accessible in the 'wordfreqs' table.

Now that we've told Hive how to find and parse our data, we can start asking questions in almost the same way as we would do if it were in a mysql table.

Try this;

hive> select * from wordfreqs limit 5;
OK
25848   the
23031   I
19671   and
18038   to
16700   of
Time taken: 4.868 seconds
hive>

So far, so good - even though that's a long time to take for a very simple query. Let's try something a little more interesting;

hive> select count(word) from wordfreqs;

Here is the output I got from this;


Total MapReduce jobs = 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=
In order to set a constant number of reducers:
set mapred.reduce.tasks=
Starting Job = job_200911121319_0001, Tracking URL = http://ip-10-227-111-150.eu-west-1.compute.internal:9100/jobdetails.jsp?jobid=job_200911121319_0001
Kill Command = /home/hadoop/bin/../bin/hadoop job  -Dmapred.job.tracker=ip-10-227-111-150.eu-west-1.compute.internal:9001 -kill job_200911121319_0001
2009-11-12 01:37:33,004 map = 0%,  reduce =0%
2009-11-12 01:37:46,653 map = 50%,  reduce =0%
2009-11-12 01:37:47,665 map = 100%,  reduce =0%
2009-11-12 01:37:55,709 map = 100%,  reduce =100%
Ended Job = job_200911121319_0001
OK
31809
Time taken: 28.455 seconds
hive>

All of that is Hive translating our sql-like query into MapReduce jobs that are then farmed out to our cluster. Since we're using a single, small instance, and since we only have one data file, there isn't any parallelisation happening, and the whole thing runs quite slowly. But, in principle, we could have terabytes of data files in our S3 bucket, and be using more and much larger machines in our cluster. Under those circumstances, we should see major gains from using Hive.

FYI, the reason the first query didn't have this kind of output is that Hive is smart enough to figure out that no MapReduce trickery is necessary for this request - it can just read a few lines from the file to satisfy the query.

This has been a very quick and simple introduction to Hive on Amazon EMR. I hope you found it useful. I plan to go into more advanced, and hopefully more useful, territory in future posts.

PS: Don't forget to terminate your Hive cluster!

Setting up a remote git project

noreply@blogger.com (Anonymous) — Wed, 04 Nov 2009 11:24:00 +0000

I often find myself working on a quick hack, using a local git repository. Eventually, whatever I've hacked up becomes something I need to keep, and I want it in my remote git repository, with the local copy tracking the remote.

After setting this up manually several times, I finally got around to scripting it.

http://gist.github.com/225970

Assuming you've got a local git repo called 'myproject', and your current working directory is something like '/home/me/projects/myproject', then running this script will create a directory called 'myproject.git' on your remote git server, push your code to the remote repo and set the local copy to track it.

Don't forget to edit the script first to set the correct server name and main git directory, below which all your projects live.

The script assumes you're using SSH as the transport layer for git.

OCaml for the impatient - part 2, reading standard input

noreply@blogger.com (Anonymous) — Wed, 21 Oct 2009 18:06:00 +0000

Now that "Hello, world" is out of the way, let's look at the next step in writing our log processing script. We want to be able to read lines from standard input.
OCaml has an "input_line" function, which takes a channel as a parameter. Standard input is available without doing any extra work as the channel 'stdin'. So, to read a line of text from standard input, we just need to call;

input_line stdin

In OCaml, parameters to functions are not enclosed with braces. There are plenty of places you do need to use braces, but surrounding parameters is not one of them.
To do something useful with our line of text, we'll need to assign it to a variable. OCaml uses the "let" keyword for that, but we'll also need to declare a scope for our variable, using "in". So, the code we want is something like this;

let line = input_line stdin in
   ... a block of code ...

To read all the lines of text from standard input, until we run out, we'll need a loop of some kind. OCaml does allow us to write code in an imperative style, so we can just use a while loop. While loops are pretty basic in OCaml (and in functional languages in general), because you're meant to do much cleverer things with recursion.
Our loop will need to terminate when we run out of lines to read. The simplest way to do that in OCaml is to catch the "End_of_file" exception. I'm not a big fan of using exceptions for normal control flow, but we can live with it for now.
So, a simple program to read lines from standard input and echo them to standard output might look like this;

try
 while true do
   let line = input_line stdin in
   Printf.printf "%s\n" line
 done;
 None
with
 End_of_file -> None
;;

There are a few points to note here. The semi-colon after "done" is necessary to tell OCaml that it should evaluate everything before the semi-colon first, and then evaluate the stuff after it. Without the semi-colon, you'll get a syntax error. It needs to be ";" and not ";;" because we're not terminating a block of code.
We're using "End_of_file -> None" to discard the exception we get when "input_line" tries to read a line that isn't there. "None" is a bit like "nil" in Ruby or "undef" in Perl.
The "None" at the end of the block is required to keep the return type consistent. OCaml, like Perl or Ruby, returns whatever is the last thing evaluated in the block. OCaml requires that the try block return the same type of value as we will return if we catch an exception and end up in the with block. If you try running the code without the "None" before with, you'll get an error saying "This expression has type 'a option but is here used with type unit" (OCaml error messages are translated from French, so they're a little idiosyncratic).
The type "unit" is the empty type, like void in Java. Our with block is returning "None", so it's return type is unit, and the try block must return the same type.
If we change the with to say;

End_of_file -> "whatever"

Then the error becomes "This expression has type string but is here used with type 'a option". So, we can make it go away by replacing the earlier None with any string constant (like "hello" - try it).
The last thing we're going to do is to take our inline "Printf.printf" statement and turn it into a function call, so that we can do something more interesting with line later.
In OCaml, functions are values we can assign to variables. So, to define a function, we use the same let statement as we used to define line. Here's a function to print out our line;

let out = Printf.printf "%s\n";;

Notice that we terminated the statement without specifying what is supposed to be printed. If you type the code above into the interactive ocaml interpreter, you get this;

# let out = Printf.printf "%s\n";;
val out : string -> unit =

That's saying "the value out is a function which takes a single string and doesn't return anything". OCaml decided we were defining a function because we didn't specify all the arguments. If we had, it would have simply evaluated it and assigned the result to 'out'.
Now, we can simplify our program a little;

let out = Printf.printf "%s\n";;

try
  while true do
    let line = input_line stdin in
    out line
  done;
  None
with
  End_of_file -> None
;;

Try running the program like this "ls | ocaml foo.ml", or by compiling it as shown in part 1.
So far, we haven't done anything very useful overall, but we've covered reading from standard input and writing to standard output, looping over all the available input, assigning each line to a variable and calling a function with that variable.
In part 3, we'll actually do something!

OCaml for the impatient - part 1, "Hello, world!"

noreply@blogger.com (Anonymous) — Sat, 17 Oct 2009 16:55:00 +0000

After an inspiring presentation by Tom Stuart at LRUG earlier this week, I decided to have a go at learning a functional programming language. I picked OCaml, partly on recommendation from Tom, although he's since changed his advice to recommend Clojure because of it's "smart and comprehensive treatment of mutable state".

I found quite a few introductions to OCaml, but many of them seem quite theoretical and a bit dry. Plus, I'm really impatient. I tend to learn best by just diving in and trying to perform a real-world task. So, I decided to start by writing a basic filter to take apache access log lines and spit out some fields in CSV form.

I know this is not necessarily the best task for OCaml, and I'm sure my coding style is missing the point and doing many things the 'wrong' way. But, at least it's a practical way to get my hands dirty with the language.

First of all, let's do the traditional "Hello, world!" exercise;

1. Installing OCaml

Lots of instructions for various platforms here;

http://wiki.cocan.org/getting_started_with_ocaml

I used the INRIA binary on Mac OS X.

2. Hello World

A basic hello world program in OCaml. Using a text editor (preferably vim, but I've heard that other programs sort of work), put the following into a file called "hello.ml"


Printf.printf "Hello, world!\n";;

Some points to note;

1. The 'printf' function comes from the 'Printf' module. This module is available to all programs, so there's no need to do anything special to gain access to it.

2. ";;" denotes the end of a chunk of code.

You can run this code interactively using the "ocaml toplevel"



$ ocaml
Objective Caml version 3.10.1

# Printf.printf "Hello, world!\n";;  <--- type this and press enter    Hello, world! - : unit = ()   # <-------- Ctrl-D to exit  $

To run the program from the command-line there are several options.

Option 1


$ ocaml hello.ml

This is the easiest way, using the ocaml interpreter.

Option 2


$ ocamlc hello.ml -o hello
$ ./hello

This creates an executable "hello" file of OCaml bytecode, which should run on any machine with the OCaml bytecode interpreter, "ocamlrun" installed. It also creates a "hello.cmo" and "hello.cmi" file, which are OCaml object and interface files, respectively. This seems to be an intermediate step, since "hello" runs just fine if you delete them.

Option 3


$ ocamlopt hello.ml -o hello
$ ./hello

This time, "hello" is a compiled binary which can run standalone. There will also be the "hello.cmi" file, as well as "hello.cmx" (OCaml *native* object file), and "hello.o" (object file for your OS).

The interactive toplevel is great for noodling around, and option 1 is what I spend most of my time doing. Option 3 is what I will use if and when I write something that I want to use in production.

So, that's "Hello, world" out of the way. In the next part, we'll look at reading from standard input and writing to standard output.

OCaml for the impatient - part 2, reading standard input

Quick DB check script without loading Rails

noreply@blogger.com (Anonymous) — Sat, 30 May 2009 11:59:00 +0000

I use Zabbix to monitor my servers. It works by having a central zabbix server make calls to zabbix-agent processes on the monitored servers. The zabbix-agent can invoke a user-created script on the server and return whatever one-line output the script provides.

I needed a quick script to check the server to see if a set of database records exist, matching a given set of conditions. In this case, I'm checking to see if some log data representing yesterday's web traffic has been imported correctly, on the monitored server.

To start with, I used something like this;


#!/usr/bin/env ruby

ENV['RAILS_ENV'] = 'production'

require File.dirname(__FILE__) + '/../config/environment'

day = 1.day.ago.to_date
hostname = `hostname`.chomp

puts LogEntry.all(:conditions => {:day => day, :hostname => hostname}, :limit => 1).size

There could be a whole lot of log_entries records for yesterday, for this hostname. All I really want is to confirm that the importer script ran OK, so checking for the existence of one record that matches these criteria is sufficient and much faster than doing an aggregate query like a count.

This script outputs a '1' if everything is OK, and a '0' if not.

Hooking up a zabbix agent check to the script is easy enough, but whenever the server tries to call it, the check times out. The problem is that loading the entire Rails stack takes too long. Besides, it's overkill when all I want to do is run a single database query.

So, with help from here I changed it to this;


#!/usr/bin/env ruby

RAILS_ROOT = File.dirname(__FILE__) + '/..'
require 'rubygems'
require RAILS_ROOT + '/vendor/rails/activerecord/lib/active_record'
yaml_file = RAILS_ROOT + '/config/database.yml'
db = YAML::load(File.read(yaml_file))['production']
ActiveRecord::Base.establish_connection(db)

class LogEntry < ActiveRecord::Base; end 

hostname  = `hostname`.chomp
yesterday = 1.day.ago.to_date 

puts LogEntry.all(:conditions => {:day => day, :hostname => hostname}, :limit => 1).size

If you've got Rails installed as a gem on your server, you don't need to jump through all the 'RAILS_ROOT' hoops. But, I vendor everything, so I need to tell ruby where to find active_record.

Now the script is fast enough to respond to zabbix checks. If I needed it to be even faster, I could use lower-level ruby database code. But, this is quick enough, and importing ActiveRecord gives us the "1.day.ago" stuff too, keeping the code clean.

Setting up Ubuntu 9.04 for Ruby on Rails development

noreply@blogger.com (Anonymous) — Sat, 25 Apr 2009 11:24:00 +0000

I've just installed Ubuntu 9.04 "Jaunty Jackalope", and so far I'm really impressed. It's slick and fast, with some really nice little touches (e.g. my laptop wakes up when I open the lid - I used to have to hit the power button to wake it up from suspend).

Here are the steps I went through to set it up for Rails development;

Launch a terminal and become root
aptitude install build-essential

This gives you a c compiler and a bunch of other stuff essential to allow you to build other packages (hence the name)

aptitude install git-core

I use git for version control. Of course, there are a bunch of subversion and other packages available too. In Ubuntu 9.04, the version of git is 1.6.0.4, which is good enough for me.

aptitude install phpunit php5-curl

I do some PHP development too. You can ignore this and the php mysql package later on, if you don't.

aptitude install ttf-mscorefonts-installer

This makes fonts look a lot nicer in firefox.

aptitude install rails vim-rails ruby1.8-dev

This will install rails 2.1.0. A later version would be nicer, but all my projects have their own version of rails in the vendor directory, so this is good enough.

I use gvim as my editor. You can skip the vim-rails if you don't.

The "ruby1.8-dev" package is important. Without this, you'll get failures loading "mkmf" whenever you try to install gems.

aptitude install mysql-server-5.0 php5-mysql mysql-client-5.0 libmysqlclient15-dev

Ubuntu 9.04 comes with mysql 5.1, but a lot of my projects are still using 5.0, so I want to stick to that for now.

"libmysqlclient15-dev" is required when we want to switch to a native mysql library for ruby. If you don't do this, you'll get deprecation warnings because the non-native, pure ruby library is going to be dropped soon.

gem install ruby-debug hpricot mongrel

So far, those have been the only gems I needed to install to get to the point where my specs would run.

Happy installing, and please add a comment if you've got any suggestions.

Private web browsing via SOCKS proxy

noreply@blogger.com (Anonymous) — Mon, 06 Apr 2009 08:17:00 +0000

So, Big Brother is now requiring ISPs to keep a log of all your emails, internet phone calls and web browsing activity. This is, of course, to keep us safe from terrorists. How we stay safe from Big Brother is less clear.

Anyway, with this in mind, here is a quick guide to using a SOCKS proxy to keep your web browsing private by routing all your activity via an encrypted ssh tunnel to a server elsewhere.

When you browse the internet normally, e.g. to www.google.com, connections go like this;

Your machine -> www.google.com

Your ISP provides the connection in between, so they can log everything for those lovely government people. When you use a SOCKS proxy, it works like this;

Your machine (1)-> Your proxy (2)-> www.google.com

Connection (1) still goes via your ISP, but it's only a connection from your machine to your proxy, so they can't log where you're browsing to (in this case, www.google.com). All they can see is that you made a connection from your machine to your proxy. The connection is via SSH, so they have no way of knowing what information is travelling up and down the pipe.

Connection (2) is from your proxy to www.google.com Whichever ISP connects your proxy to the Internet can log your browsing activity. This is why it's important to have your proxy in a country where they won't do that.

Let's get started. You will need;

A server, located somewhere with a less Stalinist government. I use a virtual server in the US, which is quite ironic, really. This machine needs to be running an SSH server.

You need to be able to ssh onto this server from the machine you want to browse from. I've tried this from Mac OSX and Linux machines, where it's easy enough. If you're unlucky enough to be using Windows, it's probably possible to do this via cygwin.

Optionally, the "SwitchProxy" add-on for Firefox makes it easy to switch from browsing via your SSH tunnel to browsing normally.

First of all, you need to have things set up so that you can access your remote server va SSH. i.e. you can start a terminal window and type "ssh yourserver" and you've got a terminal session on your remote server. If you can't do this, you're screwed, and Big Brother knows about all the nasty websites you visit.

Now, open up a terminal and type;

ssh -CND 9999 yourserver

"-N" tells SSH that we don't want to run any commands on the remote box (such as a login shell).

"-D 9999" says "listen on local port 9999 and forward whatever connections come in to that port, using the appropriate protocol, via the remote server. So, if you use local port 9999 to try to connect to www.google.com on port 80, you actually end up connecting to your remote server via SSH and then *from there* to port 80 on www.google.com

Currently, the -D option to SSH only supports running as a SOCKS proxy, but since that's all we need, that's fine.

"-C" compresses all data sent over the SSH tunnel. Browsing via a remote proxy server is a bit slower than browsing direct, so compression may help speed things up a bit.

You will need to have a terminal window open, running this command, whenever you browse the net. There are clever ways to make this automatic, but I prefer to keep things simple.

Now we just need to tell our web browser to connect via our new SOCKS proxy, instead of going straight to the target website.

In Firefox, use the Edit menu and select;

Edit -> Preferences -> Advanced -> Network -> Settings

Configure the settings shown. i.e. a SOCKS host on localhost port 9999. You can use more or less any port you like, but any port number below 1024 can only be forwarded by root, so it's easier to use a higher number. Whatever number you used in the "ssh -ND ..." command is the port you need to put here.

For any sites where you don't want to use the SOCKS proxy, add them to the "No proxy for" section. For example, if you use BBC iPlayer, it won't work via a US host, so you need to not use the proxy for bbc.co.uk

There are similar options for other browsers such as Safari and, if you really have to use it, Internet Explorer.

Once you've configured your proxy settings, you're good to go. Try browsing to a website and it should be going via your SSH tunnel. To confirm, try closing the terminal window and refreshing the page - you should see an error saying, "Proxy Server Refused Connection". Restart the tunnel and refresh again and it should work.

The SwitchProxy add-on for Firefox makes it really easy to switch proxy configurations - hence the name - via a new entry in your "Tools" menu.

Happy private web-browsing.

Redundancy using Nginx for failover

noreply@blogger.com (Anonymous) — Sat, 28 Mar 2009 13:44:00 +0000

I've got a website running on two servers, for redundancy, with an nginx load-balancer in front of them. If one server goes down, all requests will be sent to the other using this stanza in the nginx config;


upstream webservers {
  server 1.2.3.4;  # web1
  server 2.3.4.5;  # web2
}

This uses a round-robin system to send alternate requests to alternate servers. That's great for spreading the load around, but not exactly what I want.

The webservers are not heavily loaded, but they do run some quite complex reports against the database, the results of which are cached on the webserver's local filesystem. The cache is expired periodically.

Using a round-robin algorithm, we end up calculating each report twice per period - i.e. once on each webserver. What I really want is for all requests to go to web1, so that it's using its cache as efficiently as possible. web2 should only start receiving traffic if web1 goes down.

I don't know of a way to make nginx do exactly this, but you can use the 'weight' parameter to fake it, like this;


upstream webservers {
  server 1.2.3.4   weight=100000;  # web1
  server 2.3.4.5;  # web2
}

Now, only one in every 100,000 web requests will go to web2, so the cache on web1 is being utilised to a much greater extent. If web1 goes down, web2 gets all the traffic.

Redundancy plus efficient caching.

Rewired State

noreply@blogger.com (Anonymous) — Sun, 08 Mar 2009 20:04:00 +0000

I went to the Rewired State hack day yesterday.

A brilliant time was had by all. Photos here, video here.

Our hack was What's On Their Minds? (if that URL doesn't work, try this link).

Converting iTunes playlists with ruby

noreply@blogger.com (Anonymous) — Sun, 30 Nov 2008 11:48:00 +0000

Just posting this here in case anyone else has a similar setup.

I've got an xbox running XBMC hooked up to my home stereo system, but all my music is stored and organised in iTunes. The folder containing the music is shared so the xbox can see it, so all my music is available in XBMC with no problems. The one thing I was missing is all my iTunes playlists.

Along with virtually every other music player, XBMC uses a standard .m3u text file format to define playlists. I think the '#EXT...' stuff is specific to XBMC, but I'm not sure.

Needless to say, iTunes is not compatible with this at all. But, it is possible to export a playlist from iTunes, in its own bizarre XML format

One quick ruby script later, and I can convert the iTunes XML to M3U, including a remapping of file locations because iTunes has a different name for the root music folder (in fact, it has 2 different names for it because I moved the folder once, but the older playlists seem to still use the original location, somehow. Hence the alternatives in the ITUNES_ROOT regexp).

Finally, a noddy bash script to convert all the .xml playlist files in a directory to .m3u and transfer them to the xbox using ncftp

I still have to right-click on my playlists in iTunes and choose 'Export Song List' for each one, but that's not too arduous. I could probably write an AppleScript or something to do that for me, but in my case it's not worth the extra effort.

If you find any of this stuff useful, please drop a note in the comments.

Cheers

David

Client collaboration via github

noreply@blogger.com (Anonymous) — Wed, 12 Nov 2008 21:20:00 +0000

UPDATE: The post-receive URL on github has now been moved to a new "Services" tab on the project edit page.

It can be a hassle setting up and maintaining development environments
for clients; particularly if they're not very technical, or if they
use Windows.

I usually set up a separate VPS for each client's project as a
development/staging server, and it would be great if they could
just edit the source files on there. Then they could play around
and see the effect of their changes in realtime. The trouble is
that this requires some kind of remote editing or file-syncing setup
(e.g. SSH+vi), which can be troublesome to set up.

So, I'm trying out a combination of github's in-browser source
editing, plus a post-receive URL pointing to the client's development
VPS. Here's how it should work.

I set up the client's project as a github repository, and add the client as a collaborator.

The client makes changes to the view/css files using github's in-browser editor.

When they commit their changes, github posts to a listener on the client's development VPS (using a post-receive hook)

The listener on the VPS pulls the client's changes

The client views their development site to see the changes in (nearly) realtime

This is surprisingly easy to set up. I'll walk you through setting this up for a dummy "Hello, world" rails application. The ingredients we'll be using are;

github - a free account will do fine, for this demo

A development server - github will be posting to a URL on this server, so you need to be able to access it from the internet

A listener to pull changes from github, whenever there's a new commit. I'm going to use a very simple sinatra application, running via a command-line mongrel instance.

An application to act as our super-important client project

Setup a simple rails project as a github repository. You can just use a
bare "rails foobar" project - no need to set up databases.

Now, ssh to your development server (the one github will be posting to when your client makes changes). Setup a clone of the github repo.

For this demo, use the "public" clone url (the one that starts with "git://github.com").

You can use this technique for a private repo - otherwise it would be a bit pointless - but that takes a bit more SSH setup than I want to get into in this post.

I'm using "screen" so that I can start the demo application running
in a terminal and have it keep running after I log out. If you don't
know about the unix screen command, go learn about it right now.
After SSH it's one of the most useful tools to have in your toolbox.

screen
cd dstest
script/server
[Ctrl-A D to detach from screen]

So, now the application is running, and you can open a browser window to;

http://your.development.server:3000/

and see the "Welcome to Ruby on Rails" page.

Now for the listener which will pull our client's changes.

If you haven't already got them, you will need sinatra and mongrel on your development server;

sudo gem install sinatra
sudo gem install mongrel

Now, put the following into a file called "gitmon.rb"

require 'rubygems'
require 'sinatra'
post '/update_site' do
dir = '/home/david/dstest'
system "cd #{dir}; git pull"
"Site updated\n"
end

You will need to change the "dir" line to
be the full path to your rails project.

Now use screen again to start a listener that will continue to run after you log out;

screen
ruby gitmon.rb
[Ctrl-A D to detach from screen]

The listener will start up on Sinatra's
default port, 4567. You can test it by
visiting

http://your.development.server:4567

You should see "Sinatra doesn't know this diddy."

Now we just need to tell github to post to the listener whenever it gets new commits.

Login to your github account and go to
your repository's edit page.

Setup the "Post-receive URL" as;

http://your.development.server:4567/update_site

That should do it.

Now, edit a file (e.g. public/index.html) using the github in-browser editor (if you're not logged-in, you won't see the edit link). Commit your change.

Refresh the browser window you've got open to;

http://your.development.server:3000/

and you should see your changes (it can take a few seconds before your site updates).

This is just the tip of the iceberg, and there are a lot of ways this technique could be extended (e.g. hooks to run migrations after a "git pull" on the development server, or getting the sinatra listener to post to Campfire whenever a client commits something).

Also, there is a lot of information in the POST from github, which we're just ignoring here.

I hope you find this useful. I'll be setting it up for one of my clients very soon, so I'm keen to see how it works out.

Setting up Ubuntu 8.10 for Ruby on Rails development

noreply@blogger.com (Anonymous) — Sat, 01 Nov 2008 14:07:00 +0000

Here's how I've set up my laptop, running Ubuntu 8.10 (Intrepid Ibex) for Ruby on Rails development. You'll need to run the following commands from the command-line as root. These commands worked for me, but I take no responsibility if you trash your box by doing this ;)

There is some duplication here, because I sometimes setup servers with different sub-sets of the software I use on my development laptop. Don't worry about this. It does no harm if you tell apt-get or aptitude to install the same package twice - it will just ignore you the second time.

Not all of the gems here are required for RoR development, but I use cucumber for integration testing, and it has some pre-requisites.



# General/Pre-requisite packages
aptitude install \
  build-essential \
  screen \
  subversion \
  mysql-client \
  telnet \
  meld \
  vim \
  vim-gnome \
  exuberant-ctags \
  tk8.5 \
  apache2-prefork-dev \
  rcov

# mysql server
# use apt-get to avoid installing exim4
DEBIAN_FRONTEND=noninteractive apt-get install --assume-yes \
  mysql-server mysql-client \
  libmysqlclient15-dev libmysql-ruby1.8


# git
mkdir gitcore
cd gitcore
wget http://kernel.org/pub/software/scm/git/git-1.6.0.3.tar.gz
apt-get build-dep git-core --assume-yes
tar xzvf git-1.6.0.3.tar.gz
cd git-1.6.0.3/
./configure
make
make install
cd

# Ruby
aptitude --assume-yes install ruby1.8-dev ruby1.8 ri1.8 rdoc1.8 \
  irb1.8 libreadline-ruby1.8 libruby1.8 libopenssl-ruby
ln -s /usr/bin/ruby1.8 /usr/local/bin/ruby;
ln -s /usr/bin/ri1.8 /usr/local/bin/ri;
ln -s /usr/bin/rdoc1.8 /usr/local/bin/rdoc;
ln -s /usr/bin/irb1.8 /usr/local/bin/irb;
ln -s /usr/local/bin/ruby /usr/bin/ruby

# Rubygems. You REALLY don't want to let aptitude install rubygems.
wget http://rubyforge.org/frs/download.php/45905/rubygems-1.3.1.tgz
tar xzf rubygems-1.3.1.tgz
cd rubygems-1.3.1
ruby setup.rb
ln -s /usr/bin/gem1.8 /usr/bin/gem
for g in rails rake capistrano capistrano-ext hpricot treetop ruby-debug term-ansicolor mongrel cheat passenger annotate-models rak; do gem install $g; done

Have fun.

David

Finding files in vim

noreply@blogger.com (Anonymous) — Wed, 23 Jul 2008 09:10:00 +0000

Some of my vim-challenged colleagues, when they try switching to vim, miss the Cmd-T function that lets them type parts of a filename and open the corresponding file.

After a quick search of the tips on vim.org, it turns out that someone has implemented exactly this function;

http://www.vim.org/tips/tip.php?tip_id=1432

This has the drawback that it will find blah.rb.svn-base when you look for blah.rb. So, I've tweaked it a little to skip any svn files, and also not to look in the 'vendor' directory, so it doesn't find rails source files.

http://gist.github.com/1625

Have fun

Automatically "git rm" all deleted files

noreply@blogger.com (Anonymous) — Tue, 22 Jul 2008 13:40:00 +0000

Love git. Hate having to "git rm" individual files whenever I delete a bunch of stuff (yes, I know I can "git rm foo/*", but what if I didn't remove *all* the files in foo/ ?).

So, a quick ruby script: gitrmall.rb

Disabling tests in Shoulda

noreply@blogger.com (Anonymous) — Tue, 15 Jul 2008 22:31:00 +0000

I'm using Shoulda on a client project, although I prefer rspec myself.

One of the things I miss from rspec is the ability to turn off a test, temporarily, by changing;


it "should do something"

...to;


xit "should do something"

Tests which are disabled in this way show up as "Example disabled: it should do something", when the specs are executed.

Using this with autotest and a couple of quick vim macros, my workflow goes something like this;

1. See that there is a failure in a particular test
2. Position the cursor in that test and hit ",ofs" this triggers my macro to disable all specs except the current one, and save the file so that autotest runs it again.
3. Hack, hack, hack to fix the bug, or develop the feature that the failing test relates to.
4. Hit ",ons" to turn all the specs back on and save the file again.

In Shoulda, renaming a test from;


should "do something"

...to;


xshould "do something"

Gives you a nice NoMethodError.

But, this is ruby, so we just add this to our test/test_helper.rb



class ActiveSupport::TestCase
  def self.xshould(name, &block)
    puts "disabled test: #{name}"
  end
end

Now, we get pretty much the same behaviour as with 'xit' in rspec.

For any vim users in the audience, here are the macros (actually, abbreviations) I use to turn tests, specs and shouldas on and off. Just add to your .gvimrc (or .vimrc) file;


"rspec
map ,ofs :%s/ it / xit ''?xitx:w
map ,ons :%s/ xit / it /'':w

"shoulda
map ,ofh :%s/ should / xshould ''?xshouldx:w
map ,onh :%s/ xshould / should /'':w

"test/unit
map ,oft :%s/def test/def xtest''?xtestx:w
map ,ont :%s/def xtest/def test/'':w

ActiveRecord commit timing

noreply@blogger.com (Anonymous) — Tue, 01 Jul 2008 22:52:00 +0000

I'm building an application where we're using a message queue.

New objects add their IDs to a beanstalkd queue via an after_create callback. Some daemons monitor beanstalk, and grab objects to process, as soon as they hit the queue.

We were getting errors from the daemons, saying "record with ID 'N' not found", but when we looked for object N via script/console, or the database, there it was.

It turns out that the after_create callback (along with all the other object creation callbacks) occurs *inside* a transaction. So, this is what was happening;

There are a couple of ways around this;

Call "self.class.connection.commit_db_transaction" in the model, after the after_create method. This works, but it smells really bad.

A nicer way is to add an "after_commit" callback, like this

Pat Allan has some modifications to this, to make it play nicer with Rails 2.0 and 2.1

Or, there is a version here which handles nested transactions.

Many thanks to Pat for showing me the after_commit stuff.

Using non-standard primary keys with ActiveRecord in Ruby on Rails

noreply@blogger.com (Anonymous) — Sat, 07 Jun 2008 10:49:00 +0000

A Rails application I'm currently working on has a 'User' model. For the purposes of this app. a 'user' is uniquely identified by their mobile phone number - a user must have a mobile phone number, and may only have a single one. A second mobile phone number must be a second user.

I'm going to need the mobile phone number in other models - e.g. "User has_many :messages", and I'd like to have a "mobile_phone_number" attribute in my Message model, so that I can say "message.mobile_phone_number" without having to join the users table all the time, like this; "message.user.mobile_phone_number".

I can get what I want if the mobile_phone_number is the primary key of the User model. Let's try it.

Here is the migration (this is specific to mysql);



class CreateUsers < ActiveRecord::Migration
  def self.up
    create_table :users, :id => false do |t|
      t.integer :mobile_phone_number
    end
    execute "alter table users modify column mobile_phone_number bigint unsigned primary key"
  end

  def self.down
    drop_table :users
  end
end

(The alter table statement is a bit of a hack. Another way to achieve the same result would be to use the mysql_bigint gem)

Here is the model;



class User < ActiveRecord::Base
  set_primary_key :mobile_phone_number
end

Looks OK, but this is what happens when you try to create a User;



>> u = User.create :mobile_phone_number => 447123456789
=> #<User mobile_phone_number: 0>

Where did that '0' come from, and what happened to the mobile_phone_number?

A quick look in the logs shows this;



  WARNING: Can't mass-assign these protected attributes: mobile_phone_number
  SQL (0.000146)   BEGIN
  User Create (0.000222)   INSERT INTO `users` VALUES(DEFAULT)
  SQL (0.000422)   COMMIT

So, Rails is not sending the primary key column value in the SQL create statement. This would make sense if, as usual, our primary key was an autoincrement column. The database would supply that value for us, so we don't want to send it. But, in our case, we want to set the primary key ourselves, so this behaviour is wrong. (The 0 comes from the default value of the column)

You can try using 'attr_accessible' to allow you to assign to the mobile_phone_number, but it doesn't work (presumably because it's the primary key that we're trying to assign to).

We need some way to override the standard Rails primary key behaviour so it does what we want. That sounds a lot like the composite_primary_keys gem.

So, after installing the gem (in my case, after installing Rick Olson's 'gems' plugin so that I can put the composite_primary_keys gem in my project's vendor directory), here is the updated model;



require 'composite_primary_keys'
class User < ActiveRecord::Base
  set_primary_keys :mobile_phone_number
end

Note that 'set_primary_key' has changed to 'set_primary_keys'.
Now, we get this;



>> u = User.create :mobile_phone_number => 447123456789
=> #<User mobile_phone_number: 447123456789>

Much better. Even though we're not using a composite_primary_key, the gem is overriding the default Rails behaviour so that the key we want is being set by the create statement.

If you know a more elegant way to do this, please let me know.

Gotcha: composite_primary_keys gem

noreply@blogger.com (Anonymous) — Sat, 26 Apr 2008 17:40:00 +0000

Dr. Nic's composite_primary_keys gem is incredibly helpful if you're writing rails code against a legacy database, or in any other situation where you can't or won't follow the rails convention of having a single field as your model's primary key.
After doing the usual;

sudo gem install composite_primary_keys

...and requiring it in you environment.rb file, you can define a model like this;

class Membership < ActiveRecord::Base
set_primary_keys :user_id, :group_id
...
end

Brilliant.
But, there is one subtlety you need to be aware of. If your model mixes in any modules, you might end up writing something like this;

class Membership < ActiveRecord::Base
include MyAwesomeModule
set_primary_keys :user_id, :group_id
...
end

Looks fine, doesn't it? Unfortunately, your tests will now break with lots of errors like this;

ActiveRecord::StatementInvalid in 'Membership should foobar'
Mysql::Error: Column count doesn't match value count at row 1: INSERT INTO memberships (`user_id`, `group_id`, ... , id) VALUES (...whatever...)
/Users/david/myproj/vendor/composite_primary_keys-0.9.90/lib/composite_primary_keys/base.rb:106:in `create_without_callbacks'

See that last id, just before VALUES? It shouldn't be there. Something weird is going on, because composite_primary_keys doesn't seem to be doing its thing.
The solution is to make sure the call to "set_primary_keys" is the first thing executed in your model;

class Membership < ActiveRecord::Base
set_primary_keys :user_id, :group_id
include MyAwesomeModule
...
end

Remember this, and everything works fine. Forget it, and you'll have lots of fun with the debugger.
If I were braver, smarter and kinder, I would dive headfirst into the code and try to fix it. But, I've got work to do.

Testing file uploads with RSpec on Rails

noreply@blogger.com (Anonymous) — Sun, 13 Apr 2008 16:40:00 +0000

I was writing specs for a controller that processes uploaded images, and came up with this way of mocking a file upload control. So, I thought I'd share it.
Say you have a WibbleController which can replace the image belonging to a particular wibble. In your wibble/edit view you probably have something like this;


<% form_for(@wibble, :html => { :multipart => true }) do |f| %>
...
Replace with new image:
<%= f.file_field :new_file %>
...
<% end -%>

In your controller, there will be something that reads the streamed file data that the browser posts when you choose and submit a file.
In your spec, you can create an ActionController::UploadedStringIO object (which is what your controller will see), but it won't have access to the file data. So, we need to monkey patch it a bit.
The two methods we need to override are "read", which will return the file contents, and "size" (guess what that does). So, define a method in your spec file like so;


def mock_uploader(file, type = 'image/png')
filename = "%s/%s" % [ File.dirname(__FILE__), file ]
uploader = ActionController::UploadedStringIO.new
uploader.original_path = filename
uploader.content_type = type
def uploader.read
  File.read(original_path)
end
def uploader.size
  File.stat(original_path).size
end
uploader
end

Then, you could write a spec like this ('foo.png' should be a suitable image file in your spec/controllers directory);


it "should upload an image" do
 Image.delete_all
 uploader = mock_uploader 'foo.png'
 post :update, {
   :id => @object.id,
   :wibble => { :image_file => uploader }
 }
 response.should be_success
 Image.count.should == 1
 i = Image.find(:first)
 i.filename.should == uploader.original_path
 i.contents.length.should == uploader.size
end

There may well be a better way to do this, in which case please let me know via the comments.

Auto-rotate Rails log files

noreply@blogger.com (Anonymous) — Tue, 08 Apr 2008 12:17:00 +0000

I've been searching for a way to make my rails apps rotate their log files automatically, the way apache does it. In my case, I get logs like this;

access_log.20080407
error_log.20080407

i.e. a log file per day, with the date as the suffix of the filename.
When a new day rolls around, a new file is started. Simple.
Hunting around on the web, most people seem to be writing a /etc/logrotate.conf file to achieve something like this. This has a couple of problems;

After switching to a new log file, all your mongrel/fastcgi processes need to be restarted. Otherwise, you get a segfault when they try to log to a file that's not there anymore.
The /etc/logrotate.conf file is yet another file whose deployment needs to be managed (potentially along with a cron entry to kick off logrotate at the appropriate time).

So, I kept digging and eventually found this post
So, Rails can be told to log to a pipe, just like apache. I'm not familiar with cronolog, so this is what I ended up putting in my config/environments/production.rb file;

config.active_record.colorize_logging = false

log_pipe = IO.popen("/usr/sbin/rotatelogs #{RAILS_ROOT}/log/production_log.%Y%m%d 86400", 'a')

config.logger = Logger.new(log_pipe)

[the second entry is supposed to be a single line from 'log_pipe = ' to ')' ]

Turn off colorized logs, because they irritate me.
Open a pipe to rotatelogs (I use 'a' for 'append', instead of 'w' for 'write', but it doesn't seem to make any difference on my system - the file is appended to when I restart rails, whichever I choose).
Point the rails logger to the pipe.

This seems to do exactly what I want, with minimal mucking about.

Cloning Ubuntu Gutsy virtual machines on OSX with VMWare Fusion

noreply@blogger.com (Anonymous) — Tue, 18 Mar 2008 13:59:00 +0000

I use VMWare Fusion on my Mac to create virtual machines so that I can see exactly how my code will behave when it's deployed on Ubuntu Gutsy server.
I want to setup a few servers, so that I can experiment with different load-balancing solutions. So, I took a virtual machine that I had already configured, and copied it using Finder. When I launched the VM, it asked me if I had moved it or copied it. I clicked "I copied it", and the machine carried on booting.

Unfortunately, the networking was broken - the machine kept insisting that eth0 was not present. After a bit of research, I found this thread on the VMWare forums.
The problem is that VMWare assigns a new MAC address to the machine when you copy it - which is sensible, since you can't have two instances of the same MAC address on the same network. But, Linux stores the MAC address of the card when the network interface is configured, and VMWare has no way of telling Linux about the new MAC address. So, Linux thinks it's original network card has disappeared, and that a new card with a different MAC address has been installed, and gets very confused.
The solution, for Ubuntu 7.10 (Gutsy Gibbon), is to edit this file as root;

/etc/udev/rules.d/70-persistent-net.rules

Here is the file before editing;

# This file was automatically generated by the /lib/udev/write_net_rules
# program, probably run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single line.

# PCI device 0x1022:0x2000 (pcnet32)
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:0c:29:ba:e7:7a", NAME="eth0"

# PCI device 0x1022:0x2000 (pcnet32)
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:0c:29:e4:ee:d4", NAME="eth1"

(The SUBSYSTEM ... NAME stuff is all on a single line, but the blog theme causes it to wrap)
The red MAC address is the original one, the same address as in the original virtual machine we created the copy from. As far as the Linux server is concerned, this network card has just vanished, because it can't see any card installed with that MAC address.
The green address is the new MAC address that VMWare created. The Linux server thinks it has a network card plugged in, with this MAC address (because VMWare is telling it so), but it's not configured.
All we need to do is to edit the file so that the card which is installed is known as "eth0", and get rid of the old MAC address that our original source VM is still using. i.e. we delete the eth0 line and rename eth1 to eth0.
So, after editing, the file looks like this;

# This file was automatically generated by the /lib/udev/write_net_rules
# program, probably run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single line.

# PCI device 0x1022:0x2000 (pcnet32)
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:0c:29:e4:ee:d4", NAME="eth0"

Now, reboot the VM (/etc/init.d/networking restart is not enough - you need to reboot), and all should be fine. Rinse and repeat for as many instances of your server as you need.