Ruby-coloured glasses

Backporting "in_array" to older versions of should'a "ensure_inclusion_of"

2013-05-21T17:33:00.000+10:00

Here's another shoulda backport I added recently. If you're still stuck using a legacy Rails system, this backport will let you use "in_array" in the "ensures_inclusion_of" Matcher.

Save it into something like: config/initializers/shoulda_monkeypatches.rb, then use it like this:

   should ensure_inclusion_of(:widget_status).in_array(Widget::VALID_STATUSES).allow_blank.with_message(:is_invalid).use_integer_test_value

  # backport the "in_array" method for the ensure_inclusion_of matcher
  # While we're at it, add allow_blank and allow_nil too
  module Shoulda # :nodoc:
    module ActiveRecord # :nodoc:
      module Matchers
        class EnsureInclusionOfMatcher
          ARBITRARY_OUTSIDE_STRING = 'shouldamatchersteststring'
          ARBITRARY_OUTSIDE_INT = -999999999

          # to initialize the options
          def initialize(attribute)
            super(attribute)
            @options = {}
          end

          # add the method we want to allow us to pass in arrays instead of
          # just ranges
          def in_array(array)
            @array = array
            self
          end          

          # might as well also add the allow_blank and allow_nil methods too
          def allow_blank(allow_blank = true)
            @options[:allow_blank] = allow_blank
            self
          end

          def allow_nil(allow_nil = true)
            @options[:allow_nil] = allow_nil
            self
          end

          # This is a method of my own addition to point out that the
          # test-value must be an Int, not a String... because a string can
          # evaluate to 0 which is a valid Int... which will make the test
          # pass where it shouldn't :P
          def use_integer_test_value(only_integer = true)
            @options[:use_integer_test_value] = only_integer
            self
          end

          # override description so it doesn't just try to inspect the range
          def description
            "ensure inclusion of #{@attribute} in #{inspect_message}"
          end

          # override the matches method to allow arrays as well as ranges
          def matches?(subject)
            super(subject)

            if @range
              @low_message ||= :inclusion
              @high_message ||= :inclusion

              disallows_lower_value &&
                allows_minimum_value &&
                disallows_higher_value &&
                allows_maximum_value
            elsif @array
              if allows_all_values_in_array? && allows_blank_value? && allows_nil_value? && disallows_value_outside_of_array?
                true
              else
                @failure_message_for_should = "#{@array} doesn't match array in validation"
                false
              end
            end
          end

 
        private

          # provide the message-inspect method to use either array or range
          def inspect_message
            @range.nil? ? @array.inspect : @range.inspect
          end          

          # array helper methods
          def allows_all_values_in_array?
            @array.all? do |value|
              allows_value_of(value, @low_message)
            end
          end

          def disallows_value_outside_of_array?
            disallows_value_of(value_outside_of_array)
          end

          def value_outside_of_array
            test_val = @options[:use_integer_test_value] ? ARBITRARY_OUTSIDE_INT : ARBITRARY_OUTSIDE_STRING
            if @array.include?(test_val)
              raise CouldNotDetermineValueOutsideOfArray
            else
              test_val
            end
          end

          # blank and nil helper methods

          def allows_blank_value?
            if @options.key?(:allow_blank)
              blank_values = ['', ' ', "\n", "\r", "\t", "\f"]
              @options[:allow_blank] == blank_values.all? { |value| allows_value_of(value) }
            else
              true
            end
          end

          def allows_nil_value?
            if  @options.key?(:allow_nil)
              @options[:allow_nil] == allows_value_of(nil)
            else
              true
            end
          end         




        end

      end
    end
  end

Enforced computer-free time

2013-04-30T22:50:00.000+10:00

Gah - I've been computer-free for the past week, and not by choice.

My ageing macbook suddenly started to switch off... at random. This was ok for a while... but when it took four times for it to even get through the boot process - I had to take it into the Apple shop. meanwhile I finally set about getting a new desktop, and work gave me a new macbook to try and use.

I couldn't get the monitor to work on the desktop, and the macbook is too new to run the current ubuntu LTS (though I somehow managed to get 9.10 to install... but it won't find the ethernet port)...

I've finally got the desktop working (I was using the wrong video connection) but I still can't get mail to work properly (the joys of mail on a linux machine...), and my macbook is back from the shop... but the ethernet isn't working.

soooo.... still no email and it's been a week. But at least I have google now.

You realise just how dependant on something you've become when you suddenly don't have it anymore.

You also find just how stuck you can be. How do you find the location of a good internet shop if you don't have access to google to look one up? There used to be a directory-services phone number, but I have no idea what that is anymore, and of course I don't have a copy of the yellow pages anymore (who bothers with those when you have google)?

Ubuntu without the Unity!

2013-04-24T22:24:00.001+10:00

For some time now my ageing (and now quite flaky) Macbook Pro has been running Lucid Lynx (an older version of Ubuntu). I've been biding my time and avoiding upgrading since, well, April 2010, I guess.

For most of the time it was because upgrading your only machine that you rely on for 100% of your income, to a potentially unstable, non long-term release version is generally a bad idea; but then, when I went to a new employer, I tried a new install of the new long term release... and slammed headlong into the weirdness that is the Unity Interface.

To sum up my experience: when I buy a desktop, I want it to *be* a desktop... not a iPad.

Now - maybe I'm just being backwards and should embrace the new unified interface which I'm told is meant to be amazing... if a little bloated.. and sometimes corrupts your file system if something goes wrong... but this is still my only machine (for now) and I need to be sure it's ok. I don't want to upgrade to Unity.

Of course, there is now a new long-term-release version, and sadly, the old one is now no longer supported, so I'm feeling a bit like a sprite in an old platform game, being relentlessly pushed towards the future by the scrolling screen...

Thankfully, I've malingered just long enough for the Ubuntu Gnome project to have been officially blessed by . it's not an LTR version yet, but it will be.

Until then - it's also possible to upgrade to the LTR but keep gnome. I've been putting it off because I'm ultra-busy and kept finding other, much more important things to do than figuring out how to do it myself... but I've just found a neat article on how to change your ubuntu interface to gnome.

Just what the doctor ordered

Huge medieval camping event - come and join in!

2013-04-12T18:47:00.000+10:00

The reason it's been so quiet around here this month is that I've been crazy busy in my main hobby due to our major event for the year (run by our local group).

Rowany Festival is the largest Medieval re-enactment event in the Southern Hemisphere. I've heard it described as "kind of like Burning Man but with a medieval theme" - we prepare all year for this!

This year 800 of us all got together at Peats Ridge and camped medieval-style, fought wars, sang, danced, drank a *lot* of booze (some of it made by me) and generally had a great time...

Needless to say not a lot of Ruby got written... but that'll pick up again now I've got most of my washing done and camp kit packed away again.

If you'd like to join me on this stuff - I'm the president the NSW branch of the SCA, known as Rowany. The Australian kingdom is called Lochac (and also covers NZ too).

The whole Society is huge: 100,000 worldwide, and we all run regular events (camping roughly once a quarter, and feasts/tourneys once a month and just general hanging-out and learning fun stuff every week) and it's a great deal of fun.

Come along to our casual fighter-practice which is held on Tuesday nights at Petersham Town Hall on Crystal St (near to Petersham Station). We regularly get 30-50 people every week and they're running beginners fighting classes right now*. We also have regular singing, Calligraphy fibre-craft and costuming days every month.

* its $5 a night for non-members, or you can join for a year for $30 - but you have to do your membership online here (we're not allowed to take payment directly for that)

Picture Credit: Enough Wealth: Rowany Festival

Link: Why Yammer believes the traditional engineering organizational structure is dead

2013-03-05T17:06:00.002+11:00

This is a great article describing the way that Yammer has set up its organisational structure to be nimble and effective. Why Yammer believes the traditional engineering organizational structure is dead

Highlights for me:

Yammer’s biggest rule of thumb is “2 to 10 people, 2 to 10 weeks,” ... If you employ the “2 to 10″ rule, it’ll also force you to release often, test your assumptions, and not over-invest in mistakes.

and

While everybody knows how expensive context switching is, it’s staggering that nobody builds that into their organization as a constraint. With total focus, you build one thing, ship it, and then are able to move onto something else.

rubyconf was awesome

2013-02-24T20:04:00.001+11:00

that is all...

Quick and dirty facebook feed parsing

2013-02-14T17:14:00.000+11:00

So, there's this big discussion going on for my hobby group at the moment, and the main discussion has been going on in facebook - mainly because that's where I was first asked to set up a discussion and it took all of five minutes to get the page up and running.

However - now that discussions are progressing, there are a number of people *outside* the facebookiverse who have raised quite reasonable objections to the discussion happening there. not everyone is on facebook, not everyone *wants* to be on facebook, and to be honest, a facebook group kinda sucks for searching and archiving really important discussions.

thus it has been requested that I copy all the posts and comments to Someplace Else, to make them available for more general consumption.

At first I balked at this request - 24 posts and 250 comments to be individually copied/pasted??? Who has that kind of time?

Of course when I actually sat down to think about the problem seriously, it took far less time than I though to solve it. So here's what I did, including the quick-and-dirty ruby script that will massage the facebook feed into something that resembles readable format. It aint pretty - but it'll pass for government work.

Step 1: get the feed from the API

I'll assume you are actually a member of the group that you're after. You will need to be.

You need to go to your group and get the group's ID from the URL.

The facebook API page is here: Facebook Graph API explorer

First you need to create an access token to get the data out of facebook. This is essentially the same as doing one of those "allow application to access my data" things that you click on when you add a new app. In this stage, you need to allow the Graph-API application to access *your* group-data, to prove that you have access to he feed of the group - and to allow it to fetch out all the posts for you.

click the "Get access token" button
Select the "user_groups" checkbox
click the new "get access token" button
follow any prompts (if this is the first time using this API, you'll get the "allow access for this application" process - but it may not happen for subsequent attempts

You should now have a long encoded token in the box at the top of the page.

Next up we need to tell it what feed to use. There's a drop-down labelled "GET" which you should leave as-is. In the text-box next to that, type in the ID of the group in a URL-format like this: /1234567890?fields=feed and then click "Submit". The "fields=feed" tells the API to actually go and fetch the feed of posts and comments.

At this stage, you should be able to see a huge hash full of posts and comments in the box to the right hand side of the screen. Copy and paste that into a file.

Step 2: massage it into shape

Now you've got your feed data, you just need to play with it and spit it out into a nicer format. In our case, I decided to go for just a rough html format that showed what the posts were, what comments were attached, and who said what. My script is posted below - which can serve as a starting point for whatever you'd like to see done.

This script accepts the input filename and an optional output filename (or it just jams '.html' on the end of the input filename). It'll generate a really rough-and-ready html file that contains the posts and comments (with names and datetimes) plus some of the links (if present).

Enjoy...

#!/usr/bin/env ruby

DATE_FORMAT = "%H:%M:%S %d-%m-%Y"

class Object
  def blank?
    self.nil? || (self.respond_to?(:empty?) && self.empty?)
  end
  def present?
    !self.blank?
  end
end

new_file_name = nil

# if they've passed in the filename, use it
if ARGV && ARGV.length >= 1
  file_name = ARGV[0]
  if ARGV.length > 2
    new_file_name = ARGV[1]
  end
end
if file_name.blank?
  p "usage: facebookfeed <file_name> [<outfile_name>]"
  exit(1)
end
p "got file_name of: #{file_name}"

unless File.exists?(file_name)
  p "file: #{file_name} does not exist"
  exit(1)
end

# munge up an html filename for the output file
new_file_name ||= file_name.split('.').first + '.html'


if File.exists?(new_file_name)
  p "output file: #{new_file_name} already exists, please supply another"
  p "usage: facebookfeed <file_name> [<outfile_name>]"
  exit(1)
end

# parse json in file into ruby - preferably a hash
require 'rubygems'
require 'json'
require 'date'
facebook_hash = JSON.parse(IO.read(file_name))

feed_data = facebook_hash["feed"]["data"]



if feed_data.present?
  File.open(new_file_name,'w') do |outfile|
    puts "parsing #{feed_data.count} posts"
    sum = 0
    feed_data.each {|post| sum += post["comments"]["count"].to_i }
    puts "with: #{sum} total comments"

    feed_name = feed_data.first["to"]["data"]["name"]

    # html headers go here
    outfile.puts "<html>"
    outfile.puts "<head><title>#{feed_name}</title></head>"
    outfile.puts "<body>"
    outfile.puts "<h1>#{feed_name}</h1>"

    feed_data.each do |post|
      outfile.puts "<p>by <b>#{post["from"]["name"]}</b> at: <b>#{DateTime.parse(post["created_time"]).strftime(DATE_FORMAT)}</b></p>"

      if post["picture"].present?
        outfile.puts "<div style=\"float:left;\"><img src=\"#{post["picture"]}\" /></div>"
      end
      name_str = post["name"]
      name_str = "<a href=\"#{post["link"]}\">#{name_str}</a>" if post["link"].present?
      outfile.puts "<h2>#{name_str}</h2>"

      message = post["message"]
      message.each do |para|
        outfile.puts "<p style=\"clear:both;\">#{para}</p>"
      end

      comments = post["comments"]

      if comments.present? && comments["count"].present? && comments["count"].to_i > 0
        outfile.puts "<h3>Comments</h3>"
        outfile.puts "<dl>"

        comments["data"].each do |comment|
          outfile.puts "<dt>by <b>#{comment["from"]["name"]}</b> at: <b>#{DateTime.parse(comment["created_time"]).strftime(DATE_FORMAT)}</b></dt>"
          outfile.puts "<dd>#{comment["message"]}</dd>"
        end
        outfile.puts "</dl>"
      end #any comments present
      outfile.puts "<hr />"
    end # each post

    # html footers go here
    outfile.puts "</body>"
    outfile.puts "</html>"
  end # with open outfile
end #feed data present

Link: What rails security means for your startup

2013-02-08T14:08:00.000+11:00

If you hadn't already heard, Rails has a security vulnerability that affects all versions of Rails. This one is about XML-parsing of YAML strings.

This was followed by a second vulnerability in the JSON parser - again of YAML-parsed code.

So what does this all mean for all of us running Rails-based systems? Is this just a flash-in-the-pan issue that will fade away the moment it's out of the public eye? or is it a herald of the coming apocalypse?

A really cogent overview of what the rails security issue means for your startup has been written by Patrick, and I strongly recommend you read it, and pass it on.

Amongst a number of useful overviews, it covers such things as "yeah, but we're not a high-profile site, nobody's going to attack us right?" and concludes that the worst may not yet be past, and that:

You Should Be At Defcon 2 For Most Of February

Link: Help Vampires: A Spotter’s Guide

2013-02-02T12:40:00.000+11:00

Here's a great post on the ubiquitous "Help Vampire" who drains the life out of helpful communities...

Help Vampires: A Spotter’s Guide gives tips on how to spot, avoid and reform them for the future benefit of humanity...

XML-YAML-parsing security fix for older versions of rails

2013-01-27T12:57:00.000+11:00

Earlier I mentioned the Serious Rails vulnerability that affects all versions of Rails for the last six years.

A fix has been put into the latest versions of Rails 2 and 3. but it requires you to upgrade to the latest version.

If you have an older version of rails and can't upgrade for various reasons (eg we are still stuck on v 2.3.2 due to some legacy code), there's a better fix for the *link* xml parsing error than the workarounds on offer (which tend to switch off your ability to parse XML).

The fix that we've done requires that you use bundler, though you could equally-well freeze rails into vendor/gems and make the same patch there. We chose the bundler/github approach because it reduces the size of our repository.

Step 1: fork a copy of rails for yourself

fork rails
git clone it into a local directory.
checkout the *tag* that corresponds with the version you are on (eg for us: v2.3.2.1) - you can see what tags there are by running: "git tag -l"
Don't worry about the big scary message it gives you about a detached HEAD - that just means you've got a specific commit checked out instead of a branch.
create a branch for yourself eg for us: git branch v2.3.2_xmlfixes
checkout that branch eg git checkout v2.3.2_xmlfixes
push that to your repo on github with eg: git push -u origin v2.3.2_xmlfixes

Now you have a forked copy of the rails repo with a branch at the rails-version you are using that you can refer to in your Gemfile.

Step 2: actually apply the patch...

cherry-pick this commit (which if you look at github is is the one from v 2.3.15 that fixes this very error). Using:

git cherry-pick 70adb9613e4a40c5645c99da37

Note: You are likely to get conflicts with the CHANGELOG - you can keep them or just throw them away as you wish (it's just the changelog which describes the latest changes).

commit and push to your repository.

Now you have a patched version of rails in your git repository.

Step 3: update your Gemfile

Your Gemfile is likely to have a line that includes rails such as:

gem 'rails', '2.3.2'

You need to update that line to point at *your* git repository and your new branch.

The following *should* Just Work:

gem 'rails', '2.3.2', :branch => "v2.3.2_xmlfixes", 
:git =>l 'git://github.com/<your_git_repo>/rails.git'

To find the git url, you can go to your forked copy of git, look near the top of the page where it has a text-box with a git-link. Make sure you click on the "Git Read-only" button, and copy what's in the box on the right.

The branch to use is whatever you named your branch in step 5 above.

You should now be able to run bundle install to regenerate your copy of rails - and it will pull the details from your forked-and-patched copy

Troubleshooting

Unfortunately, when I used the above, it bundled correctly, but any attempt to spin up the server just caused the following error:

./script/../config/boot.rb:61:in `require': no such file to load -- initializer (LoadError)
 from ./script/../config/boot.rb:61:in `load_initializer'
 from ./script/../config/boot.rb:117:in `run'
 from ./script/../config/boot.rb:17:in `boot!'
 from ./script/../config/boot.rb:130
 from script/server:2:in `require'
 from script/server:2

Luckily the answer is in the StackOverflow question: how to use a branch in a fork of rails in a project with bundler.

First, you need to add .gemspec files into your patched copy of rails. If you're forking 2.3.10, you can copy the gemspec files from the commit Adding .gemspec files for all gems in the 2-3-stable version of rails created by the author of the above stackoverflow issue.

Otherwise you'll need ones correct for your own version. The commit above talks about generating them from the associated Rakefiles. I created them by copying the gemspec files listed in the commit above, and then just copying over the spec = Gem::Specification block with the equivalent block that is in the Rakefiles.

eg for actionpack.gemspec, I copied the actionpack.gemspec from the commit into the rails/actionpack directory in my forked copy of rails. Then I opened up rails/actionpack/Rakefile and copied the whole block of code that begins with spec = Gem::Specification into the actionpack.gemspec file, deleting the previous block from that file first.

You will know if you got the gem-dependencies wrong if you get an error like the following:

Bundler could not find compatible versions for gem "activesupport":
  In Gemfile:
    actionpack (>= 0) ruby depends on
      activesupport (= 2.3.10) ruby

    activesupport (2.3.2)

Now you have generated the gemspecs, add them to your forked copy of Rails, commit them and push the commit to your github repo.

Then you can put the following in your Gemfile:

:git => 'git://github.com/<your_git_repo>/rails.git', :branch => "v2.3.2_xmlfixes" do
  # Note: load-order is essential for dependencies
  gem 'activesupport',  :branch => "v2.3.2_xmlfixes" # this must go first
  gem 'actionpack',     :branch => "v2.3.2_xmlfixes" # this must go second
  gem 'actionmailer',   :branch => "v2.3.2_xmlfixes"
  gem 'activerecord',   :branch => "v2.3.2_xmlfixes"
  gem 'activeresource', :branch => "v2.3.2_xmlfixes"
  gem 'rails',          :branch => "v2.3.2_xmlfixes" # this must go last
end

Note: make sure the gems are in the order above, with rails last - otherwise you'll get something like:

Could not find gem 'activesupport (= 2.3.10) ruby', 
   which is required by gem 'activerecord (>= 0) ruby', in any of the sources.

Also note: you *must* explicitly set the branch on the git-repo line and *also8 on all the gem-lines (and they must match) otherwise bundle install will work fine, but if you try anything else you'll get the infuriating error:

git://github.com/<your_git_repo>/rails.git (at v2.3.2_xmlfixes) is not checked out. Please run `bundle install`

Finally

you should now be able to run bundle update and bundle install and it should now work.

This has been tricky to explain, and the steps are complex - if something's not clear, let me know and I'll try and make it more plain.

Link: DHH - testing like the TSA

2013-01-21T10:23:00.000+11:00

Learning how to test is an important skill, learning what *not* to test is also an important skill. DHH shares his thoughts on this subject, with a brief article: Testing like the TSA which cuts through the "security theatre" aspect that can sometimes begin to surround our testing efforts.

There's also a long discussion on Y-combinator about the article here Hacker News: Testing like the TSA

Link: Why 2012 was the best year ever

2013-01-15T13:48:00.000+11:00

So many people are on the doom-and-gloom bus - so much so that it blinds them to the spectacular, amazing things happening in the world today. New things are being created and built, and medicine is practically screaming along.

Not that this stops the doom-sayers who claim, darkly that it all comes at a heavy price, that the advances of the first world are killing the rest of the world, and that poverty and death are on the rise...

Well, pooh to that. Here's an article that explains that actually, the world is improving at a rapid pace, including all of the usual favourite doom-and-gloom topics: Why 2012 was the best year ever

Read that before you try and tell me we "shouldn't bring children into this world" or that the world isn't as good as it was back when...

Serious rails vulnerability - read this!

2013-01-09T11:16:00.001+11:00

A serious vulnerability in *all* versions of rails (for the last six years) has been spotted.

In brief: complex xml-style params go through an XML-parser that will interpret based on types. "yaml" is a valid type, and that loads the YAML-parser... which instantiates any embedded classes that can include arbitrary code - leading to all kinds of injection-attack possibilities.

A general discussion of the problem, including patched versions and workarounds for old versions is available here: Multiple vulnerabilities in parameter parsing in Action Pack

A more in-depth look at what the problem entails is available here: Analysis of Rails XML Parameter Parsing Vulnerability

Link: Everything you need to know about the National Broadband Network

2012-12-06T18:00:00.000+11:00

Hi all, this one is aimed at other Australians. Since it was first announced in 2009, there's been a lot of rumbling about the National Broadband Network plan - what it actually is, whether it's worth it, what it actually means for us or for the Australian budget etc etc

This link provides an article that fully explains what it's all about and is well worth a read.

The NBN: everything you need to know

TL;DR: it's a plan to replace the antiquated Telstra copper-network with high speed fibre (preferably to-the-home) so that we can keep up with the massive influx of bandwidth we're all using these days just like all the other high-tech countries did ages ago... and yes, it's worth it.

Link: Why engineers are grumpy

2012-11-30T17:17:00.000+11:00

There have been many great articles on how to deal with IT-folks. This is another of them: The care and feeding of software engineers (or, why engineers are grumpy)

It explains: what makes engineers tick, what makes them grumpy and why, and also solutions to make it all Just Work. It covers both the strengths and weaknesses of engineers, and how to build on the former, while avoiding the latter.

I'll leave you with a few quotes:

In almost every other industry where things are built, it is expected that all requirements and details are agreed upon and finalized before building commences. Except in software. In software there’s “not enough time” to gather all the requirements ahead of time. The importance of moving quickly is hammered into us from day one. And so engineers learn to fill in the gaps left by product managers just to keep the project going. Product managers, of course, also have the reputation for changing their minds frequently, which means engineers assumptions are often invalidated partway through the process. Is it any wonder that software engineers tend to burn out quickly and change jobs frequently?

True priorities aren’t transient, they are static. The frequency with which people above us change their minds is incredibly frustrating for software engineers. We frequently stand ready to march into battle and just want a direction to march in. But if you tell us one day that we’re building a house and the next day that we’re building a car, you should expect some dissension in the ranks.

There are few phrases that anger software engineers more than, “I used to code.” ... If I were to ask LeBron James how much time he needs to prepare for a game, I’m sure he’d be amused if I disagreed because I played basketball in high school. Software engineers get the equivalent all the time.

We software engineers are an interesting bunch. There’s a definite personality that comes along with us, and we really do want to make the best thing possible. If you stop treating us like short-order cooks and start treating us like part of the creative process, you are likely to get much farther, faster than you would otherwise. The teams on which I’ve worked have all had varying degrees of friction caused by not understanding the engineers’ mindset and what drives them. It’s my sincere hope that this article will lead to better communication between engineers and those that they work with. It’s really not that hard. We’re all just people who want to feel like a part of the solution rather than a worker bee.

Deleting un-named foreign keys in a migration

2012-11-24T15:25:00.000+11:00

So, we use foreign-key constraints in our db. It's pretty annoying to get Rails to work nicely with that, but we have some basic helper methods (which I didn't write, so I cant share).

However I recently had some trouble dropping an old table. I kept getting these errors:

Mysql::Error: Error on rename of './mydbname/#sql-1ca8_f7842' to 
   './mydbname/my_table_name' (errno: 150): DROP INDEX `my_table_name_idx3` 
   ON my_table_name

I discovered that this was because it had foreign-key constraints. The awful error message is itself listed as a bug on mysql... but the real problem is that it doesn't want to drop an index I've asked it - because there's a foreign-key constraint on the column referenced by that index*.

Fair enough.

Unfortunately, the "remove_foreign_key" code we have assumes the foreign-key has been named in a certain way (table_name_column_name) but in this case it wasn't. So Trying a drop index on that caused it to give an equally unhelpful message along the same lines...

This foreign key has been hanging around since the dawn-o-time, and it has one of those automagically-generated constraint-names built by the db itself... something like: my_table_name_ibfk_16.

Now that'd be fine to drop if we just had one client. with one database... you can easily put the following into a migration:

   execute "ALTER TABLE `my_table_name` DROP FOREIGN KEY `my_table_name_ibfk_16`"

Unfortunately for us, we have 150 clients - each with their own db... and it looks like that constraint-name differs depending on which order the foreign-keys got created. ie sometimes it's my_table_name_ibfk_16 and sometimes it's my_table_name_ibfk_3 - and if you drop it by number - you could be killing the WRONG foreign-key constraint... which would just be embarrassing.

The way out of this quandary is to query the information_schema table to find the foreign-key's actual constraint-name. and here it is for your amusement. (note: put this into initializers eg by saving it as config/initializers/migrations.rb

class ActiveRecord::Migration
  # grab the db-name out of the connection and persist it
  # it's not going to change over the course of a single migration
  def self.fetch_database_name
    @@database_name ||= connection.database_name
  end

  # Use this if the foreign-key was created without an explicit name - 
  # and has one of the automatically-generated constraint-names.
  #
  # This method queries the information-schema table to fetch out the key
  # name before continuing to drop the foreign-key
  #
  # Use this in your migrations with:
  #    remove_legacy_foreign_key :table_name, :field_name
  # eg:
  #    remove_legacy_foreign_key :widgets, :wodget_id
  def self.remove_legacy_foreign_key(table, column_name)
    # first pull the foreign-key name from the information schema
    result = execute "select constraint_name from information_schema.key_column_usage as ke where ke.table_schema = '#{fetch_database_name}' and ke.table_name = '#{table}' and ke.column_name = '#{column_name}';"
    name = result.fetch_hash['constraint_name']
    raise "Got no foreign key by that name" unless name.present?
    execute "ALTER TABLE `#{table}` DROP FOREIGN KEY `#{name}`"
  end
end

[*] Note: mysql also gives a similar error if you're dropping a foreign-key constraint that doesn't exist at all by the name you give it:

ERROR 1025 (HY000): Error on rename of './mydbname/my_table_name' to 
   './mydbname/#sql2-1ca8-f6cc5' (errno: 152)

Kiva is addictive

2012-11-17T11:59:00.001+11:00

No IT angle today...

I have a mixed approach to giving to charity. I have a number of regular direct debits for various causes, and I also give sporadically in lump sums when inspiration strikes me.

My latest of these is Kiva (here's a Kiva invite) - which is a site that does micro-loans to people in developing countries.

Kiva has been on my radar for a while now, but I'd been putting it off because I had a few misconceptions about the whole process and I finally got down and actually did some reading on what it's all about.

I was mainly concerned by the fact that in most cases, the loans have already been given out, and you are paying for something that is already a fait accompli - which I thought was weird, and that it meant that you weren't really lending the money to the people you had chosen.

Turns out to make a bit more sense. These people need loans immediately, so the partner in that country does so, and then posts the loan up to Kiva. When you choose to "fund" that loan, your money then underwrites it. You are taking on the risk for the loan.

Now I've had it explained, it makes sense to me, so I've been having fun donating... it's a bit more addictive than I expected. I've already made five loans in as many countries, three of which have now been fully funded - one of which I was the final person that funded it (and that feels pretty good). We'll see how they go.

Link: Unit tests don't find bugs

2012-11-08T10:16:00.000+11:00

A quick article called unit tests don't find bugs points out what unit tests are really for, and advises us not to forget that unit tests cannot be used as a substitute for QA-style testing.

ruby gotcha: beware trailing commas

2012-11-02T11:51:00.000+11:00

I had something like the following (though with more values):

 h = OrderedHash.new()
 h[:a] = {:a => 1}
 h[:b] = {:b => 2}, # note the trailing comma - oops
 h[:c] = {:c => 3}

and started getting errors telling me that h[:b] was an array, when I was expecting it to be a hash...

Here's what's actually happening (via irb):

>> a = {:a => 1}
=> {:a=>1}
>> b = {:b => 2},
?> c = {:c => 3}         # still part of the previous statement
=> [{:a=>1}, {:c=>3}] # so we end up with an array

Note to self: beware trailing commas

[SOLVED] undefined method `name' for "Ascii85":String

2012-10-27T09:39:00.000+11:00

So, we're still using Rails 2.3.2 for our client... but I'm *trying* to upgrade it piece by piece. In doing so, I recently had to upgrade rubygems, so that it would work with a recent version of a gem we desperately needed.

Unfortunately it led to the undefined method `name' for "Ascii85":String bug that I wrote about when I last tried to upgrade rubygems

Back when it broke rubygems the first time, the only solution seemed to be to downgrade rubygems and hope for a future time when I could upgrade everything and get it working again.

But that wasn't an option this time... and so I had to hack up a solution.

The problem is described below in solution 2, but it's basically a problem with rails - which isn't using the newer rubygems gem-comparing "==" function properly any more.

Solution 1: Gem bundler

One solution that definitely works is to upgrade to using gem bundler (or the Rails2.3 version of bundler).

This definitely works because bundler doesn't have the same bug in it that Rails 2 has - it uses gem-comparison the new way.

But some of our clients aren't on the bundled release yet, and I need to quickly fix bugs and push to those clients... so I needed a solution that would let me still load the bundler-free version without having to downgrade rubygems to do it.

Solution 2: Fix rails

This solution requires a (very small) hack on Rails... but I figured that was ok as it was just to get my own system working so I can fix things without having to go through the pain of installing rvm (which I don't have time for right now).

So far the fix seems to be working, so I'll share (with the caveat that it's not well tested yet so YMMV).

The problem is really a Rails bug... not rubygems. So that's where you can safely hack (for now). From the stacktrace of the bug, find the copy of "rails/gem_dependency.rb" that is causing the issue:

/usr/lib/ruby/gems/1.8/gems/rails-2.3.2/lib/rails/gem_dependency.rb:239:in `==': 
undefined method `name' for "Ascii85":String (NoMethodError)
 from /usr/local/lib/site_ruby/1.8/rubygems/dependency.rb:218:in `==='

In my case, I opened /usr/lib/ruby/gems/1.8/gems/rails-2.3.2/lib/rails/gem_dependency.rb in a text editor (with sudo permissions). It'll have something like the following:

    def ==(other)
      self.name == other.name && self.requirement == other.requirement
    end

The actual problem occurs because Rails is sometimes passing the name already as a string to this function, instead of an actual spec. So to fix it, we just check for that with the following code:

    def ==(other)
      # The following line has been inserted to get around the bug:
      # "undefined method `name' for "Ascii85":String (NoMethodError)"
      return self.name == other if other.is_a?(String) # bugfix
      self.name == other.name && self.requirement == other.requirement
    end

It shouldn't alter existing functionality, it'll just get past the broken code that is in rails until you can upgrade rails, or use bundler.

If you (horror) actually need to install this fix on a live system; I'd recommend freezing rails to your gems directory before applying the fix. But in reality, I'd actually recommend just finding a way to upgrade to using bundler. It's a neater solution that won't require ongoing maintenance.

Link: Primer on sexism in the tech-industry

2012-10-21T11:54:00.000+11:00

"The topic of sexism and its role in the technology industry has seen a huge resurgence over the past 12 to18 months. Yet despite being discussed and examined with increasing frequency, a lot of the subject remains unclear and under-explained, making it difficult for those who care deeply about our industry to partake in these discussions. This is, in part, because the problems are incredibly complex, nuanced and difficult to explain, making it impossible for any one article to address them sufficiently (lest that article becomes a book). Nevertheless, today we're going to try and see how much of the basics we can clear up."

Faruk Ateş has written a most-excellent primer on sexism in the tech industry

You should read this (especially if you are male) before even considering posting an article or comment about women in the tech industry.

Making views/triggers/functions work in mysql for rails

2012-10-15T09:52:00.000+11:00

We use mysql with Rails. We also use foreign-keys, views, triggers and stored procedures.

Rails's mysql gem still doesn't know how to handle any of the above yet... which is kind of inconvenient if you want to test your application.

The problem arises because even with a SQL schema dump - it only dumps the tables and foreign keys. No views, no triggers, no stored procedures. If you have code that relies on these... your application starts getting MySql errors.

Luckily there seems to be a solution with the db_structure_ext gem, which I've been using happily now for a couple of weeks.

I found the README isn't at all clear as to how to get it set up, so here's some basic instructions on how I made it work with MySQL on Rails 2.3.X

1) Install it as per the README:

gem install db_structure_ext and/or add gem 'db_structure_ext' to Gemfile and then bundle install

2) require it in your Rakefile

# extended db-structure-dump (also does triggers, routines etc)
require 'db_structure_ext/tasks'

3) Monkey-patch ActiveRecord

Create a file call config/initializers/mysql_adapter.rb (or similar) and add the following code:


module ActiveRecord
  module ConnectionAdapters
    class MysqlAdapter
      require 'db_structure_ext/init_mysql_adapter'

      # This is an overridden implementation of the structure_dump so that the
      # rake take db:structure:dump will dump out the schema elements.
      def structure_dump
        connection_proxy = DbStructureExt::MysqlConnectionProxy.new(ActiveRecord::Base.connection)
        connection_proxy.structure_dump
      end

    end
  end
end

You need this so that Active Record actually extends the new methods. If you don't do this, then you can call "structure_dump" yourself in your code, independently for certain tables. But the new functionality won't come through as the default for all of your db tables unless you extend MysqlAdapter as per above.

Link: Women at work in Silicon Valley

2012-10-09T13:38:00.001+11:00

Balancing work with family is one of the Hard Problems of Social Engineering a career... especially if you're female, and already suffer from several disadvantages in a high-tech workplace.

This article on The Guardian gives a great overview about how women in Silicon valley are making "Women at work" work.

Link: Amazon Review-spoofing scandal

2012-09-27T09:15:00.000+10:00

This is a lesson in how any system can be gamed.

Amazon reviews have been built up to be an integral part of how we evaluate books for sale on amazon. The ability to see what people are saying about a product has turned the process of book-buying on its head. Instead of briefly skimming the back of the jacket and taking a gamble on whether it's good or not, you can see the general consensus of the community of readers and *hopefully* get a better idea on whether this book is any good.

Unfortunately, like any process, it can be gamed, and this week we've seen how deep this can get.

A guy called Rutherford set up a review-buying service where authors could come and purchase good-reviews on amazon. In one case making a new author (John Locke) into an overnight millionaire.

Here's a link to the full story on the John Locke book review scandal

The uproar in the authorial world has been tremendous. and amazon are investigating what happened and what they can do about it. It's a tough job, because it's extremely difficult to figure out whether reviews are sincere.

Hopefully they can find a way because I'd hate to lose the ability to see the true measure of the community's feelings towards a book.

Link: count vs length vs size in Rails

2012-09-21T10:01:00.000+10:00

Every now and then I have to remind myself of the differences between "count", "length" and "size" in Rails. I've never found a better source for this than the has_many :through article: count vs length vs size

It's well worth a read, even if you're an old Rails pro.