Recursive Design

Hiding ruby gem 'is deprecated with no replacement' messages

2011-05-21T00:00:00+12:00

MAY 21 2011

Hiding ruby gem ‘is deprecated with no replacement’ messages

Filed under quickie, rails, ruby, and rvm.

I recently installed ruby 1.9.2 via rvm, and suddenly every command started spitting out hundreds of line of deprecation messages :

> rails server
NOTE: Gem::Specification#default_executable= is deprecated with no replacement. It will be removed on or after 2011-10-01.
Gem::Specification#default_executable= called from /Users/dave/.rvm/gems/ruby-1.9.2-p0@global/specifications/rubygems-update-1.8.2.gemspec:11.
NOTE: Gem::Specification#default_executable= is deprecated with no replacement. It will be removed on or after 2011-10-01.
Gem::Specification#default_executable= called from /Users/dave/.rvm/gems/ruby-1.9.2-p0@rails3/specifications/annotate-models-1.0.4.gemspec:10.
NOTE: Gem::Specification#default_executable= is deprecated with no replacement. It will be removed on or after 2011-10-01.
Gem::Specification#default_executable= called from /Users/dave/.rvm/gems/ruby-1.9.2-p0@rails3/specifications/beefcake-0.3.1.gemspec:10.
NOTE: Gem::Specification#default_executable= is deprecated with no replacement. It will be removed on or after 2011-10-01.
Gem::Specification#default_executable= called from /Users/dave/.rvm/gems/ruby-1.9.2-p0@rails3/specifications/bundler-1.0.7.gemspec:10.
NOTE: Gem::Specification#default_executable= is deprecated with no replacement. It will be removed on or after 2011-10-01.
Gem::Specification#default_executable= called from /Users/dave/.rvm/gems/ruby-1.9.2-p0@rails3/specifications/churn-0.0.12.gemspec:10.
.....                        
=> Booting WEBrick
=> Rails 3.0.7 application starting in development on http://0.0.0.0:3000
=> Call with -d to detach
=> Ctrl-C to shutdown server
[2011-05-21 20:12:23] INFO  WEBrick 1.3.1
[2011-05-21 20:12:23] INFO  ruby 1.9.2 (2010-08-18) [x86_64-darwin10.7.0]
[2011-05-21 20:12:23] INFO  WEBrick::HTTPServer#start: pid=52918 port=3000

A bit of googling turned up this stack overflow post.

Run the suggested gem pristine command and restart rails, and voilà:

> gem pristine --all --no-extensions
> rails server             
=> Booting WEBrick
=> Rails 3.0.7 application starting in development on http://0.0.0.0:3000
=> Call with -d to detach
=> Ctrl-C to shutdown server
[2011-05-21 20:12:23] INFO  WEBrick 1.3.1
[2011-05-21 20:12:23] INFO  ruby 1.9.2 (2010-08-18) [x86_64-darwin10.7.0]
[2011-05-21 20:12:23] INFO  WEBrick::HTTPServer#start: pid=52918 port=3000

The post above and this other one both also recommend this if you’re using rvm:

> rvm rubygems current
Cannot switch to 1.3.7 for this interpreter.

… which just spat out that error in my case and didn’t seem to do anything else, but it’s possible it may work for you if gem pristine doesn’t.

'405 Not Allowed' error serving Facebook apps with nginx

2011-05-20T00:00:00+12:00

MAY 20 2011

‘405 Not Allowed’ error serving Facebook apps with nginx

Filed under facebook, linux, nginx, and programming.

I’ve spent the last couple of evenings playing around with some Hello World-type stuff for Facebook app development in preparation for an up-coming Facebook/Rails gig.

After creating a new app on Facebook, you are presented with the code for a basic starter-page for the app:

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml"
      xmlns:fb="http://www.facebook.com/2008/fbml">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
    <title>Hello World!</title>
  </head>
  <body>
    <h1>Hello World!</h1>
    <p><fb:login-button autologoutlink="true"></fb:login-button></p>
    <p><fb:like></fb:like></p>

    <div id="fb-root"></div>
    <script>
      window.fbAsyncInit = function() {
        FB.init({appId: 'xxxxxxxxxxxxxxx', status: true, cookie: true,
                 xfbml: true});
      };
      (function() {
        var e = document.createElement('script');
        e.type = 'text/javascript';
        e.src = document.location.protocol +
          '//connect.facebook.net/en_US/all.js';
        e.async = true;
        document.getElementById('fb-root').appendChild(e);
      }());
    </script>
  </body>
</html>

While I’m not quite sure what all of that does yet, I threw it up on a server as index.html just for kicks, added a quick subdomain to Nginx:

server {
    listen       80;
    server_name  facebook.recursive-design.com;

    # Main location
    location / {
         root /var/www/facebook.recursive-design.com;
    }
}

…and lo-and-behold we have our Hello World:

While everything is fine-and-dandy so far, accessing this from within Facebook is another story:

After triple-checking the Nginx config (and there’s not much that could go wrong there) I was none the wiser what was triggering the 405 Not Allowed.

A bit of googling turned up a post on the Nginx forums that shed some light on the matter:

News to me. Since everything that comes from Facebook appears to be a POST, we’re not going to get anywhere with this static index.html app of ours.

As mentioned in the thread, Igor has posted a workaround, or alternatively you can proxy the page to apache or something upstream which doesn’t complain. I went the upstream apache root (though in hindsight the workaround seems like less work), and voilà:

I’d love to know the motivation behind this restriction – I can’t think of any security issues that posting to a static file would cause, but then again i’m no Igor Sysoev :)

Any ideas?

Continuous integration for PHP with phpUnderControl

2011-05-13T00:00:00+12:00

MAY 13 2011

Continuous integration for PHP with phpUnderControl

Filed under ant, continuous.integration, java, php, and programming.

Whenever I start a new PHP project for a client, one of the first things I set up is usually phpUnderControl – a CI (continuous integration) server specifically for PHP built on top of CruiseControl.

It provides a bunch of stuff out-of-the box, from the essential (broken-build email alerts) to the nice-to-have (code mess-detection, copy/paste-detection) to the quite frankly bewildering (yes I’m looking at you, mystifying code-dependency graph).

The PHP CodeSniffer integration is worth the price of admission by itself, and if nothing else you can impress your non-tech boss with some nice shiny graphs :

… and a nice row of green ticks to show that your tests are all passing :

I’ve installed this on Ubuntu twice in the last month, and finally got sick enough of figuring-it-out-from-scratch to actually write down some reproducible install steps.

Since CruiseControl is Java-based, we need to install Java first :

> sudo aptitude install sun-java6-jre sun-java6-jdk

You’ll need to agree to some licenses etc while this is installing.

In more recent versions of Ubuntu (10.4 onwards), these packages have been moved to a partner repository, and are no longer part of the default Lucid repositories. If this is the case, you will get some errors similar to the following when you try to install them:

No candidate version found for sun-java6-jre
No candidate version found for sun-java6-jdk
No candidate version found for sun-java6-jre
No candidate version found for sun-java6-jdk
No packages will be installed, upgraded, or removed.

If this is the case, you need to add the following line to the end of your /etc/apt/sources.list:

deb http://archive.canonical.com/ lucid partner

… and then update the source list:

> sudo apt-get update

Next, install some of the required PHP packages

> sudo aptitude install php5-dev php-pear php5-xdebug

phpUnderControl needs the Xdebug PHP extension to work its magic, so make sure that xdebug.ini is present and contains the appropriate config (it should be somewhere like /etc/php5/conf.d/xdebug.ini) :

> more /etc/php5/conf.d/xdebug.ini
zend_extension=/usr/lib/php5/20090626/xdebug.so
xdebug.profiler_enable_trigger = onc

Check the PHP info output to make sure it’s installed correctly

> php -i | grep xdebug
/etc/php5/cli/conf.d/xdebug.ini,
xdebug
xdebug support => enabled
xdebug.auto_trace => Off => Off
...

Next, we need to install CruiseControl itself. We’re going to do this in /opt/.

> cd /opt/
> sudo wget http://heanet.dl.sourceforge.net/sourceforge/cruisecontrol/cruisecontrol-bin-2.8.4.zip
> sudo unzip cruisecontrol-bin-2.8.4.zip
> sudo ln -s cruisecontrol-bin-2.8.4 cruisecontrol
> sudo rm cruisecontrol-bin-2.8.4.zip

We also need to create a start-up script for CruiseControl, since it doesn’t come with one by default. Save the following as /etc/init.d/cruisecontrol :

#!/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:
. /lib/lsb/init-functions
JAVA_HOME=/usr/lib/jvm/java-6-sun
NAME=cruisecontrol
DAEMON=/opt/cruisecontrol/cruisecontrol.sh
PIDFILE=/opt/cruisecontrol/cc.pid
 
test -x $DAEMON || exit 5
 
RUNASUSER=www-data
UGID=$(getent passwd $RUNASUSER | cut -f 3,4 -d:) || true
 
case $1 in
start)
log_daemon_msg "Starting Cruisecontrol server" "cc"
if [ -z "$UGID" ]; then
log_failure_msg "user \"$RUNASUSER\" does not exist"
exit 1
fi
cd /opt/cruisecontrol/
./cruisecontrol.sh > /dev/null 2>&1
log_end_msg $?
;;
stop)
log_daemon_msg "Stopping Cruisecontrol server" "cc"
start-stop-daemon --stop --quiet --oknodo --pidfile $PIDFILE
log_end_msg $?
rm -f $PIDFILE
;;
restart|force-reload)
$0 stop && sleep 2 && $0 start
;;
status)
pidofproc -p $PIDFILE $DAEMON >/dev/null
status=$?
if [ $status -eq 0 ]; then
log_success_msg "Cruisecontrol server is running."
else
log_failure_msg "Cruisecontrol server is not running."
fi
exit $status
;;
*)
echo "Usage: $0 {start|stop|restart|force-reload|status}"
exit 2
;;
esac

We need to make this script executable :

> sudo chmod a+x /etc/init.d/cruisecontrol

We also want CruiseControl to start up automatically when the server boots up – use update-rc.d to do this :

> sudo update-rc.d cruisecontrol defaults

I had problems with CruiseControl not knowing where JAVA_HOME was for some reason. This probably points to some kind of Java mis-configuration on my end, but since I don’t use Java for anything else on the build server it’s easier just to fix it by defining JAVA_HOME in the CruiseControl startup script by editing the script itself :

> sudo vi /opt/cruisecontrol/cruisecontrol.sh

.. and adding a line after the initial bash line :

#!/usr/bin/env bash

JAVA_HOME="/usr/lib/jvm/java-6-sun"

At this point, you should be able to start CruiseControl (keeping in mind we haven’t installed phpUnderControl itself yet) :

> /etc/init.d/cruisecontrol start

After a minute or so (it takes a while to start up), you should be able to see the basic CruiseControl page at http://localhost:8080/cruisecontrol/.

The next step is to install phpUnderControl itself. One of the main dependencies here is PHPUnit, and it is here that I usually hit the first hurdle.

By default, Ubuntu installs PEAR version 1.9.0 – this version doesn’t support the current version of PHPUnit, which is required by phpUnderControl. Depending on your system, you may not need to upgrade anything at this point, but on a fresh Ubuntu system I usually get the following error performing the next steps :

phpunit/PHPUnit requires PEAR Installer (version >= 1.9.2), installed version is 1.9.0

To avoid this, we want to make sure we have the latest version of PEAR before we try and install more dependencies :

> sudo pear upgrade pear

Depending on your setup, you may also need to install a newer version of Xdebug. You can check the version you have installed with :

> php -i | grep Xdebug   
with Xdebug v2.2.0-dev, Copyright (c) 2002-2011, by Derick Rethans

If you have a version less that 2.0.5, you’re probably going to hit the wall later on in the installation with the error :

phpunit/PHP_CodeCoverage requires PHP extension "xdebug" (version >= 2.0.5), installed version is 2.0.4

If you have a version less than 2.0.5, save yourself some hurt in the next couple of minutes and just go ahead and upgrade it now. Your PHP module directory may differ depending on the Ubuntu version, but should be somewhere under /usr/lib/php5/ :

> cd /tmp
> svn co svn://svn.xdebug.org/svn/xdebug/xdebug/trunk xdebug
> cd xdebug
> phpize
> ./configure --enable-xdebug
> make
> sudo mv /usr/lib/php5/20060613+lfs/xdebug.so /usr/lib/php5/20060613+lfs/xdebug.so.orig
> sudo cp modules/xdebug.so /usr/lib/php5/20060613+lfs/

Now that everything is up-to-date, we want to install some of the PHP dependencies, as well as phpUnderControl itself.

> sudo pear channel-discover components.ez.no
> sudo pear install -a ezc/Graph
> sudo pear config-set preferred_state beta
> sudo pear channel-discover pear.phpunit.de
> sudo pear channel-discover pear.symfony-project.com
> sudo pear channel-discover pear.phpundercontrol.org
> sudo pear install --alldeps --force phpuc/phpUnderControl-beta
> sudo pear channel-discover pear.pdepend.org
> sudo pear install channel://pear.pdepend.org/PHP_Depend-0.9.11
> sudo pear channel-discover pear.phpmd.org
> sudo pear install channel://pear.phpmd.org/PHP_PMD-0.2.4
> sudo pear install --alldeps phpunit/phpcpd 
> sudo pear channel-discover pear.phing.info
> sudo pear config-set preferred_state alpha
> sudo pear install --alldeps --force phing/phing

If everything installed correctly, we should now be able to apply the phpUnderControl patches to CruiseControl, which will give us our nice shiny PHP-tailored interface :

> sudo phpuc install /opt/cruisecontrol 
> sudo /etc/init.d/cruisecontrol restart

Next we want to remove the example project that ships with CruiseControl, and add our actual PHP project. We’re going to call our new project php-example , and assume we’re checking it out from subversion (although git is do-able too) – you’ll need to adjust the following instructions to fit your own project and checkout procedure.

> cd /opt/cruisecontrol/projects
> sudo rm -Rf connectfour/
> sudo mkdir -p php-example/build/api
> sudo mkdir php-example/build/charts
> sudo mkdir php-example/build/coverage
> sudo mkdir php-example/build/graph
> sudo mkdir php-example/build/logs
> sudo svn co svn://path/to/php-example/trunk php-example/source

At this point you also want to set up your newly checked-out project so that database connections work, unit tests pass etc. Creating a PHPUnit test suite is beyond the scope of this article, but if you don’t have any tests yet, grab the documentation from the PHPUnit page, write and commit a couple of simple tests, then come back here afterwards – we’ll wait for you!

After you are confident that your unit tests are running correctly, your new project will need a build.xml file to tell CruiseControl how to build it. Make an /opt/cruisecontrol/projects/php-example/build.xml file similar to the one below (and make sure you add it to version control too). phpUnderControl will use this file to perform the build process for your project.

Obviously you’ll need to customize this to fit the needs of your own project – make note of the entries with path-to-ignore and path-to-check and adjust them according to your own needs.

<?xml version="1.0" encoding="UTF-8"?>
<project name="php-example" default="build" basedir=".">

  <!--
  Helper target that initializes the CruiseControl project and creates the base 
  directory structure. 
  -->
  <target name="init">
    <!-- Create the source directory -->
    <mkdir dir="source" />
    
    <!-- Create the different build directories -->
    <mkdir dir="build" />
    <mkdir dir="build/api" />
    <mkdir dir="build/charts" />
    <mkdir dir="build/coverage" />
    <mkdir dir="build/graph" />
    <mkdir dir="build/logs" />
    
    <!-- Checkout phpUnderControl trunk into source -->
    <exec executable="svn">
      <arg line="co /path/to/php-example/trunk source" />
    </exec>
  </target>
    
  <!--
  The clean target is used to remove build artefacts of previous builds. Otherwise
  CruiseControl will present old, maybe successful results, even if your build 
  process fails.
  -->
  <target name="clean">
  	<!-- Remove old log files -->
    <delete>
      <fileset dir="${basedir}/build/logs" includes="**.*" />
    </delete>
  	<!-- Remove old api documentation -->
    <delete>
      <fileset dir="${basedir}/build/api" includes="**.*" />
    </delete>
  	<!-- Remove old coverage report -->
    <delete>
      <fileset dir="${basedir}/build/coverage" includes="**.*" />
    </delete>
        <!-- Remove old graphs -->
    <delete>
      <fileset dir="${basedir}/build/graph" includes="**.*" />
    </delete>
  </target>

  <!--
  The default build target for this project. It simply depends on all sub tasks
  that perform the project build. The sub targets are executed in the listed 
  order.
  
  1. 'clean' Clean old project build artefacts
  2. 'checkout' Update project working copy
  3. 'php-documentor' Generate api documentation
  4. 'php-codesniffer' Check for coding violations.
  4. 'pdepend' Code dependency information.
  4. 'phpcpd' Copy-paste detection.
  4. 'phpcpd' Mess detection.
  5. 'phpunit' Execute unit tests, generate metrics, coverage etc. 
  -->
  <target name="build" 
          depends="clean,checkout,php-documentor,php-codesniffer,pdepend,phpcpd,phpmd,phpunit" />

  <!--
  Performs an 'svn up' for the working copy.
  -->
  <target name="checkout">
    <exec executable="svn" dir="${basedir}/source">
      <arg line="up" />
    </exec>
  </target>

  <!--
  Generates the project documentation into the <project>/build/api directory.
  phpUnderControl uses the command line output of PhpDocumentor that is logged
  by CruiseControl.
  -->
  <target name="php-documentor" depends="init">
    <exec executable="phpdoc" dir="${basedir}/source">
      <arg line="-ct type -ue on -t ${basedir}/build/api 
                 -tb /usr/share/php/data/phpUnderControl/data/phpdoc/ -o HTML:Phpuc:phpuc
                 -d ${basedir}/source
                 -q
    	           -i paths-to-ignore1,paths-to-ignore2"/>
    </exec>
  </target>

  <!--
  Execute code sniffer. You can use a different coding standard if
  you want (--standard=Squiz) - check the php-codesniffer docs
  for more information.
  -->
  <target name="php-codesniffer" depends="init">
    <exec executable="phpcs" 
          dir="${basedir}/source" 
          error="/dev/null"
          output="${basedir}/build/logs/checkstyle.xml">
      <arg line="--report=checkstyle 
                 --standard=Squiz 
                 --extensions=php 
                 --tab-width=4 
                 -n
                 --ignore=path-to-ignore1,path-to-ignore2
                 ./path-to-check1 ./path-to-check2"/>
    </exec>
  </target>

  <!--
  Calculates dependencies and adds new graphs to the metrics page.
  -->
  <target name="pdepend" depends="init">
    <exec executable="pdepend" dir="${basedir}/source" logerror="on">
      <arg line="--summary-xml=${basedir}/build/logs/pdepend.xml
                 --jdepend-chart=${basedir}/build/charts/jdepend.svg
                 --overview-pyramid=${basedir}/build/charts/overview-pyramid.svg
                 --coderank-mode=inheritance,property,method
                 ./path-to-check1,/path-to-check2" />
    </exec>
  </target>

  <!--
  Copy-paste detection.
  -->
  <target name="phpcpd" depends="init">
    <exec executable="phpcpd" failonerror="false">
      <arg line="--log-pmd ${basedir}/build/logs/pmd-cpd.xml 
                 --exclude ${basedir}/path-to-ignore1
                 --exclude ${basedir}/path-to-ignore2
                 ${basedir}/path-to-check1 ${basedir}/path-to-check2" />
    </exec>
  </target>


  <!--
  Checks that the project source does not contain any 'mess'.
  -->
  <target name="phpmd" depends="init">
    <exec executable="phpmd" failonerror="false">
      <arg line="${basedir}/source
                 xml
                 codesize,unusedcode,naming
                 --reportfile ${basedir}/build/logs/pmd.xml
                 --exclude path-to-ignore1,path-to-ignore2" />
    </exec>
  </target>
  
  <!--
  Executes the project unit tests and stores the different logs in the 
  <project>/build/logs directory. Furthermore it generates the coverage report
  under <project>/build/coverage.
  -->
  <target name="phpunit" depends="init">
    <exec executable="phpunit" dir="${basedir}/source" failonerror="on">
      <arg line="--log-junit       ${basedir}/build/logs/phpunit.xml 
                 --coverage-clover ${basedir}/build/logs/phpunit.coverage.xml 
                 --coverage-html   ${basedir}/build/coverage
                 --configuration   ${basedir}/tests/phpunit.xml
                 ${basedir}/path-to-check"/>
    </exec>
  </target>
</project>

Now that our project has a build.xml file, we need to tell CruiseControl where it is. Edit the main config file ( /opt/cruisecontrol/config.xml ) and enter something similar to the following :

<?xml version="1.0"?>
<cruisecontrol>

  <project name="php-example" buildafterfailed="false">
    <plugin name="svnbootstrapper" 
            classname="net.sourceforge.cruisecontrol.bootstrappers.SVNBootstrapper" />
    <plugin name="svn" 
            classname="net.sourceforge.cruisecontrol.sourcecontrols.SVN" />

    <listeners>
      <currentbuildstatuslistener file="logs/${project.name}/status.txt"/>
    </listeners>

    <modificationset>
      <svn localWorkingCopy="projects/${project.name}/source/"/>
    </modificationset>

    <bootstrappers>
      <svnbootstrapper localWorkingCopy="projects/${project.name}/source/" />
    </bootstrappers>

    <schedule interval="60">
      <ant anthome="apache-ant-1.7.0" 
           buildfile="projects/${project.name}/build.xml"/>
    </schedule>

    <log dir="logs/${project.name}">
      <merge dir="projects/${project.name}/build/logs/"/>
    </log>

    <publishers>
      <!--
      Copies the generated api documentation into project artifacts directory.  
      -->
      <artifactspublisher dir="projects/${project.name}/build/api" 
                          dest="artifacts/${project.name}"                          subdirectory="api"/>
      <!--
      Copies the generated code coverage report into project artifacts directory.  
      -->
      <artifactspublisher dir="projects/${project.name}/build/coverage" 
                          dest="artifacts/${project.name}" 
                          subdirectory="coverage"/>
                          
      <!--
      Generates the different metric charts with the phpUnderControl ezcGraph
      extension. 
      -->
      <execute command="phpuc graph logs/${project.name} artifacts/${project.name}"/>
      
      <!--
      Sends simple text emails after a project build. For nicer html emails, 
      checkout the original CruiseControl documentation.
    
        * http://cruisecontrol.sourceforge.net/main/configxml.html#email
        * http://cruisecontrol.sourceforge.net/main/configxml.html#htmlemail 
      -->
      <email mailhost="smtp.localhost"
             username="username"
             password="password"
             returnaddress="build@php-example.com"
             buildresultsurl="http://build.php-example.com/buildresults/${project.name}"
             skipusers="true" 
             spamwhilebroken="true">
        <always address="build@php-example.com"/>
        <failure address="build@php-example.com"/>
      </email>
    </publishers>

  </project>
  
</cruisecontrol>

Note that the project name you enter (php-example) must match the name of the folder you checked your source files out to earlier.

Finally, restart CruiseControl and your build server should be ready to go!

> sudo /etc/init.d/cruisecontrol restart

Using phpUnderControl

You should now be able to view phpUnderControl at http://localhost:8080/cruisecontrol/ :

If you add other projects in /opt/cruisecontrol/config.xml , they will show up on this screen as well. Click the green tick button to force a build, or click the project name to view the build results for that project.

CruiseControl will check subversion for updates every minute or so and perform a build automatically if it finds any, so you shouldn’t need to force a build too often. If you entered your email details in config.xml , you should also get an email if the build fails for any reason.

Null pointer exceptions viewing the metrics page

With more recent versions of phpUnderControl , I’ve been getting a java.lang.NullPointerException whenever i try and access the metrics page :

java.lang.NullPointerException
	at net.sourceforge.cruisecontrol.chart.PieChartData.produceDataset(PieChartData.java:52)
	at de.laures.cewolf.taglib.DataContainer.getDataset(DataContainer.java:53)
	at de.laures.cewolf.taglib.SimpleChartDefinition.getDataset(SimpleChartDefinition.java:34)
	at de.laures.cewolf.taglib.SimpleChartDefinition.produceChart(SimpleChartDefinition.java:30)
	at de.laures.cewolf.taglib.AbstractChartDefinition.getChart(AbstractChartDefinition.java:81)
	at de.laures.cewolf.taglib.ChartImageDefinition.ensureRendered(ChartImageDefinition.java:131)
	at de.laures.cewolf.taglib.ChartImageDefinition.getBytes(ChartImageDefinition.java:125)
	at de.laures.cewolf.storage.SerializableChartImage.(SerializableChartImage.java:51)
	at de.laures.cewolf.storage.SessionStorage.storeChartImage(SessionStorage.java:57)
	at de.laures.cewolf.storage.SessionStorage.storeChartImage(SessionStorage.java:35)
	at de.laures.cewolf.taglib.tags.ChartImgTag.doStartTag(ChartImgTag.java:74)
	at org.apache.jsp.main_jsp._jspx_meth_cewolf_img_0(org.apache.jsp.main_jsp:1781)
	at org.apache.jsp.main_jsp._jspService(org.apache.jsp.main_jsp:695)
	at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:93)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:373)
	at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:470)
	at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:364)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:362)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:729)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:206)
	at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:324)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:829)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:513)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488)

It took me hours to get to the bottom of this because while it looks like a java problem, the actual problem is in PHP. The ClassComplexityInput.php class (used for generating the complexity graph) has an undefined constant or something in it (I forget the exact cause), which causes the graph not to be rendered, which in turn causes CruiseControl to fall over when it tries to render the metrics page.

The quick-and-dirty solution to this is simply to move the file somewhere else :)

> cd /usr/share/php/phpUnderControl/Graph/Input/
> sudo mv ClassComplexityInput.php ClassComplexityInput.php.bak

You lose the code dependency graph by doing this, but until I have some time to dig a bit deeper and put a patch together it’s good enough for me.

Weekend Project - Time Out In NZ

2011-05-04T00:00:00+12:00

MAY 04 2011

Weekend Project – Time Out In NZ

Filed under announcement, programming, rails, and ruby.

I’ve been flat out with a couple of contracting projects recently – apologies for the four-month blogging hiatus!

One of these projects (albeit a quickie) was a weekend spent making a rails site for a couple of New Zealand organizations to help the victims of the Japanese quakes and tsunami relocate themselves and/or their children to New Zealand temporarily.

There’s quite a few big names backing it (both in New Zealand and Japan), and when we were approached about doing the site it seemed like the perfect opportunity to give something back to the people of Japan, who’ve been kind enough to give me visas and keep us employed for the past seven years :)

It was built on top of Rails Kitchen Sink (shameless plug) and took most of a weekend to get the main functionality in, with a few more evenings of tweaking as requirements solidified a little.

The good folks at Heroku were kind enough to provide 6 months free hosting (with extra dynos on top of the free plan), so the whole initiative has only cost a few hours of my time, plus a few hours for Toshi at Second Drip who was kind enough to do the Japanese translation.

If any of you are in New Zealand, feel free to sign up and help out!

Simple video remixes using The Echo Nest API

2011-01-05T00:00:00+13:00

JAN 05 2011

Simple video remixes using The Echo Nest API

Filed under api, echonest, music, and programming.

Here’s a couple of remix videos I did earlier in the year using The Echo Nest API.

I’ve been meaning to post these for a while, but finally decided to get off my ass and put them up since the folks at Echo Nest were kind enough to send me a T-shirt :

(source video one)
(source video two)

It’s pretty straightforward to create these.

Create an account over at The Echo Nest and grab an API key.
Download a couple of videos off of Youtube (or elsewhere)
Grab the vafromb.py example script that comes bundled with the Echo Nest developer sources.
Add a couple of lines near the top of the script (before the from echonest imports) with your API key :
```
from pyechonest import config
config.ECHO_NEST_API_KEY="<YOUR_API_KEY_HERE>"
```

Run the script on your input files :

> python vafromb.py <inputfilenameA> <inputfilenameB> <outputfilename> <Mix> [env]

The Mix parameter is a number between 0 and 1 that determines the relative mix of the resynthesized song and the original input A. i.e. a mix value of 0.9 yields an output that is mostly the resynthesized version.

A Mix of zero will contain only the original song, whereas a Mix of 1 will contain %100 resynthesized sound.

The ‘env’ flag applies the volume envelopes of the segments of A to those from B.

This flag is optional, and is kind of hard to explain if you don’t know what an envelope is – try it out and you’ll figure it out pretty quickly.

The hardest part is finding two videos that will mix well together – I’ve found that using one with a fairly sparse, strong rhythm track for inputfilenameA works quite well, with a more melodic track in inputfilenameB.

These examples barely scrape the surface of what you can do with The Echo Nest API – I highly recommend playing around with it yourself if you’re interested in music and computers.

A Christmas present from The Echo Nest

2011-01-05T00:00:00+13:00

JAN 05 2011

A Christmas present from The Echo Nest

Filed under api, echonest, music, and programming.

A couple of weeks before Christmas I got a mail from The Echo Nest developer team promising a free T-shirt for filling out a survey :

I didn’t expect that they’d send a shirt all the way to Australia (plus I thought I’d probably missed the cutoff), but they provide an amazing service so I thought I’d fill out the survey anyway.

A couple of weeks later, lo and behold, they came through with the goods!

If you have any interest whatsoever in music programming, I can’t recommend their API enough – they are doing some really incredible work.

Update : I’ve put up a couple of remixed videos, in case you’re wondering what kind of stuff the API allows you to do.

Face detection with OS X and python

2010-12-14T00:00:00+13:00

DEC 14 2010

Face detection with OS X and python

Filed under opencv, osx, programming, and python.

I’ve been messing about with face detection for a while, and was surprised how easy it is to get basic detection working out-of-the-box with open-source libraries.

Today we’ll look at a simple way to get started with face detection on OSX using python.

http://www.flickr.com/photos/rogue3w/2877522084/

We’re going to be using the OpenCV computer vision library, originally developed by Intel but subsequently released under the BSD license.

We’re going to use Homebrew to install everything. I’ve also had success installing OpenCV from MacPorts, as well as compiling it from source, but Homebrew seems to be the easiest and cleanest option these days.

> brew install opencv

This ran for about 5 minutes trying to install cmake, at which point I ran into an error :

ld: symbol(s) not found
collect2: ld returned 1 exit status
make[2]: [bin/ccmake] Error 1
make[1]: [Source/CMakeFiles/ccmake.dir/all] Error 2
make: *** [all] Error 2
Exit status: 2

There was a note in the homebrew error message pointing to this GitHub issue (I wish more package managers did this!) which had a few comments suggesting that updating Homebrew and reinstalling cmake would fix the problem :

> brew update
> brew install -v cmake

With this update done, installing opencv proceeded without problems :

> brew install opencv

You probably also need to update your PYTHON_PATH (thanks Duncan for pointing this out) :

> export PYTHONPATH="/usr/local/lib/python2.6/site-packages/:$PYTHONPATH"

You can add the export command to your ~/.profile file if you want to avoid having to do this in the future.

The Homebrew install should come with python bindings built-in – to make sure everything is working, start the python interpreter and type import cv :

> python
Python 2.7 (r27:82500, Sep 10 2010, 08:44:08) 
[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv
>>>

If you don’t get any errors, you’re good to go!

Next, we need a simple python script to do the actual face detection. Luckily, the OpenCV source comes with a bunch of examples – here’s their facedetect.py example :

#!/usr/bin/python
"""
This program is demonstration for face and object detection using haar-like features.
The program finds faces in a camera image or video stream and displays a red box around them.

Original C implementation by:  ?
Python implementation by: Roman Stanchak, James Bowman
"""
import sys
import cv
from optparse import OptionParser

# Parameters for haar detection
# From the API:
# The default parameters (scale_factor=2, min_neighbors=3, flags=0) are tuned 
# for accurate yet slow object detection. For a faster operation on real video 
# images the settings are: 
# scale_factor=1.2, min_neighbors=2, flags=CV_HAAR_DO_CANNY_PRUNING, 
# min_size=<minimum possible face size

min_size = (20, 20)
image_scale = 2
haar_scale = 1.2
min_neighbors = 2
haar_flags = 0

def detect_and_draw(img, cascade):
    # allocate temporary images
    gray = cv.CreateImage((img.width,img.height), 8, 1)
    small_img = cv.CreateImage((cv.Round(img.width / image_scale),
			       cv.Round (img.height / image_scale)), 8, 1)

    # convert color input image to grayscale
    cv.CvtColor(img, gray, cv.CV_BGR2GRAY)

    # scale input image for faster processing
    cv.Resize(gray, small_img, cv.CV_INTER_LINEAR)

    cv.EqualizeHist(small_img, small_img)

    if(cascade):
        t = cv.GetTickCount()
        faces = cv.HaarDetectObjects(small_img, cascade, cv.CreateMemStorage(0),
                                     haar_scale, min_neighbors, haar_flags, min_size)
        t = cv.GetTickCount() - t
        print "detection time = %gms" % (t/(cv.GetTickFrequency()*1000.))
        if faces:
            for ((x, y, w, h), n) in faces:
                # the input to cv.HaarDetectObjects was resized, so scale the 
                # bounding box of each face and convert it to two CvPoints
                pt1 = (int(x * image_scale), int(y * image_scale))
                pt2 = (int((x + w) * image_scale), int((y + h) * image_scale))
                cv.Rectangle(img, pt1, pt2, cv.RGB(255, 0, 0), 3, 8, 0)

    cv.ShowImage("result", img)

if __name__ == '__main__':

    parser = OptionParser(usage = "usage: %prog [options] [filename|camera_index]")
    parser.add_option("-c", "--cascade", action="store", dest="cascade", type="str", help="Haar cascade file, default %default", default = "/usr/local/Cellar/opencv/2.2/share/opencv/haarcascades/haarcascade_frontalface_default.xml")
    (options, args) = parser.parse_args()

    cascade = cv.Load(options.cascade)
    
    if len(args) != 1:
        parser.print_help()
        sys.exit(1)

    input_name = args[0]
    if input_name.isdigit():
        capture = cv.CreateCameraCapture(int(input_name))
    else:
        capture = None

    cv.NamedWindow("result", 1)

    if capture:
        frame_copy = None
        while True:
            frame = cv.QueryFrame(capture)
            if not frame:
                cv.WaitKey(0)
                break
            if not frame_copy:
                frame_copy = cv.CreateImage((frame.width,frame.height),
                                            cv.IPL_DEPTH_8U, frame.nChannels)
            if frame.origin == cv.IPL_ORIGIN_TL:
                cv.Copy(frame, frame_copy)
            else:
                cv.Flip(frame, frame_copy, 0)
            
            detect_and_draw(frame_copy, cascade)

            if cv.WaitKey(10) >= 0:
                break
    else:
        image = cv.LoadImage(input_name, 1)
        detect_and_draw(image, cascade)
        cv.WaitKey(0)

    cv.DestroyWindow("result")

One minor change I made was updating the path to haarcascade_frontalface_default.xml on line 117 – I’ve hard-coded in the path to the file installed by Homebrew in the example above for simplicity.

Save the script above as facedetect.py, grab a sample image to test it on, and you should be able to run it with regular mac python :

> python facedetect.py example.jpg

The command above will launch a python window containing the image, with red boxes surrounding detected faces.

Here’s the result of running the script on some random creative-commons photos tagged with ‘family’ on Flickr :

http://www.flickr.com/photos/stuckincustoms/2002931359/

http://www.flickr.com/photos/mareen/752734261/

http://www.flickr.com/photos/thomashawk/513666576/

http://www.flickr.com/photos/soldiersmediacenter/4308562775/

http://www.flickr.com/photos/freeparking/478631087/

http://www.flickr.com/photos/18854914@N04/2075741899/

As you can see, the results are mixed. Full-frontal faces work best (because we’re using haarcascade_frontalface_default.xml to do the detection), as do medium-resolution images. I’ve found that processing very high-res images can result in a lot of false positives, since the image has so much detail.

The example above can also process real-time video from the camera on your Mac – instead of using a filename as an argument, input the camera number you wish to use (zero for most Macs) :

> python facedetect.py 0

If you feel like digging in a little deeper, there’s various other training data sets in /usr/local/Cellar/opencv/2.2/share/opencv/haarcascades/ for detecting other shapes (such as full bodies, eyes, noses etc), which open up some cool possibilities.

Enjoy!

Jekyll plugins for categories, projects and sitemaps

2010-12-08T00:00:00+13:00

DEC 08 2010

Jekyll plugins for categories, projects and sitemaps

Filed under jekyll, plugin, programming, ruby, and web.

I’ve finally gotten around to cleaning up and releasing the source for a couple of the Jekyll plugins used to build this site.

You can find them all on the Jekyll Plugins project page, or browse the source.

As always, if you come across any problems please create a ticket and we’ll try to get it fixed as soon as possible.

Enjoy!

Project page generator

A generator that creates project pages for Jekyll sites from git repositories.

This was inspired by the project pages on GitHub, which use the project README file as the index page. It takes git repositories, and automatically builds project pages for them using the README file, along with downloadable zipped copies of the projects themselves (for example, all the project pages linked to from our project page were auto-generated with this).

The goal is to automate the construction of online project pages, keep them in sync with README documentation, and provide an up-to-date zip archive for download.

Category page generator

A generator that creates category pages for Jekyll sites (for example our plugin category).

I found one or two category page generators floating around on pastebin etc, but nothing that really achieved what I was after, so I ended up putting this script together.

sitemap.xml generator

A fairly simple generator that creates a sitemap.xml page for jekyll sites, suitable for submission to google etc.

jQuery SimpleColor plugin hits version 1.0

2010-12-07T00:00:00+13:00

DEC 07 2010

jQuery SimpleColor plugin hits version 1.0

Filed under announcement, javascript, jquery, plugin, programming, and web.

A couple of weeks ago Huga commented on the SimpleColor plugin with a nice suggestion – adding the selected color as text inside the color selection button. I finally got around to implementing this last night, and added a couple of extra options to support this :

displayColorCode:   Display the color code (eg #333333) as text inside the 
                    button. true or false.
                     default value: false

colorCodeAlign:     Text alignment used to display the color code inside
                    the button. Only used if 'displayColorCode' is true.
                     default value: 'center'

colorCodeColor:     Text color of the color code inside the button. Only
                    used if 'displayColorCode' is true.
                     default value: '#FFF'

The finished result looks something like this :

Since there’s been no bug reports for a while and the plugin does basically what I need, I also took the opportunity to upgrade it from version 0.9.0 to 1.0.0.

As always, if you come across any problems please create a ticket and we’ll try to get it fixed as soon as possible.

CompSci 101 - Big-O notation

2010-12-07T00:00:00+13:00

CompSci 101 – Big-O notation

I recently had a couple of Google interviews in Tokyo, and while preparing for them I ended up with a huge list of things I wanted to brush up on before the interview.

It turns out I didn’t get the job (next time!), but I thought I might be able to learn something anyway by working through the list and blogging about the main areas that companies like Google expect you to know.

I’ve grabbed the domain computerscience101.org (which currently redirects back here), and when I’ve collected enough posts I plan to throw everything up there as a kind of chapter-by-chapter interview-primer in the hope that it might help someone else out.

Without further ado, first on the list is Big-O notation:

So what is Big-O notation anyway?

Big-O measures how well an operation will “scale” when you increase the amount of “things” it operates on.

Big-O can be used to describe how fast an algorithm will run, or it can describe other behaviour such as how much memory an algorithm will use. For the purposes of this article, we’re only going to focus on Big-O as it relates to runtime.

For example, deciding whether a number n is even or odd will take the same time no matter how big n is. On the other hand, to find an item in an unsorted list of n items, you have to look at each item individually – the time taken will increase as the size of the list increases.

Common complexity cases

Let’s have a look at a few different cases, and what they mean in real-world terms:

Big-O	Operations for 10 “things”	Operations for 100 “things”
O(1)	1	1
O(log n)	3	7
O(n)	10	100
O(n log n)	30	700
O(n^2)	100	10000
O(2^n)	1024	2^100 – a lot!
O(n!)	3628800	100! – also a lot!

O(1)

O(1) means that no matter how large the input is, the time taken doesn’t change.

O(1) operations run in constant time. Some examples of O(1) operations are :

Determining if a number is even or odd.
Using a constant-size lookup table or hash table.

The following function will take the same time to execute, no matter how big array is :

def is_first_element_null(array)
  if array[0] == nil
    return true
  else
    return false
  end
end

O(log n)

Any algorithm which cuts the problem in half each time is O(log n).

O(log n) operations run in logarithmic time – the operation will take longer as the input size increases, but once the input gets fairly large it won’t change enough to worry about. If you double n, you have to spend an extra amount of time t to complete the task. If n doubles again, t won’t double, but will increase by a constant amount.

An example of an O(log n) operation is finding an item in a sorted list with a balanced search tree or a binary search :

def binary_search(array, value, low=0, high=array.size-1)  
  # if high and low overlap, nothing was found.
  return false if high < low
  # Determine the middle element.
  mid = low + ((high - low) / 2)
  # Split the result in half and search again recursively until we succeed.
  if array[mid] > value
    return binary_search(array, value, low, mid-1)
  elsif array[mid] < value
    return binary_search(array, value, mid+1, high)
  else
    return mid # Found!
  end
end

O(n)

O(n) means that for every element, you are doing a constant number of operations, such as comparing each element to a known value.

O(n) operations run in linear time – the larger the input, the longer it takes, in an even tradeoff. Every time you double n, the operation will take twice as long.

An example of an O(n) operation is finding an item in an unsorted list :

def contains_value(array, value)
  for entry in array
    return true if entry == value
  end
  return false
end

O(n log n)

O(n log n) means that you’re performing an O(log n) operation for each item in your input. Most (efficient) sort algorithms are an example of this.

O(n log n) operations run in loglinear time – increasing the input size hurts, but may still be manageable. Every time you double n, you spend twice as much time plus a little more.

Examples of O(n log n) operations are quicksort (in the average and best case), heapsort and merge sort.

def mergesort(list)
  return list if list.size <= 1
  mid = list.size / 2
  left  = list[0, mid]
  right = list[mid, list.size]
  merge(mergesort(left), mergesort(right))
end
 
def merge(left, right)
  sorted = []
  until left.empty? or right.empty?
    if left.first <= right.first
      sorted << left.shift
    else
      sorted << right.shift
    end
  end
  sorted.concat(left).concat(right)
end

O(n^2)

O(n^2) means that for every element, you do something with every other element, such as comparing them.

O(n^2) operations run in quadratic time – the operation is only really practical up to a certain input size. Every time n doubles, the operation takes four times as long.

Examples of O(n^2) operations are quicksort (in the worst case) and bubble sort.

The following function is an example of an O(n^2) operation, where every element in an array is compared to every other element :

def contains_duplicates(array)
    i = 0
    while i < array.size
        j = i + 1
        while j < array.size
            return true if array[i] == array[j] # Found a match!
            j += 1
        end
        i += 1
    end
    return false
end

O(2^n)

O(2^n means that the time taken will double with each additional element in the input data set.

O(2^n) operations run in exponential time – the operation is impractical for any reasonably large input size n.

An example of an O(2^n) operation is the travelling salesman problem (using dynamic programming).

O(n!)

O(n!) involves doing something for all possible permutations of the n elements.

O(n!) operations run in factorial time – the operation is impractical for any reasonably large input size n.

An example of an O(n!) operation is the travelling salesman problem using brute force, where every combination of paths will be examined.

Big-O Visualized

The graph below¹ maps number of operations (on the X-axis) to time taken (on the Y-axis). It’s pretty clear that an O(2^n) algorithm is going to cause problems quickly as the input size increases!

Amortized analysis

Some operations are said to run in amortized time :

If you do an operation say a million times, you don’t really care about the worst-case or the best-case of that operation – what you care about is how much time is taken in total when you repeat the operation a million times.

So it doesn’t matter if the operation is very slow once in a while, as long as “once in a while” is rare enough for the slowness to be diluted away. Essentially amortised time means “average time taken per operation, if you do many operations”.

Let’s take [an] example of a dynamic array, to which you repeatedly add new items. Normally adding an item takes constant time (that is, O(1)). But each time the array is full, you allocate twice as much space, copy your data into the new region, and free the old space. Assuming allocates and frees run in constant time, this enlargement process takes O(n) time where n is the current size of the array.

So each time you enlarge, you take about twice as much time as the last enlarge. But you’ve also waited twice as long before doing it! The cost of each enlargement can thus be “spread out” among the insertions. This means that in the long term, the total time taken for adding m items to the array is O(m), and so the amortised time (i.e. time per insertion) is O(1). ²

Exercises

Decide the Big-O complexity of the following examples :

def bubble_sort(list)
  for i in 0..(list.size - 1)
    for j in 0..(list.size - i - 2)
      if (list[j + 1] <=> list[j]) == -1
        list[j], list[j + 1] = list[j + 1], list[j]
      end
    end
  end
  return list
end

def factorial(n)
  if n.zero? 
    return 1 
  else
    return n * factorial(n - 1)
  end
end

def palindrome?(input)
  stack  = []
  output = ""
  input.each_char do |x| 
    stack.push x
  end
  while not stack.empty?
    output << stack.pop
  end
  return (output == input)
end

def sum_of_divisors(n)
  result = 0
  i      = 1
  while i < n
    if n % i == 0
      result += i
    end
    i += 1
  end
  return result
end

def is_prime(num)
  return false if num == 1
  for i in 1..(num-1)
    if num % i == 0
      return false
    end
  end
  return true
end

def word_occurrence(word, phrase)
  result = 0
  array  = phrase.split(' ')
  for item in array
    result += 1 if item.downcase == word.downcase
  end
  return result
end

Sources

A Beginners’ Guide to Big O Notation (rob-bell.net)
Amortized analysis (en.wikipedia.org)
Big-O for Eight Year Olds? (stackoverflow.com)
Big O notation (en.wikipedia.org)
Constant Amortized Time (stackoverflow.com)
Merge sort (en.wikipedia.org)
Table of common time complexities (en.wikipedia.org)
Time complexity graph (therecyclebin.wordpress.com)

Footnotes

¹ Improved graph kindly provided by Xorlev. The original graph was from The Recycle Bin

² Quote from Stack Overflow user Artelius

A weekend project - fromthecache.com

2010-11-03T00:00:00+13:00

NOV 03 2010

A weekend project – fromthecache.com

Filed under announcement, company, html5, programming, rails, ruby, sass, and web.

I was playing around on the weekend screen-scraping and analyzing word-frequencies for various sites (don’t ask), and was getting some slow responses (and accidentally got my IP blocked from one site when I hit them a few too many times).

Eventually I hit upon the idea of hitting Google Cache for each URL (the pages I was scraping had sequential ?id=xxx URLs so it was easy to automate), with the aim of speeding things up a bit and taking some load off the target sites.

With this in mind, I spent a few hours Saturday and Sunday developing fromthecache.com – it’s built on rails, and designed to provide transparent access to the Google cache, while fetching the original page as a fallback if necessary.

It occurred to me halfway through that it’s also useful for providing mirror links if a site gets slash-dotted – just put fromthecache.com/ in front of the URL and you have an instant cache link.

There’s a fairly good chance that the server’s IP will get blocked from Google for looking like a bot, but I’m hoping requests out of Heroku might come from a few different IPs and mix things up a bit.

You can view the demo at fromthecache.com, browse the source, or
download it from the project page and try it out for yourself.

Introducing Rails Kitchen Sink

2010-10-21T00:00:00+13:00

OCT 21 2010

Introducing Rails Kitchen Sink

Filed under announcement, company, html5, programming, rails, ruby, sass, and web.

I’m happy to announce the release of Rails Kitchen Sink, our rails-project-starter, which we will be building all future Rails apps off of.

It includes everything we commonly need when starting a new project, such as registration, login, password reset, authorization, HAML, SASS, CoffeeSript etc.

You can view the demo at http://railskitchensink.com/, or download it from the project page and try it out for yourself.

I’m considering including support for a couple of recurring-billing companies, but not sure if the interest is there – please let us know if you would find this useful. It would probably involve writing a separate gem to abstract out the details and provide a common billing interface.

As always, contributions (particularly translations) are much appreciated!

Static blogging the Jekyll way

2010-10-12T00:00:00+13:00

OCT 12 2010

Static blogging the Jekyll way

Filed under company, html5, jekyll, programming, ruby, sass, and web.

In building the new site, I decided to take the opportunity to try out something new. I’ve been using WordPress to power my blog for about 5 years, and while it has served me well, I wanted to try out something different. The main criteria for choosing a new blogging platform were :

It should serve true static pages, with no need for server-side processing.
It should be easy to store the entire site in Git.
It should support the concept of layouts – I wanted to have a single ‘template’ file that blog posts could easily inherit from.
It should support MarkDown or Textile for formatting posts, since I want to keep away from posting raw HTML as much as possible.
It should support comments.
I should be able to deploy updates to the site with a single command.

By patching together a few different open-source technologies, I’ve been able to achieve the above and more.

Jekyll

After trying out several different static site generators, I settled on Jekyll. In its own words :

Jekyll is a simple, blog aware, static site generator. It takes a template directory (representing the raw form of a website), runs it through Textile or Markdown and Liquid converters, and spits out a complete, static website suitable for serving with Apache or your favorite web server.

Jekyll has a healthy community built around it, and has a plugin system, which I figured would allow me to work around any shortcomings I came across. The fact that Jekyll is used to power GitHub Pages also helped sway my decision, since I know it isn’t going to go away anytime soon.

I won’t go into how to get your first Jekyll site going here, but the documentation on GitHub is excellent, and there’s a nice getting started guide to help you jump straight in.

Custom Jekyll Plugins

I ended up writing several custom Jekyll plugins to provide things that aren’t included by default :

A short generator to create a google-compatible XML sitemap automatically.
A generator to create category pages automatically (for example the Ruby category).
A more in-depth generator that takes git repositories, and automatically builds project pages for them using the README file, along with downloadable zipped copies of the projects themselves (for example, the jquery-simple-color plugin page). I only have a couple of these projects up at this point, but have quite a few queued up to release when time permits, so I wanted the process to be automated as much as possible.

The plugin documentation is a little sparse, but there are a couple of examples to get you going. I’ll be releasing the source of my own custom generators over the next couple of weeks, so stay tuned to RSS if you want more information.

HTML5 Boilerplate

Since this is the first site I’ve built with HTML5, I decided to use HTML5 Boilerplate as a starting-point, which bills itself as the professional badass’s base HTML/CSS/JS template for a fast, robust and future-proof site .

Overall it works very nicely, although things still look a bit wacky on Internet Explorer (my fault, not BoilerPlate’s). We’re serving the site with Nginx, so we miss out on some of the Apache .htaccess goodness that comes with HTML Boilerplate, but overall it’s nice and easy to dive in and get started with.

One minor niggle is that the meta tag for triggering Google Chrome Frame in IE doesn’t validate, which doesn’t really affect anything one way or the other, but it’s always nice to get the green tick from validator.w3.org.

Boilerplate was a cinch to get running with Jekyll – just a matter of dropping the CSS and JavaScript files in the right places, and editing the main layout.

Compass/Sass

After using Compass and Sass on a couple of projects in the past I was determined to use them again this time.

Compass is a stylesheet authoring framework that makes your stylesheets and markup easier to build and maintain. With compass, you write your stylesheets in Sass instead of CSS. Using the power of Sass Mixins and the Compass community, you can apply battle-tested styles from frameworks like Blueprint to your stylesheets instead of your markup.

The Compass/Sass combo is a huge time-saver when building CSS-heavy sites, and even on a reasonably minimal site like this it helps keep things nicely organized, readable and maintainable. Compass was relatively easy to get running with Jekyll – Bjørn Arild Mæland has a few tips that helped here.

The fact that Compass was designed to work with ruby/rails was an added bonus here, since it allowed me to handle all common Jekyll and Compass tasks in the same Rakefile.

Pygments

Jekyll uses Pygments to do code syntax highlighting. Again I was pleasantly surprised how easy this was to integrate, since it’s written in Python rather than Ruby. Installation on Leopard is as simple as :

$ sudo easy_install Pygments

Once it is installed, you need to generate the CSS required for highlighting :

$ pygmentize -S default -f html > _sass/3rd_party/pygments/default.scss

This generates the default color scheme, and drops it into a folder inside _sass (you may want to organize your SASS files differently). Once you have the CSS installed, you can highlight code in your posts by wrapping it in highlight tags :

{% highlight ruby %}
...code goes here...
{% endhighlight %}

Pygments supports a bunch of languages, and it doesn’t look terribly difficult to add your own if the need arises.

Rake

Being a huge fan of automation, I used Rake to script some of the common tasks involved with running the site. These shortcuts handle tasks like compiling the CSS, compiling the HTML, cleaning all build products, generating pygments CSS, and deploying the site.

namespace :jekyll do

  desc 'Delete generated _site files'
  task :clean do
    system "rm -fR _site"
  end
  
  desc 'Run the jekyll dev server'
  task :server do
    system "jekyll --server --auto"
  end
  
  desc 'Clean temporary files and run the server'
  task :compile => [:clean, 'compass:clean', 'compass:compile'] do
    system "jekyll"
  end

end

namespace :deploy do
  desc 'Deploy to production'
  task :prod => ['dev:off'] do
    system "cap deploy"
  end
  
end

namespace :compass do  

  desc 'Delete temporary compass files'
  task :clean do
    system "rm -fR css/*"
  end

  desc 'Run the compass watch script'
  task :watch do
    system "compass watch"
  end

  desc 'Compile sass scripts'
  task :compile => [:clean] do
    system "compass compile"
  end
  
end

namespace :pygments do  

  desc 'Delete pygments CSS files'
  task :clean do
    system "rm -f _sass/3rd_party/pygments/*.scss"
  end

  desc 'Generate pygments CSS'
  task :compile => [:clean] do
    system "mkdir -p _sass/3rd_party/pygments"
    system "pygmentize -S default -f html > _sass/3rd_party/pygments/default.scss"
  end

end

namespace :dev do

  desc 'Un-publish old posts to speed up development'
  task :on => ['jekyll:clean'] do
    system 'find . -name "*.textile" -exec sed -i "" "s|^published: true|published: false|g" {} \;'
    system 'find . -name "*.yml" -exec sed -i "" "s|^published: true|published: false|g" {} \;'
  end

  desc 'Re-publish old posts for deployment'
  task :off => ['jekyll:clean'] do
    system 'find . -name "*.textile" -exec sed -i "" "s|^published: false|published: true|g" {} \;'
    system 'find . -name "*.yml" -exec sed -i "" "s|^published: false|published: true|g" {} \;'
  end
  
end

Capistrano

I’ve used Capistrano on several projects in the past, and was very keen to use it again.

Capistrano is an open source tool for running scripts on multiple servers; its main use is deploying web applications. It automates the process of making a new version of an application available on one or more web servers, including supporting tasks such as changing databases.

Again, Capistrano was very easy to get going with Jekyll. I had to write a simple deploy.rb script to push the site to production :

require 'capistrano/ext/multistage'
set :stages, %w(production)
set :default_stage, "production"

set :application, "recursive-design"
set :domain, "#{application}.com"

f = "config/passwords.yml"
unless File.exists?(f)
  abort "Couldn't find config/passwords.yml"
end
set :passwords, YAML::load(File.open(f))

set :deploy_via, :copy
set :repository, "_site" 
set :scm, :none 
set :copy_compression, :gzip

set :use_sudo, false

namespace :deploy do

  desc <<-DESC
  A macro-task that updates the code and fixes the symlink.
  DESC
  task :default do
    transaction do
      compile
      update_code
      symlink
    end
  end
  
  task :update_code, :except => { :no_release => true } do
    on_rollback { run "rm -rf #{release_path}; true" }
    strategy.deploy!
  end
  
  desc "Compile the site and CSS. "
  task :compile do
    system "rake dev:off"
    system "rake jekyll:compile"
  end
    
end

… and a supporting production.rb file :

server "#{domain}", :app, :web, :db, :primary => true

set :user, passwords['production']['user']
set :deploy_to, "/var/www/#{domain}"

… as well as a passwords.yml containing login information for the production server.

Once Capsitrano and Rake are set up, deployment is as simple as doing a

$ rake deploy:prod

This cleans out all the old build products, rebuilds the site and associated CSS, grabs all my projects from Git, builds pages for them and adds a link to the latest zipped-up version, and pushes everything to the production server.

Disqus

Since Jekyll only generates static HTML, there is no support for comments. Luckily Disqus offers a great (free!) service to provide comments to your site via embedded JavaScript.

My only gripe with the service is that it has proved impossible to import all the comments from the old WordPress site into Disqus – I’ve tried their WordPress plugin, a direct upload of the XML exported by WordPress, as well as a site crawl, and nothing seems to have worked. I haven’t gotten around to contacting them about this yet, so possibly there’s a simple way to do it that I’m missing. For new content though, Disqus works great.

Feedburner

Once again I’m using FeedBurner to serve the RSS feed – it gives some nice stats, and looks a little better than a plain feed. Again, this was fairly straightforward to get running with Jekyll – I posted a how-to several weeks ago laying out how to get it all going.

So what’s the verdict?

With services like Disqus appearing, pure-static sites are now a viable option for blogging. I love being able to write in a regular text editor instead of the WordPress WYSIWYG editor, and love the safety-net that keeping everything in Git provides. Jekyll is extremely light-weight and pluggable – I don’t think getting Compass and Pygments going with WordPress would have been quite as straightforward.

Re-building the site (and researching the various options beforehand) really drove home to me how many great, open-source projects there are these days – the fact that I could just plug in all these disparate pieces to build a site in a weekend was a pleasant surprise.

HABTM before_filter in rails models

2010-10-03T00:00:00+13:00

OCT 03 2010

HABTM before_filter in rails models

Filed under database, habtm, quickie, rails, and ruby.

I’ve been playing about with Devise and CanCan for rails authentication and authorization recently – this great pair of posts from Tony Amoyal have helped a great deal.

I have a User model and a Role model, and wanted to automatically add the :user role to each user whenever roles are assigned. The roles were set in the controller as follows:

if @user.save
  @user.role_ids = params[:user][:role_ids]
    
  ...
    
end

… so I wanted to automatically add the :user role when @user.role_ids was set. Initially I tried adding a ‘before_filter’ to the User model:

class User < ActiveRecord::Base
  
  has_and_belongs_to_many :roles
  
  before_filter :add_default_roles
  
  ...
  
  def add_default_roles
    self[:role_ids] << Role.by_name(:user).id
  end
  
  ...
  
end

Obviously this didn’t work – setting the habtm field in this case happens after the model is saved (as you can see in the 2 lines from the controller above).

Next I tried to over-ride the roles setter in the model :

class User < ActiveRecord::Base
  
  has_and_belongs_to_many :roles
  
  ...
  
  def role_ids=(_role_ids)
    _role_ids << Role.by_name(:user).id
    self.role_ids = _role_ids
  end
  
  ...
  
end

The obvious (in hindsight) problem with this is that it is recursive – setting self.role_ids from inside the setter calls itself again, and results in a stack level too deep error. This recursive call can be fixed by changing the line

self.role_ids = _role_ids

with:

self[:role_ids] = _role_ids

or even

write_attribute(:role_ids, _role_ids)

This fixes the recursion problem, but still doesn’t add the default role properly, due to some complexities with the way habtm is implemented.

After a lot of digging around I came up with a solution that works using alias_method_chain:

class User < ActiveRecord::Base
  
  has_and_belongs_to_many :roles
  
  ...
  
  # Make sure all users have at least the :user role.
  def role_ids_with_add_user_role=(_role_ids)
    _role_ids << Role.by_name(:user).id
    self[:role_ids] = _role_ids
    self.role_ids_without_add_user_role = _role_ids
  end 
  alias_method_chain :role_ids=, :add_user_role  
  
  ...
  
end

After some testing this seems to do exactly what we need – when @user.role_ids is set, the role_ids_with_add_user_role function is called, and adds the :user role to the list of role ids before saving.

Excluding fields from rails json and xml output

2010-10-03T00:00:00+13:00

OCT 03 2010

Excluding fields from rails json and xml output

Filed under json, quickie, rails, rest, ruby, and xml.

Rails has a very handy ‘respond_to’ feature to render your pages in different formats :

respond_to do |format|
  format.json { render :json => @user }
  format.xml  { render :xml => @user }
  format.html      
end

However, by default it outputs every field in the model, which exposes some data (encrypted-password and password-salt) that I’d rather keep to myself :

<user> 
  <encrypted-password>$2a$10$a2K.PXEXpgpWH6MMLD/AFe84mjB4u3ZNf4MYTPten42LkPpjd.wZm</encrypted-password> 
  <created-at type="datetime">2010-10-02T00:12:43Z</created-at> 
  <updated-at type="datetime">2010-10-03T03:28:27Z</updated-at>  
  <id type="integer">12</id> 
  <password-salt>$2a$10$a2K.PXEXpgpWH6MMLD/AFe</password-salt> 
  <email>mail@recursive-design.com</email> 
</user>

Luckily, you can disable specific fields using the except option :

respond_to do |format|
  format.json { render :json => @user, :except=>  [:encrypted_password, :password_salt] }
  format.xml  { render :xml => @user, :except=>  [:encrypted_password, :password_salt] }
  format.html      
end

… which removes the offending fields from output :

<user> 
  <created-at type="datetime">2010-10-02T00:12:43Z</created-at> 
  <updated-at type="datetime">2010-10-03T03:28:27Z</updated-at>  
  <id type="integer">12</id> 
  <email>dave@recurser.com</email> 
</user>

If you find yourself doing this in a lot of different places, it may be easier to filter these options at the model level, by over-riding the ‘to_json’ and ‘to_xml’ methods :

class User < ActiveRecord::Base
  
  ...
   
   # Exclude password info from xml output.
   def to_xml(options={})
     options[:except] ||= [:encrypted_password, :password_salt]
     super(options)
   end
   
   # Exclude password info from json output.
   def to_json(options={})
     options[:except] ||= [:encrypted_password, :password_salt]
     super(options)
   end
  
  ...
  
end