ghickman

Expanding your RAID-5 Array

2013-01-06T00:00:00+00:00

Every so often the time comes around again where I run out of space on my fileserver. Thankfully I'm using RAID-5 so expanding the array is easy enough and the wonderful mdadm utility means I can have software RAID so the expansion won't take eons.

Expanding an array with mdadm is fairly straightforward:

Add the new drive to the array
Expand the array
Expand the file system

However there are some gotchas along the way that can make the process painful.

Preparation

Before I start working on any file system I prefer to know that nothing else is trying to access it so I turn off file sharing services as the first order of business.

Then unmount the volume and stop the array:

sudo umount /mnt/media

sudo mdadm --stop /dev/md0

Install the New Drive

This has always been the most annoying part I've found with expansions. Quite frequently the drive labels will move around. This is no doubt motherboard related and isn't an issue once you're aware of it! But before then it can be confusing and potentially hazardous.

Add the New Drive to the Array

Now that the drive is physically in place and you know the label you can get on with the fun stuff and add the drive to your array.

sudo mdadm --add /dev/md0 /dev/sdf

Replace /dev/md0 with your array's volume label /dev/sdf with your drive label.

Note: Some people prefer to partition drives before adding them to the array. Personally I've never had any issues with unpartitioned devices.

Expand the Array

Now tell the array to use the new drive:

sudo mdadm --grow --raid-devices=4 /dev/md0

Note: This part can take anywhere from hours to days, or more, depending on the speed of your machine and the size of disk you're adding to the array.

Expand the File System

Finally expand your filesystem to fill the new array:

sudo resize2fs /dev/md0

Finishing Touches

The last few bits are reversing what you did before working on the array:

sudo mount /dev/md0 /mnt/media

sudo mdadm --start /dev/md0

Don't forget to re-enable any services, such as samba or AFP, that you disabled.

How to Install Raspbmc when raspbmc.com is Down

2012-08-09T00:00:00+01:00

raspbmc.com down for you? Never fear, the internet has your back.

Install Script

Copy and paste the python code (starts with #!/usr/bin/python) from here into a file and save it as install.py.

Set permissions on it: chmod +x install.py.

When you've downloaded the binary (below) run it with sudo ./install.py

Pre-Download the Install Binary

The install script will use installer.img.gz if it finds it in the same directory so grab that off the University of Arizona mirror:

curl -O http://mirrors.arizona.edu/raspbmc/downloads/bin/ramdistribution/installer.img.gz

Now run the install script and you're away!

A Big Thank You

To Sam Nazarko for all his hard working actually making Raspbmc! Also to the various people providing mirror services for the install binary.

Problems?

It's more than likely that the information above will be out of date pretty soon after I post this entry but don't let that deter you. You can always browse raspbmc.com using Google's Cache by placing any URL from the site after http://webcache.googleusercontent.com/search?q=cache:.

You should get automatically routed to a download mirror by going to:

http://webcache.googleusercontent.com/search?q=cache:http://download.raspbmc.com/

Then find the install image, it's path should look like

/downloads/bin/ramdistribution/installer.img.gz

How we use GitHub Flow at Incuna

2012-07-29T00:00:00+01:00

Our team at Incuna has been growing rapidly of late and like most businesses at this stage of growth we've had to come up with some development flows to cope with the influx of new Incunanauts. Currently we have three (technical) teams: Design, Front End and Back End with work flowing through them in roughly that order. (Technically we have an Ops team, but really that's just me part time).

We currently use GitHub to host our code, track issues and do pull requests with hub for extra features like attaching code to an issue to make a pull request.

Branching

When do we branch?

Anything other than a trivial change goes on a branch so it can be pull requested (more on this later). Following GitHub Flow we try to keep a project's master branch production ready which branching allows us to do easily. However, sometimes we have a large, long running, feature to implement which requires it's own master feature branch. This is a bit more involved as you have to make sure you pull request onto the right branch but otherwise works well.

The most common problem I've seen with this style of branching (as with all programming) is "What do I call my branches". We try to name our branches directly around the piece of work, even if it isn't immediately obvious to every developer, that's what git log is for.

Pull Requests

As an extension of our branching we try to use pull requests as our main area of code review. When we first started using this as a formal method it worked quite well but more recently we've started to see the benefit of using the pull request as a forum of conversation around a bug/feature. We're still slowly moving Designers and Account Managers onto this flow and it seems to work well once they get into it.

Code Review

While pull requests are great it's been important for us to avoid hard rules for this sort of work flow. Even if the code has been done on a branch it's quite feasible for someone to have migrations that break the database or a block of code that everyone else needs ASAP to carry on. In this situation we bring in the Master Code Reviewers [read: the nearest developer who is ideally not on your project, thus fresh set of eyes].

The most important part of this for us is to have more than one set of eyes look at new code, ideally someone who isn't familiar with it. This way we're more likely to avoid obvious bugs and create maintainable code.

Testing

Always a sticky subject, but we're slowly getting better at it. Charlie & Marc have made great headway in setting up our large internal libraries with proper test suites and a general purge of all fixture based unit tests in favour of factories using the very awesome Factory Boy. I'm hoping Travis CI will bring out pro/private accounts soon as their testing suite would suit our pull request flow perfectly and we're already moving our open source projects to them.

What Can We Do Better?

Deployments.

Our current situation is sharded across three different service but unfortunately this is due to the nature of our business as customers have differing needs. Currently I do the majority of the deployments. However, it can be easier. Eventually I want any technical person to be both capable and willing to deploy an existing site.

Testing Culture

I think this is something that will come over time and is really a social problem, especially as we begin to firm up the ways in which we actually test projects, it should become second nature.

The Front End team has been talking about setting up testing for our Javascript projects for a while along with Selenium which I'd really like to see in action.

Conclusion

Our workflow is centred around being social and avoiding project specialists (i.e. only one person knows about a project). I think the current situation is very good and we're improving upon it all the time, but it can always be better. Hopefully time and new Incunanauts will show we're on the right track!

TL;DR

Be social, use good tools and don't have rules just best practices.

Incuna is hiring!

Setup Single Sign On in Django using Google OAuth2

2012-07-22T00:00:00+01:00

We recently had our second Incuna Hack Day where Charlie and I made the decision to start breaking up the main internal site, our venerable Dashboard. It was well on its way to becoming a monolithic beast and the only thing that had stopped me breaking it up before was an easy way to add Single Sign On to the many apps it would become.

Enter Django Social Auth. A cover-all-the-bases authentication app that provides backends for pretty much every service you can think of.

This write-up is based on my experience setting up a few internal apps so it's fairly opinionated towards that goal, but hopefully it's still adaptable to other situations!

Setup your App in Google's API Console

In Google's API Console create a new project.

You don't need to turn on any extra services, so go directly to API Access and hit the giant blue button to get started.

Enter your Product name and the URL to a logo if you have one. These are the details users will see when they authenticate via Google. Click Next.

Now set up the credentials for your application (this is per environment due to the redirect URI). The example below is for development, but only the hostname needs to change between environments.

Click more options to set the callback path. The default Social Auth URL for this is /complete/google-oauth2/.

Click Create client id and grab your Client ID/Client secret combo for the next step.

Setup Django Social Auth

Add social_auth to your INSTALLED_APPS and the other settings below:

AUTHENTICATION_BACKENDS = (    'social_auth.backends.google.GoogleOAuth2Backend',    'django.contrib.auth.backends.ModelBackend',)LOGIN_REDIRECT_URL = '/'
GOOGLE_OAUTH2_CLIENT_ID = os.environ['GOOGLE_OAUTH2_CLIENT_ID']GOOGLE_OAUTH2_CLIENT_SECRET = os.environ['GOOGLE_OAUTH2_CLIENT_SECRET']GOOGLE_WHITE_LISTED_DOMAINS = ['incuna.com']SOCIAL_AUTH_USER_MODEL = 'auth.User'
            view raw            settings.py            This Gist brought to you by GitHub.          

Here I whitelist our Google Apps domain to only allow authentication by users from work email addresses and tell Social Auth to use the auth.User model when creating new users which it will do by default (I believe you can turn this off with another setting). This lets met forget about registration completely which is perfect for internal applications.

Make sure you've set GOOGLE_OAUTH2_CLIENT_ID and GOOGLE_OAUTH2_CLIENT_SECRET in your environment when you do runserver or you'll get the crappy settings error message Unknown command: 'runserver'. You can avoid this using .get() instead of square braces notation when getting the Google credentials however the error may be more archaic. I'm currently favouring square braces notation and getting used to fixing my environment!

Create Some Basic Views

from django.core.urlresolvers import reversefrom django.contrib import messagesfrom django.http import HttpResponse, HttpResponseRedirectfrom django.views.generic.base import Viewfrom social_auth.backends.exceptions import AuthFailedfrom social_auth.views import complete

class AuthComplete(View):    def get(self, request, *args, **kwargs):        backend = kwargs.pop('backend')        try:            return complete(request, backend, *args, **kwargs)        except AuthFailed:            messages.error(request, "Your Google Apps domain isn't authorized for this app")            return HttpResponseRedirect(reverse('login'))

class LoginError(View):    def get(self, request, *args, **kwargs):        return HttpResponse(status=401)
            view raw            views.py            This Gist brought to you by GitHub.

Social Auth requires you add a view for when login fails. So far this hasn't been an issue for me so I've done the pure basics here with LoginError.

The second view was to cope with the whitelisting of domains which, pleasingly, raises an AuthFailed exception when you try to authenticate with a domain not in the whitelist.

Now all we need is to plumb this in with some URLs:

from django.conf.urls import *from django.contrib import admin
from .views import AuthComplete, LoginError

admin.autodiscover()
urlpatterns = patterns('',    # some other urls
    url(r'^admin/', include(admin.site.urls)),    url(r'^complete/(?P<backend>[^/]+)/$', AuthComplete.as_view()),    url(r'^login-error/$', LoginError.as_view()),    url(r'', include('social_auth.urls')),)
            view raw            urls.py            This Gist brought to you by GitHub.          

Profit

I usually put login_required on the whole site with a little piece of middleware (like this or this) but otherwise that's it!

Generating unique slugs with Django

2012-07-15T00:00:00+01:00

This is something I've come across a couple of times over the last few months and each time ended up acking through various projects so putting this up here to save myself the pain next time!

There are two versions to this method - model specific and a more generic one intended for a utils module if you're going to use it on more than one model.

Neither of these methods is built to scale, but since I've never worked at scale this hasn't been an issue. However you've been warned!

The functions make the slug unique by appending a count to the end and checking the length is still valid for the slug field.

Model Specific

from django.template.defaultfilters import slugify
@classmethoddef generate_slug(self, name):    count = 1    slug = slugify(name)
    def _get_query(slug):        if MyModel.objects.filter(slug=slug).count():            return True
    while _get_query(slug):        slug = slugify(u'{0}-{1}'.format(name, count))        # make sure the slug is not too long        while len(slug) > MyModel._meta.get_field('slug').max_length:            name = name[:-1]            slug = slugify(u'{0}-{1}'.format(name, count))        count = count + 1    return slug
            view raw            specific.py            This Gist brought to you by GitHub.          

example usage: MyModel.objects.create(name='foo', slug=MyModel.generate_slug('foo'))

Generic

from django.template.defaultfilters import slugify
def generate_slug(cls, value):    count = 1    slug = slugify(value)    if not isinstance(cls, type):        cls = cls.__class__
    def _get_query(cls, **kwargs):        if cls.objects.filter(**kwargs).count():            return True
    while _get_query(cls, slug=slug):        slug = slugify(u'{0}-{1}'.format(value, count))        # make sure the slug is not too long        while len(slug) > cls._meta.get_field('slug').max_length:            value = value[:-1]            slug = slugify(u'{0}-{1}'.format(value, count))        count = count + 1    return slug
            view raw            generic.py            This Gist brought to you by GitHub.          

example usage: AnotherModel.objects.create(name='bar', slug=generate_slug(AnotherModel, 'foo'))

Extending OS X Lion's Maxfiles Limit for Neo4J

2012-02-25T00:00:00+00:00

After upgrading to Neo4j 1.6 today I got this warning when starting the server:

Detected a limit of 512 for maximum open files.

A limit of 40000 is suggested instead and a quick look on Google suggests the use of launchctl limit or ulimit -n. However the results I found were split between how Linux deals with this issue (ulimit -n) and pre Lion fixes that involve using unlimited as the hard limit (which hasn't worked since Leopard).

Setting a higher limit on lion is fairly simple. Edit launchd.conf like so (it might not exist):

sudo vim /etc/launchd.conf

Then add this line to it:

limit maxfiles 1000000 1000000

This tells Lion to set both the soft and hard limit (in that order) to 1000000. I picked such a high value to avoid any issues of the same nature in the future.

Make sure you restart afterwards for the changes to take effect.

Setup Gitalist with Gitolite on Nginx

2011-07-24T00:00:00+01:00

I recently gave Github's paid service a go when my vimrc ended up needing some passwords in it. While I'm a big fan of Github and what it's done for the Git community as a whole I just can't justify paying the £5 a month so I can use my vimrc at home, work and a few servers. Of course the downside is the loss of being able to quickly view code on the web, but as fate would have it Twitter came to my rescue within a couple of days via the sagely Joel Moss.

Gitalist - a modern git web viewer http://j.mp/pJEQtNless than a minute ago via Reeder Favorite Retweet ReplyJoel Moss
joelmoss

Diving into Gitalist there were a couple of surprises, least of all it's written in Perl. Perl?! That's a dead language right? (Unless you're slashdot). However playing around with the demo (guys, please up whatever server you're running that on, it's dire) was great, not to mention it looks really slick.

Gitalist also presented an opportunity to coax my workplace from Mercurial/Bitbucket (Github's corporate pricing has so far been the major reason not to use Git) onto Git, but to do so some sort of access control would be needed, thus Gitolite.

I've tested these instructions on Ubuntu Server 10.04 so they should work reasonably well on other Ubuntu versions and child distros.

Gitolite

Gitolite is an access control system for Git repositories, a natural successor to Gitosis, providing fine grained control on a per branch basis. I won't bother going into much detail as there's so much to it and Sitraramc provides a far more comprehensive description on the Github page.

I followed the root install instructions with a couple of caveats:

I specify the folder locations when running src/gl-system-install as it didn't seem to use the default one's listed for me:

src/gl-system-install /usr/local/bin /usr/local/share/gitolite/conf /usr/local/share/gitolite/hooks

When adding the git user, rather than just doing a plain useradd git I set some options:

sudo adduser --system --shell /bin/bash --gecos 'git version control' --group --disabled-password --home /home/git git

and then run the su - git command with sudo so you can enter your own password.

Check you can push your projects to the server and you're away!

Gitalist

Installation (CPAN)

After some fruitless attempts to install from source and bootstrap I turned to the IRC channel where a Big Damn Hero pointed me at CPAN as the "Way to Go".

First of all CPAN needs a little love. By default it asks you what to do when it finds a dependency it doesn't have.

"Module X is required, shall I install it?" Well… yea…less than a minute ago via Twitter for Mac Favorite Retweet ReplyGeorge Hickman
ghickman

Thankfully it's easy enough to configure CPAN to follow the default options with the prerequisites_policy option. Open the CPAN console by running cpan.

Note: If CPAN hasn't been configured before you'll be asked if you want it to "configure as much as possible automatically", choose yes then follow the instructions below. However if you do want to go through them manually you'll be asked for a local CPAN mirror after the proxies section.

To configure type the following into the CPAN console:

o conf prerequisites_policy follow

then

o conf commit

This sets the prerequisites_policy to follow the default option for each dependency. Thanks to Mithaldu on #gitalist for helping me out and saving hours of tedium.

Finally it's time for some installations! First up is YAML, which isn't required, but every install complains that it's not there because all the package descriptors are written in it. Installing packages from CPAN is as easy as:

install YAML

Note: CPAN might update itself at this point, which can take a significant amount of time depending on the power of your machine.

When that's done, it's time for Gitalist:

install Gitalist

Unfortunately some questions still come up, so it's not a completely unattended installation and did take quite a while for me (at least an hour). Default answers seem fine though.

Test the install by running sudo gitalist_server.pl and having a look at http://<server>:3000/.

If you get an error about the location of the config then try the methods suggested here. I found none of these worked for me so I grabbed the source from Github and copied gitalist.conf to /usr/local/share/perl/5.10.x/Gitalist/.

Later on we'll setup Gitalist with FastCGI, at which point you'll need Perl's FCGI::ProcManager installed as it's the default "Process Manager" for the Gitalist FastCGI script:

install FCGI::ProcManager

Combining with Gitolite

Combining Gitolite and Gitalist is as "simple" as pointing Gitalist at Gitolite's repository directory. Gitolite is running under the git user and stores the repositories under /home/git/repositories/ which won't be accessible to you under another user. The easiest way around this is to run the gitalist_server.pl command as the git user like so:

sudo -u git gitalist_server.pl --repo_dir /home/git/repositories/

Of course you don't want to be stuck with a command running in the terminal all the time, or having to suffix it with & just to have it run in the background so we'll setup Supervisor to handle all of that for us.

Supervisor

Supervisor looks after a process and can be configured to perform useful duties like autorestarting and running your process under a different user, both of which we're going to take advantage of. Thankfully supervisor can be installed with ease, like so:

sudo aptitude install supervisor

Next you'll need a config file for Gitalist:

sudo vim /etc/supervisor/conf.d/gitalist.conf

Paste in the following - setting the path to your gitalist_server.pl to the appropriate place if it's not in the default location.

[program:gitalist]command=/usr/local/bin/gitalist_fastcgi.pl --listen /var/run/gitalist/gitalist.sock --nproc 2 --pidfile /var/run/gitalist/gitalist.piduser=gitautostart=trueautorestart=trueredirect_stderr=true

view raw supervisor This Gist brought to you by GitHub.

I've put the socket and pid files in /var/run/ since Gitalist is installed via CPAN into and doesn't really have an install directory as such so that's the next logical place. However you'll need to create the gitalist directory there and chown it to your git user so it can be written to by the FastCGI script (which is now running under the git user). The --nproc switch tells the script how many processes to run, like Nginx's workers directive. To see all the options run /usr/local/bin/gitalist_fastcgi.pl --help in your terminal.

Open up Supervisor's nifty console with sudo supervisorctl and tell it to update so that it uses your Gitalist config (you'll need to do this after any updates to a configuration file). status will show you a list programs you've setup which you can start, stop, restart and tail (for program output, with -f for continuous output). The gitatlist server should now be running under Supervisor, check with the tail gitalist to make sure there are no errors in the output.

Gitalist Config

Since we're using FastCGI to pass requests from Nginx through to Gitalist we'll use the Gitalist config file (you can't pass Gitalist configuration values to the FastCGI script). Open it up in your favourite editor:

sudo vim /usr/local/share/perl/5.10.1/Gitalist/gitalist.conf

and set the repo_dir option to /home/git/repositories/:

name Gitalist
<Model::CollectionOfRepos>    #git /path/to/git    # Configure this to where your repositories are.    repo_dir /home/git/repositories/</Model::CollectionOfRepos>
sitename "Gitalist"
<paging>  log = 20  summary = 17</paging>
# Support gitweb patches action.<patches>  max = 16</patches>

view raw gitalist.conf This Gist brought to you by GitHub.

Nginx

Create yourself a virtual host in nginx's sites-available directory and add the following, changing the server name to something suitable:

server {    server_name www.example.com;    rewrite ^/(.*) http://git.example.com/$1 permanent;}   
server {    server_name example.com;
    access_log /var/log/gitalist/access.log combined;    error_log /var/log/gitalist/error.log;
    location / {        include /etc/nginx/fastcgi_params;        fastcgi_pass unix:/var/run/gitalist/gitalist.sock;    }}

view raw vhost This Gist brought to you by GitHub.

I've setup the logs under /var/log/gitalist/ for the same reason as the socket and the pid files, again you'll have to create that directory but make it writable by www-data so Nginx has access to it.

Link the virtual host into sites-enabled and test with sudo nginx -t to check there are no errors, then reload nginx sudo /etc/init.d/nginx reload (or use upstart) and your Gitalist should now be accessible on your domain!

Notes

FastCGI vs. Reverse Proxying

While getting FastCGI setup I toyed with Nginx as a reverse proxy to the one or more instances of the Catalyst development server, but had issues with hiding the port number and it felt a bit wrong to use a development server in production.

Gitalist Install Directory

Having installed Gitalist via CPAN it lives under /usr/local/share/perl/5.10.1/Gitalist/ which seems a bad place to store a socket file, a pid file or any logs which is why I chose to put them all under /var/.

Extra Reading

Adding Django Inline Forms with Javascript

2011-07-10T00:00:00+01:00

Using Django formsets on a new section for a project at work I came across something I haven't considered before: adding new inline forms to a formset without reloading the page. Javascript was the obvious choice. Two pages required the functionality: one with six formsets, the other with one, which meant the solution needed to take into account formset prefixes. I found various solutions around the Internet, but all were outdated (lots of table based layouts!) or not generic enough (didn't deal with multiple formsets).

The idea behind the script is pretty simple: duplicate an existing form then update the form counter.

function add_inline_form(prefix) {    var count = parseInt($('#id_' + prefix + '-TOTAL_FORMS').val(), 10);    var last_form = $('.' + prefix + ':last');
    var new_form = last_form.clone(false).html(last_form.html().replace(          new RegExp(prefix + '-\\\\\\\\d-', 'g'), prefix + '-' + count + '-'));    new_form.find('input[type="text"], textarea').each(function () {        $(this).val('');    });    new_form.hide().insertAfter(last_form).slideDown(300);
    // Update the total form count    $('#id_' + prefix + '-TOTAL_FORMS').val(count + 1);
    // re-initialise triggers
    return false;}
var regex = /(?:inline\\-form) ([\\\\w\\-]*) (?:add|existing)/;$('.add-inline').each(function () {    var match = regex.exec($(this).closest('.body').find('.inline-form').attr('class'));    if (match && match.length > 1) {        $(this).click(function () {            return add_inline_form(match[1]);        });    }});
            view raw            add_inline.js            This Gist brought to you by GitHub.          

Which is setup to work on html that looks like this:

Note: I'm using Django UniForm to output the form in <div>'s

<div class="body">    {{ prefix_formset.management_form }}    {{ prefix_formset.non_form_errors }}    {% for form in prefix_formset.forms %}    <div class="inline-form prefix">        {{ form|as_uni_form }}    </div>    {% endfor %}    <div class="new-inline">        <input type="button" class="add-inline" value="Add another form" />    </div></div>
            view raw            forms.html            This Gist brought to you by GitHub.          

So How Does it Work?

Skip to Gotchas if you already understand my javascript

I start by looping my form class add-inline (you can call it anything you like) and then running the regular expression from line 20 to find the form prefix, which is another class on the form. It expects the classes applied to the form in the order add-inline <formset prefix> add/existing. The add/existing bit on the end isn't necessary but gives you an example of where to put any other classes that might exist on your form. If you don't want to use it, just remember to remove it from the regex! Inside the add_inline_form function I'm grabbing the count from the hidden div and the last form with jQuery's :last selector.

When new_form is set I'm using jQuery's clone method to take a copy of the last form and grab the raw html. The false passed into clone tells it to ignore any triggers and binds. The regex sets the correct count in the element ids. The next line clears the contents of every element in the new form since clone will pull this in with the elements. The new form is hidden then added after the last form with a nice little bit of slideDown candy for some UI goodness.

Finally increment the form count and return false so the form isn't submitted.

Gotchas

So now you've got your nice little bit of javascript (if I don't say so myself) all setup to grab the last form in each formset, duplicate it and add it below the last form. However there are some caveats to this...

The script grabs the last form in the formset every time so say you want the first form to differ from the rest (as I did) and implement this in the templates then you'll get the non-standard first form duplicated each time. The easiest way around this is to check the count and add an if to look for the first form.

Nose Tests - no such option

2011-04-30T00:00:00+01:00

Some time ago now I added proper command line options to Tv Renamr, followed shortly by a test suite with Python's Nose. Along the way I ran into a bug that has frustrated me for a long long time, until today when I finally found a work around - hooray, go me!

The bug is hardly a game stopper as it only affects the UI candy on my tests. Simply put, I couldn't use options with nose. Every time I did, an error was thrown saying no such option existed for nose while displaying the usage string for tvr:

This initially prompted me to split the front end script's options into a separate file which didn't solve the issue, but did wonders for cleaning up the codebase.

This seems to have affected at least one other person, whose post finally gave me the lightbulb today. Ionel mentions in his post "Luckily, setuptools has aliases for commands and instead of" and a snippet of a config file. It's only taken me a couple of years (I've tried to fix this at least 3 times now and always drawn blanks sadly) to realise that he was using a config file. A quick google for the appropriate section in the nose docs turned up enough information to construct a useful rc file:

[nosetests]with-spec=1spec-color=1with-growl=1

view raw .noserc This Gist brought to you by GitHub.

And so I finally have my lovely coloured output, with Growl integration, back. This is shaping up to be quite the productive weekend.

Using Passenger with RVM and Sinatra

2011-04-04T00:00:00+01:00

After making the move to Linode (finally!) I had some issues with getting the contact pages on this site and Penderry up and running. RVM was throwing an error about the sinatra gem missing. A quick scan of the error message and it was obvious that Passenger wasn't using my gemsets.

A bit of googling turned up this little snippet which, when placed in app/config/, will tell RVM to use the gemset associated with the folder.

if ENV['MY_RUBY_HOME'] && ENV['MY_RUBY_HOME'].include?('rvm')  begin    rvm_path     = File.dirname(File.dirname(ENV['MY_RUBY_HOME']))    rvm_lib_path = File.join(rvm_path, 'lib')    $LOAD_PATH.unshift rvm_lib_path    require 'rvm'    RVM.use_from_path! File.dirname(File.dirname(__FILE__))  rescue LoadError    # RVM is unavailable at this point.    raise "RVM ruby lib is currently unavailable."  endend
            view raw            config/setup_load_paths.rb            This Gist brought to you by GitHub.          

Ecohandle Shopping Bag

2011-01-08T00:00:00+00:00

This year's Christmas Haul brought me an interesting little gem: The Ecohandle Shopping Bag. I got round to giving it a bit of a go today when I did a veg shop and thought doing a review on it would be fun.

So, first things first - what is it?

It's a shopping bag, duh. The bonus here though is that the bag part compacts down into the handle for when you're going to the shop. Nice and neat. If you feel like a real glutton for punishment on your forearms (I'll get to that properly in a minute) you can add extra bags. There are two slots with retractable clips that you can put the bags into, assuming you can carry them of course. I managed to get at least 5 kg into the Ecohandle bag alone without pushing it's limits! Getting it home was the bigger issue issue as I had to keep swapping hands when my grip muscles died.

The Good

You're saving bags...thus, Puppies.

The bag is really quite strong. I was worried that packing it full might have put too much strain on it, but it held up surprisingly well.

The Bad

The embossed name. When holding it in my left hand I found the edge of the embossed text rubbed against my finger tips in such a small way that it left them raw within a couple of minutes of use. It's certainly not a deal breaker, but smoother letters would be appreciated.

The Fence

It's a handle. Whereas you would normally have your bags cutting a nice line across your fingers - pulling directly down on your arm, you've got to actually grip the Ecohandle. I ended up swapping hands every few minutes and cursing my decision to put 5Kg+ of shopping in it. Good workout though (thus why it's on The Fence©).

TL;DR

It's a reusable shopping bag that packs down into it's own handle. You can attach other shopping bags and it'll give you a great workout on your grip muscles.

Creating a Contact Page for Jekyll with Sinatra

2010-09-26T00:00:00+01:00

Freneticvirus

Jekyll is a fantastic static site generator with a great little community of modifications around it and I've used it for my own blog and Penderry.com. However my biggest problem with it is the lack of a way to deal with a contact form. Of course, this really isn't Jekyll's fault as it's only built to create HTML pages.

Enter Sinatra stage left.

Sinatra is a rad little web framework, well a micro framework really - in fact it's so small the Hello World app is 5 lines long! It's also perfect for creating a contact form.

To combine the two I needed both of them to display the same layout since I didn't want to have to maintain two. The contact page needed the same layout as the portfolio page, so no sidebar. The easiest way to do this is to plug your Sinatra application into Jekyll's layout mocking up any Jekyll specific objects/variables you need to (like page). I did give using a header and footer includes a go but it broke validation, pah.

My test app is hosted on Github for your viewing pleasure but I'll go through the important parts here too.

!!!%html  %head    %title Jekyll's Contact Form    %meta{:"http-equiv" => "Content-type", :content => "text/html; charset=utf-8"}        %link{:rel =>'shortcut icon', :href=>'/favicon.ico', :type=>'image/x-icon'}    %link{:rel=>'stylesheet', :type=>'text/css', :href=>'/stylesheets/master.css', :media=>'screen'}
  %body    %header      %h1 Jekyll's Contact From brought to you by Sinatra
    %nav      %ul        %li          %a{:href=>'/', :title=>'Home'} Blog        %li          %a{:href=>'http://localhost:4567/contact', :title=>'Contact'} Contact        %section#content== #{content}          %footer      %div        %section#jekyll          Powered by           %a{:href=>"http://github.com/richguk/jekyll", :title=>"Jekyll"} Jekyll        %section#from          A          %a{:href=>"http://ghickman.co.uk", :title=>"GHickman Blog"} GHickman          project
            view raw            default.haml            This Gist brought to you by GitHub.          

Nothing special here, just a bog standard Jekyll layout (well almost, this one is in HAML as I use RickGuk's Jekyll fork) with a HAML interpreter bolted on.

The contact page itself is an HTML(5) form that displays some extra bits if we find errors. The test app is done using some HTML 5 specific elements like the email input box but there's no big leaps from HTML 4.

#contact  %form{:method=>"post"}    %fieldset      .input        %label{:for=>"name"} Name        %input{:name=>"name", :type=>"text", :id=>"name", :class=>('error' if !@errors[:name].nil?), :required=>{}}      - if !@errors[:name].nil?        .error          %p=@errors[:name]            .input        %label{:for=>"email"} Email Address        %input{:name=>"email", :type=>"email", :id=>"email", :class=>('error' if !@errors[:email].nil?), :required=>{}}      - if !@errors[:email].nil?        .error          %p=@errors[:email]            .input        %label{:for=>"message"} Message        %textarea{:name=>"message", :id=>"message", :rows=>'5', :class=>('error' if !@errors[:message].nil?), :required=>{}}      - if !@errors[:message].nil?        .error          %p=@errors[:message]            #submit.input        %input{:type=>"submit", :value=>"Send"}
            view raw            contact.haml            This Gist brought to you by GitHub.          

The actual Sinatra application is nice and simple given that we're plugging it into a different layout.

require 'rubygems'require 'sinatra'require 'pony'require 'haml'
set :haml, {:format => :html5}set :public, File.dirname(__FILE__)set :views, File.dirname(__FILE__)
# Create the page class and give it a title of Contact for the layoutclass Page  def title    'Contact'  endend
def contact  # create the variables that the layout will expect  page = Page.new  content = haml :contact    # render the contact page using jekyll's layout and with our mock jekyll vars  haml :contact, :layout=>:'_layouts/default', :locals=>{:page=>page, :content=>content}end
get '/contact' do  @errors={}  contactend
post '/contact' do  @errors={}  @errors[:name] = 'No Anon allowed here.' if params[:name].nil? || params[:name].empty?  @errors[:email] = 'Sinatra needs an email to send your message from!' if params[:email].nil? || params[:email].empty?  @errors[:message] = 'No message?! Sounds like heavy breathing on the phone to me.' if params[:message].nil? || params[:message].empty?    if @errors.empty?    Pony.mail(:to=>'george@ghickman.co.uk', :from=>"#{params[:email]}", :subject=>"Contact Message", :body=>"#{params[:message]}")    redirect 'http://localhost:4000/index.html'  else    contact  endend
            view raw            _contact.rb            This Gist brought to you by GitHub.          

I've started off by setting the views path to Jekyll's source directory so Sinatra knows where to find the contact page and since you can specify a subdirectory the site layout can easily be found later.

The Page class is there to emulate the page class that Jekyll uses. Here I've created a title method as my site (what this was originally created for) checks to see what the title of the page is before rendering the sidebar. You can use this method to mock up any of the Jekyll specific objects you wanted, i.e. site.

The contact method is where the magic really happens. Here we're rending the contact page with Jekyll's layout and passing it our mocked up variables so that Sinatra doesn't freak out when the layout asks for them.

The script is finished off by the get and post methods that both call contact when they want to render a page (making my code nice and DRY). Voila! Jekyll has a contact page!

The last thing to note in the script is the redirect on a successful submission. Since your Sinatra is running on a different port to Jekyll (likely the default 4567) you'll want to redirect back to your Jekyll setup during development. In production this can be changed to redirect '/index.html'.

Setting it up in a Production Environment

Setting this all up in production requires a bit more than Jekyll since you're running a ruby application in the background now. Luckily this is where Phusion Passenger comes in. You'll need a rack file in your Jekyll root that looks like this:

require 'source/_contact'
set :run, falseset :environment, :production
FileUtils.mkdir_p 'log' unless File.exists?('log')log = File.new("log/sinatra.log", "a")$stdout.reopen(log)$stderr.reopen(log)
run Sinatra::Application
            view raw            config.ru            This Gist brought to you by GitHub.          

Don't forget to update the redirect path back from Sinatra to Jekyll, then you're good to go!

Mumbai

2010-09-18T00:00:00+01:00

Coming into Mumbai's domestic airport we got to see a fair number of shanty town looking dwellings, contrasting what I'd read about the metropolitan downtown area. We booked a prepay taxi and headed outside to the taxi rank where, to my surprise and child like joy, they had incredibly stereotypical Indian taxis that look like they are from the 50's. They are called Black and Yellows (for their paint job...), have bench seats, usually no wing mirrors and are dented to hell. We took a ride in one of the mini van type ones to our hotel in Downtown. Alas, this was an hour and a half away... Sweaty!

MEAT and Other Delectables

As previously mentioned Gujerat is predominately a vegetarian province and we'd miss meat, along with cheese and eggs at the school. Thus, as in Rajkot, we promptly pigged out on the heaviest meat/cheese/egg dishes we could find. Mumbai is very much like Bangkok in some ways, there are lots of market stalls down the main Bazaar road in Colaba (the district we were staying in), street vendors are everywhere and my favourite - juice bars. Freshly squeezed juice practically on tap. Nom nom nom. On our last day we picked up a selection of spices to take home from a local supermarket. Damn it's cheap to buy spices in India. We bought about 4 or 5 kilos split between a variety of spices and the bill was barely £5. Win! Garam Masala seems to be the thing to get, it's a mix of a selection of spices that you can just chuck in with some meat and you've got a full curry.

Hotel Bentleys

I couldn't talk about our time in Mumbai without mentioning our awesome hotel. Hidden away down a side street in central Colaba, Hotel Bentleys is a few minutes away from the Gateway to India and completely secluded from the hubbub of the main road. It even had an old school lift. So. Much. Fun. We were the first people to use our room since it had been refurbished which meant it was possibly the cleanest room we'd stayed in all trip. The only downside was the pretty solid 'mattresses' we had to sleep on, but there was free breakfast so who can complain?!

Bon Voyage

Aaaaaaaaand home time! An interesting journey to the airport... Our 'booked' taxi involved the hotel front-gate-man getting us a taxi on the day. The driver turned out to be pretty drunk with reaction speeds to match which wasn't so bad until he clipped a guy knocking his phone out of his hand. Rather than stop he just gave the guy an angry look, shouted and carried on driving. He was a very angry man. So very very angry. Once again the airport was rammed with armed guards, even a mounted sentry post on the way in! Then it was home time on our nine and a half hour flight home.

BYE INDIA!!!

Back to School

2010-09-17T00:00:00+01:00

One of our main aims in India was to help out at a school we had vague ties to. The school had been variously described to us as an orphanage (and is indeed for children who have not had the best start in life, but also for others) and a hostel. We had assumed, before arrival, that it was a small school with 200 girls attending where we could help with teaching English. It actually turns out the 'school' is more of a community. Based on 5000 acres of land it is a charity run organisation with a large school, hostel for the children, various (large scale) cooking facilities, a water treatment plant, enough farm land to be self sufficient for 9 months of the year, a college/university, an old peoples home, a blind school, a general farm and more. Our initial figure of 200 girls is about right, however no-one thought to mention the other 2000 male students living here! Almost all staff live on-site and their are a good number of administrators and other personnel for things like building, farming, cooking, etc.

This post is going to differ a bit from my previous India ones in that I'm going to pick on various parts of our stay and talk about them. Two weeks here has left me with far too much to talk about!

Meal Times

Being guests we ate in a different kitchen to either the elderly/blind or the main school body and mostly ate by ourselves - sometimes being joined. We were served by the same kitchen as the elderly/blind school just in a different room and I don't think either of us have been so voraciously offered food in our lives. At first I felt a bit beseeched having to defend my platter from being filled every time I looked away but slowly learnt the game of covering my food while pushing the cook and his serving away. It's a hard life being a guest in Gujerat! Water had to be filtered to make it drinkable, for us at least, and most people drink far more butter milk than water anyway. I gave butter milk a go, it was not a great experience. Poppy was a little sick in her mouth when she tried it. She did not try it again. I gave curd a go which was like plain/greek yoghurt when mixed up. I had some sort of brown powder added to it (possibly cinnamon) that added a sweet edge. Unfortunately this was probably the source of my day of bad bowels (rusty water much?!). After expressing a like for potato, a few days in, the cook jumped at the opportunity and every meal time onward had some sort of potato (aloo in Hindi) in one of the dishes - yes, even breakfast.

It would be prudent to note at this point that every meal consists of 3+ dishes essentially, called a Gujerati Thali - it's like a set meal. Your main plate is a platter with edges and on it you have 2 or 3 bowls and a cup. All metal of course. You are served with a variety of dishes. We got a lot of lentil based dishes that I loved and generally there was theme to the meal. Dal (a kind of broth made from lentils) was served with almost every meal and I loved it at first but went off it after having it for two meals a day for a week. We discovered that chapatis and rotis are basically the same thing - having an english speaking chaperone was great as I asked about every dish we got served and learnt the rest of the dish names that I hadn't worked out. My favourite was by far the mash potato with chopped potato bits in it (me likey aloo) and butter. Fresh butter is the best thing. Ever. They make it daily (from what I could tell) and use unthinkable amounts of it in everything. I was in heaven. It's not salted and is almost like whipped cream. Pancakes would have gone very, very well with the butter. Nom nom nom.

[Farm] Animals

Like everywhere else animals roamed freely at the school. The main difference is that the cows are farmed. Gujerat is both a dry and vegetarian state in honour of it being Ghandi's birth place. There are of course exceptions to this, mostly meat being available (chicken and mutton) primarily for tourists. So we were a little surprised to see cows being farmed when we arrived. It was later explained to use that while cows are a sacred animal they can be farmed for their produce. The cows at the school are the biggest healthiest cows I've ever seen (all adult cows were a match, if not bigger, than prime bulls I've seen in the UK...). On average they produce 70 litres of milk per day over two sittings (explains why dairy products feature so heavily in the diet there!) and the herd is roughly 200 strong. Any cow that can grow horns has got an epic set of people stabbers on their head and as with all the cows we'd seen - they go where they please. We had one cow in the field outside our room for a few days who would randomly wander through a flower bed to get there before stuffing his face from dawn til dusk.

A lot of equipment used to farm is mechanical but it blends seamlessly in with the Ox drawn carts and ploughs. Having never seen oxen before we first thought they were just humungous cows with even better horns (these look like your stereotypical devil/fully grown hellboy horns). Got a little schooling on that....get it schooling, at a school... #terriblejoke

Dogs randomly potter around the place but are some of the skinniest we'd seen. It turned out that residents were told not to feed the dogs in an attempt to keep them away but the elderly, doing as they pleased (some things never change), seemed to break this almost constantly - giving them a permanent pooch presence.

Teaching Techniques

I am no teacher. This is clearly evident if you have ever met me. I am happy to wander off on random tangents of thought and end up with cotton mouth 5 minutes into any public speaking (there is never any water around). So I do not, by any means, presume to pass judgement on the teaching techniques of the school. However I found them quite interesting. Theory and textbooks rule. Everywhere I went textbooks are being used by staff and students alike and whenever I tried to lend my expertise (only at the college/university - software engineering doesn't have much application for 3-16 year olds) the question that constantly popped up is: What book should I consult about this? This is not my way of doing things. I am a very practical person - something my academic career can attest to and books are by no means my preferred method of learning. In my opinion they are for stories, which has probably never helped me with textbooks! So I found this culture of text books hard to understand. Of course I don't think it's wrong, just very alien! I certainly know of a number of people who would flourish in the environment.

Leaving

We were fortunate to be driven up to Rajkot when it came time to leave. Which was no doubt for the best, it probably would have taken us the whole day if we'd tried to do it ourselves. We said goodbye to everyone, which was no quick thing and grabbed pictures of everything we could think of (still managed to forget the room of course) and headed off for the night in Rajkot before jetting over to Mumbai.

To anyone reading this from the Institute - thank you so very much for a wonderful two weeks. I hope the lab is still working!

Diu - Home of Sand, Sea and Forts

2010-09-16T00:00:00+01:00

Diu is a little island off the south coast of Gujarat that was under Portuguese control until some point (I'm vague on any actual sort of date or time frame....I'm no history buff). The general architecture is a weird mix of Indian construction with the loud colours that you see in a lot of Portuguese culture, made even more obvious when you see the grey pure concrete buildings being constructed next to the ADHD coloured buildings. Connected to the mainland by a bridge the whole island is pretty tiny at 14km in length. Apparently it's a big holiday destination in the high season but this is MONSOON so it randomly rained like rain is going out of fashion and everything was a fraction of the price (our hotel was 500 rupees a night instead of 1500).

Highway to HELL

Allow me, if you will, to take you on a literary journey. Close your eyes and imagine the worst smell you can think of, throw in a heavy over tone of body odour, if this makes you gag/throw up in your mouth a little then you're on the right track. Now infuse that very smell into a cold, ever so slightly, damp mattress and put a blanket over the top. The blanket is dry but has the same smell and is covered in bits of general muck that stick to you the minute you touch it along with it's fluff. Now take a 10 hour night bus on that very mattress and blanket and try very hard not to think or talk about your situation. It will make you sick.

This was our bus journey from Ahmedabad to Diu. We covered ourselves in our sleeping bag liners. It was terrible. I'm still trying to repress the memory.

La Dolce Vita Music Garden

The first night in Diu we went for something budget-as-hell that the book said was cheap but okish. We were craving a shower/bleach after the bus so didn't look too closely when showed the room and ended up in Jay Shankar. While perusing the balcony [read: bored] I noticed a little restaurant over the road call La Dolce Vita Music Garden. It looked nice enough as being the lazy person I am I made sure we ate their first rather than trying to find somewhere else. Best decision of the trip.

La Dolce Vita is a collection of tables under a thatched reed roof attached to the owners' house. They have a fairly large selection of dishes on the menu that they cook in their own kitchen and were always happy to see us - probably because we were their only customers 99% of the time. They even helped us to learn a lot of the dish names, which became invaluable in other restaurants as up until that point I'd mostly picked dishes based on either randomly pointing at one or whichever had the coolest name. This had mixed results... I found a dish there, that was much like the first one I had in Delhi, called Aloo Gobi which means Potato Cauliflower. It didn't have much sauce but tasted great and mixed with one of their mammoth portions of rice formed more of a feast than a single meal. Nom nom nom!

My favourite part of La Dolce Vita is how the dishes always vary from day to day. This can be seen more explicitly in the lassis (like a milkshake but with yoghurt) and milkshakes that were practically a different drink every time we had one and just adds to the homely vibe of the place.

Fort Boyard and Fort Bill

In various places around Diu I've seen pictures of the sea fort in the main port bay. I promptly named it Fort Bouyard in my head and would strongly recommend the Diu authorities follow suit. You can't visit the actual fort so the only way to see it is to take a fishing boat around the outside and since the weather took a turn for the worse it got sacked off. Of course there are totally tigers in there for people who don't make it out of the cage fast enough, this I am sure of.

We did go and see the main fort regardless of the rain. It's one bad ass fort! Built into the bedrock of the peninsula, surrounded on three sides by sea and a double moat on the final side it seems pretty impenetrable. Somehow the Portuguese were ousted by the Indian army though. Fail. The two moats are backed by walls with with three bastions on the outer wall and four on the inner, each bastion packed to bursting with cannons...how the Portuguese lost is beyond me. L for love! The fort is still in use as the local prison and for some government offices since these existed back when it was built so it does make sense, but it does mean some parts are off limits which is a shame.

The first spectacle as you enter the fort proper is St George's Bastion which, apart from being named the best, overlooks the landing pier ready to bombard any marauding pirates. We saw another random bastion after this, St Luzia I think, which was more of the same, an armoury which seemed to have some rather more modern (post WWI) artillery pieces and a chapel before we got historied out (ADHD much?) and left.

We saw some restorative work had been done to walls here and there and some workmen were fixing up a wall but it's nothing like you'd see in England at say an English Heritage or National Trust site. The new brick work seems far less long lasting than the 500 year old stuff they are removing, which is a shame. There's a general feel of disrepair about the place which isn't helped by the large patches (old buildings that have since fallen down possibly) of overgrowth. Had English Heritage got their hands on it we could have at least had an overpriced coffee at the end.

Overall the fort can be summed up as RAD^Sick especially if you're into history but I'd forgotten most of the broken english translation of it's history by the first bastion. The slippery steep slope (stairs anyone?) up to each bastion was enough to pique my Fear of Heights into rearing it's ugly head. Flip flops were a bad choice.

I couldn't help but think that had the place been done up it would have been amazing for paintballing or airsofting on an epic scale. Pirates attack the fort anyone?!

Reservoir Dogs

Towards the end of our Diu tour we were, once again, visiting our favourite restaurant when the second group of people we'd ever seen there appeared. Six shifty looking indian fellas came in and perused the menu before sitting down. I didn't pay them much attention after the initial shock at seeing someone other than the owners in my restaurant. After we'd finished yet another scrumptious gut busting meal I popped out my iPhone to look up a route from the restaurant to the local fort when one of the shifty gentlemen appeared at my shoulder giving me a pretty good fright. This put me on edge a bit until I later found out they had all been watching us the whole time and seemed especially interested in our wallets when it came time to pay up. Worse still the wife had stayed inside and when one of them approached us to ask for, what we thought, was a photo of the group the husband had told Poppy to sit down almost the moment she stood up. I found most of this out as we were walking away and immediately felt uneasy, promptly topped off by one of the guys coming out of the restaurant to watch us walk down the road. He was still there when we went round the corner a few hundred metres away. With paranoia rising we walked back to the hotel at almost running pace and promptly hid in the room for the next few hours, leaving the fort for another day.

Looking back on the experience the group were probably not muggers/the mafia and simply really leery guys which would explain the wife staying inside and the husband barely talking to us after they arrived (he was too busy keeping an eye on them). The worst part of the whole thing is we'd both become very comfortable in Diu, with Poppy venturing out unaccompanied occasionally and me using various tech in public but we were a little on edge afterwards and it took us a couple of days to get comfortable again.

Cheese It!

A very early start (5.30am) saw us on the 7am (sorry 7.30am...) bus to Veraval which needed to be jump started and whose horn got stuck on after mass overuse so the driver had to disconnect it. Sitting down the front of the bus meant we got to ignore almost all the usual stares (the door was in the middle) - a nice change. Once in Veraval we went looking for the bus we'd been assured we could get to Visavadar, which turned out to actually be a bus to Junagh and then get a connection from there, an extra 96km. It's only 120km from Diu to Visavadar.

A hilariously roundabout negotiation for a rickshaw got us to the train station for 10 rupees saving us the trip in the pouring rain. We just about managed to purchase a ticket to Visavadar on a train that was leaving 3 hours later and headed for our platform. The bridge was really high and floored with tiles that were slick with rain, so so utterly dim cool with a giant bag and flip flops on. Even after being up close to 9 or 10 hours I still loved being on the train. Over half the stops had no visible buildings but people would be waiting around, sometimes to sell you things when you got off for a break or because it was a 'station'. At one stop there were monkeys outside, so being the ultimate tourist I leap from the train, camera in hand, and tried to get as close as possible to get a good picture. These monkeys were enjoying the food they were being given but apparently the white man getting closer than 3 feet is a faux pas....I got hissed and had teeth bared at me. It was rad. I could have totally taken that monkey....