Ben Hall's Blog

With the launch of branch I'm considering writing again

2013-01-14T18:59:00.000+00:00

Install Apache and PHP on OSX 10.8 Mountain Lion

2012-10-06T11:35:00.001+00:00

This is a great post on how to setup and configure Apache and PHP on OSX 10.8.

http://coolestguyplanettech.com/downtown/install-and-configure-apache-mysql-php-and-phpmyadmin-osx-108-mountain-lion

Looking for a developer to join the team in London

2012-07-18T14:34:00.003+00:00

I'm currently recruiting for a new C# developer to join the team in London. We're growing quickly and are looking for developers who want to build products that will people love.

More details on the role can be found here:
http://bit.ly/NhNCYc

If you're interested in applying then please contact us via the link above or directly via Blog {at} BenHall.me.uk

Storing Node.js application config data

2012-02-01T16:00:00.000+00:00

I've tried multiple approaches to storing application configuration with Node.js but most have been painful until I found this solution.

tl;dr: Executable configuration FTW via exports.

With Node.js you can export an object or function for use within another module. This is key to keeping your Node.js application readable and structured which as I've found can be an art-form in itself.

My current approach is as follows.

I have a config.js which looks like this.

var url = require('url')

var config = {}

config.google = {};
config.redis = {};

config.google.id = process.env.GOOGLE_ID || 'DEVELOPMENT.googleusercontent.com';
config.google.secret= process.env.GOOGLE_SECRET || 'DEVELOPMENT';
config.google.callback= process.env.GOOGLE_CALLBACK || 'http://127.0.0.1:3000/google/callback';

config.redis.url= url.parse(process.env.REDISTOGO_URL || 'http://127.0.0.1:6379');

module.exports = config;

Because it's a standard JavaScript module, when Node.js loads the file it will be executed with the config hash created. This has two important aspects.

Firstly, I can include other helper modules, such as being able to return a parsed url.

Secondly, I can take advantage of the current environment to determine if production or development values should be returned. This is great when combined with Heroku.

With the configuration defined, the logic to access the configuration looks like this:

var config = require('./config')
var id = config.google.id
var url = config.redis.url

No if statements, no redefining variables, no swapping files around. Clean, simple, effective.

If I wanted to have additional confidence then if NODE_ENV equaled production I could ensure all environment variables are not undefined.

This is working for me, but has anyone found any better solutions? Leave a comment or tweet me @Ben_Hall

Using Redis and RedisToGo to store Node.js sessions on Heroku

2012-01-31T16:00:00.000+00:00

Storing data in the session state is a little bit naughty as HTTP should remain stateless but there is a trade-off. In this case I wanted to test an assumption around a potential feature for Mayday. I felt it was more important to release than over architecture the entire solution with the possibility of dropping the feature the next day.

When it comes to Node.js, session state is stored in-memory meaning any restarts or new deployments will delete the data.

Enter Redis

Redis is an ultra-fast, open source, advanced key-value store. To make it even easier, RedisToGo offer a hosted version with a Heroku plugin. After adding the plugin, an account will be created with a database URL provided as a configuration variable.

$ heroku config
REDISTOGO_URL => redis://redistogo:PASSWORD@SERVER.redistogo.com:9712/
NODE_ENV => production

To use this as your session store you will need to configure the middleware by defining a RedisStore from the connect-redis npm.

The require statements should look like this:

var express = require('express');
var RedisStore = require('connect-redis')(express);

var url = require('url')

For development you will want to use your local server.
app.configure('development', function(){
app.use(express.session({ secret: "password",
store: new RedisStore({
host: "127.0.0.1",
port: "6379",
db: "name_of_my_local_db"
})
}));
});

For production you should use the RedisToGo URL provided.
app.configure('production', function(){
var redisUrl = url.parse(process.env.REDISTOGO_URL);
var redisAuth = redisUrl.auth.split(':');

app.use(express.session({ secret: "password",
store: new RedisStore({
host: redisUrl.hostname,
port: redisUrl.port,
db: redisAuth[0],
pass: redisAuth[1]
})
}));
});

Node.js and Connect-Redis will do the rest for you.

The key will be the session id for the user with the value being a JSON serialised object of req.session.

In place editing of a file with sed

2012-01-27T09:00:00.000+00:00

When writing a file modification script I find it annoying (read: boring) to write the logic of saving the file under a different name, moving to override original and then cleaning up. This is what I recently had to do and instead decided to look around at what other options were available, one of which was sed.

The following command will replace the word testdomain with proddomain.

$ echo 'www.testdomain.com' > test.txt
$ cat test.txt
www.testdomain.com
$ sed -i '' s/testdomain/proddomain/ test.txt
$ cat test.txt
www.proddomain.com

The key to how this works is the -i '' argument of the sed command. -i specifies the file extension, by passing an empty string it uses the same file.

Getting node-compress to work on Node 0.6

2012-01-26T14:43:00.002+00:00

I have a lot of respect for Node, but sometimes certain npm packages become out of sync with the latest version and break. Today, that npm package was node-compress.

While the package looked to build, when attempting to require the module it would error.

$ node
> var c = require('compress')
Error: Cannot find module 'compress'

After a quick look around on GitHub I found a fork with the fix https://github.com/elliotttf/node-compress/

In your package.json, simply reference the tarball, clean out the node_modules directory, install and everything should work again.

Package.json

, "compress":"https://github.com/elliotttf/node-compress/tarball/1edaa48bf33f7c836f1e275691e1d8645f0a71c3"

Video of How To Deploy a .NET App with Mono on Heroku

2012-01-26T12:41:00.002+00:00

Wow. Chris Kemp (Advance technical solutions team at Salesforce.com) has create a great short video based on my blog post about How To Deploy a .NET App with Mono on Heroku. A huge thank you!

Link: http://www.youtube.com/watch?v=LkdiTzgqqsk&hd;=1

Help us by answering a questionnaire on Google Analytics

2012-01-25T08:30:00.000+00:00

To help with development of Mayday we are asking people to fill out a questionnaire about how they use Google Analytics. If you use Google Analytics and have a moment to fill out the questionnaire then we would be most grateful.

Visit our Google Analytics Questionnaire

Thank you.

A JavaScript equivalent to Ruby's respond_to?

2012-01-24T08:00:00.000+00:00

While working on the new version of Mayday, I wanted to show a message if no data was returned from Google Analytics. To add to the complexity, I wanted to be able to override the default message on a per page basis.

I'm already using the ICanHazJS client-side tempting engine which has a method for each template block on a page. However, if the page doesn't have the block then the method won't exist and an error will be thrown.

What I needed was functionality similar to Ruby's respond_to. With this method I can ask the object if it will respond to the method call. For example:

Object.respond_to? 'test' #=> false
Object.respond_to? 'to_s' #=> true

Luckily this is just as easy to do in JavaScript using 'in'

> 'test' in Object
false
> 'toString' in Object
true

This allowed me to write the following:

var e = $('#data')
if('nodata' in ich)
e.append(ich.nodata());
else
e.append('No Data Found');

If a nodata ICanHaz template block appears on the page then it will be rendered, otherwise it will fall back to the default. Problem solved.

Update:

Problem almost solved. As pointed out on Twitter by @nmosafi and @theprogrammer, just using 'in' along is not enough. For example:

> Object.test = "test"
"test"
> 'test' in Object
true
> Object.test()
TypeError: Property 'test' of object function Object() { [native code] } is not a function

What you need to do is ensure that the property is also a function. Something I had assumed previously.

> 'test' in Object && typeof(Object.test) == "function"
false

> 'toString' in Object && typeof(Object.toString) == "function"
true

Remove X-Powered-By for Express and NodeJS

2012-01-23T10:30:00.000+00:00

When responding to a web request it's common for servers to tell the client various bits of information. The one they enjoy most is some promotion around the name and version "powering" the site. Sadly, hackers also love this as it gives them more information for an attack vector.

By default, ExpressJS with NodeJS will return a X-Powered-By header.

$ curl -I 0.0.0.0:3000/
HTTP/1.1 302 Moved Temporarily
X-Powered-By: Express

I wasn't overly impressed by this but it's easy to remove. In your application configuration, at the top, add a new middleware function which removes the header.

app.configure(function(){
app.use(function (req, res, next) {
res.removeHeader("X-Powered-By");
next();
});

app.set('views', __dirname + '/views');
app.set('view engine', 'jade');
app.use(express.bodyParser());
app.use(express.methodOverride());
app.use(express.cookieParser());
app.use(express.static(__dirname + '/static'));
});

Simple.

Finding out where that console.log output is coming from

2012-01-21T18:55:00.003+00:00

While trying to solve a problem, we have all sometimes done a little console.log outputting to help us gain additional understanding. While it's far from the most effect way of debugging, it's more annoying when those statements appear in your production logs. That was my scenario today.

Thankfully, with a simple grep command I was able to identified all of my "debugging" statements and the lines they occurred on.

$ grep -nr console * | grep -v "node_modules/*" | grep -v "static/"

endpoints/browser_stats.js:35: if(error) { console.log(error); }
endpoints/browser_stats.js:51: if(error) { console.log(error); }
endpoints/browser_stats.js:67: if(error) { console.log(error); }
endpoints/events.js:16: console.log(eventsUrl);
endpoints/events.js:17: console.log(hitsUrl);
endpoints/events.js:20: if(error) { console.log(error); }
endpoints/events.js:27: if(error) { console.log(error); }
passport.js:23: console.log("Auth");

In this case, I filtered out any matches from node_modules and static javascript files. Keep in mind that in some cases it's a node module causing the output.

A simple command to help keep your production logs clean and readable.

Javascript WTF: The Date object

2012-01-20T09:30:00.000+00:00

Almost every time I have to deal with the Date object in Javascript I always hit the same WTF.

The documentation is clear but the API has some strange aspects that personally I don't think reflect the real world and how we naturally handle dates.

For example:
getDay() returns the day of the week (0-6) leaving getDate() to return 1-31. Personally, I would have imagined getDate() to return DMY with getDay() returning 1-31.
getMonth() returns 0-11. Classic computer science with starting the count at 0 while the Gregorian calendar is 1-12. This is also inconsistent with getDate() which starts at 1.
getYear() returns the year (usually 2-3 digits). For example, 2012 is obviously 112. You need to use getFullYear() to return 2012. The logic about how the method came to 112 is found at https://developer.mozilla.org/en/JavaScript/Reference/Global_Objects/Date/getYear

JavaScript 1.0 - we love you and you have left an impact in many ways.

Setting a cookie in Javascript shouldn't be THAT difficult

2012-01-19T18:02:00.000+00:00

With Javascript becoming increasingly popular, I'm still shocked at how bad the online documentation and examples are. Let's take the search phase "setting a cookie in javascript", a fairly common problem which I always forget the correct syntax for.

The first four results are as follows:
http://www.w3schools.com/js/js_cookies.asp
http://www.quirksmode.org/js/cookies.html
http://techpatterns.com/downloads/javascript_cookies.php
http://www.javascripter.net/faq/settinga.htm

Sorry to the original authors, but each one of them IS AWFUL!

Let's take W3Schools as an example. It defines a variable called ARRCookies are could be for Pirate Cookies (via @Blowdart) or ARRay (via @AndrewVos) - both of which are great examples of creating Clean Code. To make matters even worse, this example has spread like a virus with Google returning 288,000 results and GitHub returning 398.

The best link which describes handling cookies, and generally any Javascript documentation, can be found on the Mozilla Developer Network (MDN).

https://developer.mozilla.org/en/DOM/document.cookie

While I'm not an SEO expert, this looks to be light on SEO (Search Engine Optimisation) related to phases people search for. As a result the link was found in the lonely spot on page 3 which is should be number 1.

Wouldn't it be great if MDN had SEO focused content to support the excellent reference material it already has available? The PromoteJS movement has made a start but I think it needs a kick-start.

Who's with me?

UPDATE (20/1/12):

In a moment of great timing, the Mozilla WebDev team started a AMA (Ask Me Anything) thread on Reddit. I pointed out the above comments and had some great responses from the team.

It turns out that MDN is a wiki so it's in the community (and my hands) to make a difference.

You can read the comments here:

http://www.reddit.com/r/IAmA/comments/oonrg/iama_member_of_the_mozilla_webdev_team_ama/c3iup1b

How to deploy Mono projects with Heroku

2012-01-11T16:48:00.000+00:00

Last week I mentioned how I had successfully deployed Nancy / Mono (C# frameworks) onto Heroku. This post covers how you can deploy your own Mono projects onto Heroku.

As before, this approach is completely unsupported by Heroku and I'm not 100% sure if it's production ready. I would love to hear your experiences.

0) Prerequisites
There are a few requirements before deploying a project.
    - The solution can compile using Mono on Linux (Ubuntu 10.04 - Lucid Lynx).
    - The launch application must be a self-hosted executable.
    - The self-hosted executable must stay alive and not exit. Example: https://github.com/BenHall/nancy-demo-hosting-self/blob/master/src/Program.cs#L20-23

1) Update heroku gem
Before you continue, make sure your running the latest heroku gem as you need the buildpack option which is only supported in later versions. Thanks Andy Pike for pointing this out.

$ gem update heroku

2) Listening to Heroku requests
This is the only part which I'm not too happy with. In order for Mono to accept Heroku requests, you need to bind the listener to the full hostname provided by Heroku - for example http://deep-moon-1452.herokuapp.com.

The application also needs to listen on a particular port randomly defined by Heroku. This is provided via an environment variable.

Example:
https://github.com/BenHall/nancy-demo-hosting-self/blob/master/src/Program.cs#L12

Ironically this is the part which blocked me and after multiple combinations I was left with hard-coding the name as my last resort - see the commit https://github.com/BenHall/nancy-demo-hosting-self/commit/4d9e07bed92559a53b1463c37e837045d59c3a9a

3) How projects are built during deployment
Projects are built on Heroku during the git push process. To find a solution to build, the command "ls $BUILD_DIR/*.sln | head -1" is used. This will return the name of the first solution file found.
Once the name has been determined, xbuild is called with the output streamed back. If the build fails, the deployment is stopped.

4) How the application is launched - Procfile
Once the project has been built, the self-hosted executable needs to be executed via a Procfile.

An example can be found here:
https://github.com/BenHall/nancy-demo-hosting-self/blob/master/Procfile

The web property specifies the command used by Heroku. The first part is the path to mono which is defined in the buildpack and shouldn't be change. The second part is the relative path to your compiled executable.

In the example I also defined a local environment command allowing me to replicate the hosted environment locally via "foreman start local".

5) Creating the Heroku environment - Cedar and buildpacks
When creating your Heroku application to host your project, it's important to specify the stack as cedar along with the buildpack for Mono. The buildpack defines the steps required to download Mono from my S3 account, configure the environment and build your solution.

$ heroku create --stack cedar --buildpack http://github.com/BenHall/heroku-buildpack-mono

Remember to change the hostname your exe listens on otherwise you'll received a Bad Request (Invalid Host) response.

If you receive the error "!Name must start with a letter and can only contain lowercase letters, numbers, and dashes" then make sure your using the latest Heroku gem (currently 2.18.1).

6) Push
A simple push should now be all you need.

$ git push heroku master

Done! Technical details about how the buildpack works in a future post.

Important links:
Buildpack: https://github.com/BenHall/heroku-buildpack-mono
Example: https://github.com/BenHall/nancy-demo-hosting-self

tl;dr: The main steps are
0) $ gem update heroku
1) $ git clone https://github.com/BenHall/nancy-demo-hosting-self
2) $ cd nancy-demo-hosting-self
3) $ heroku create --stack cedar --buildpack http://github.com/BenHall/heroku-buildpack-mono
4) Replace deep-moon-1452 in src/Program.cs to the application name created above
5) $ git commit -am "Changed application name"
6) $ git push heroku master
7) Tweet to say how amazing it is.

Experiment: Deploying C# / Mono on Heroku

2012-01-03T18:15:00.001+00:00

Over the past couple of days I've been trying to get Mono running on top of Heroku.

I'm pleased to report that I have successfully deployed Nancy (a C# web framework) on top of Heroku.

It's far from a production ready solution and is completely unsupported by Heroku but it's an interesting experiment. More technical details in future posts but here's some proof.

Server response headers:
$ curl -i http://deep-moon-1452.herokuapp.com/
HTTP/1.1 200 OK
Content-Type: text/html
Date: Tue, 03 Jan 2012 16:33:02 GMT
Nancy-Version: 0.9.0.0
Server: Mono-HTTPAPI/1.0

Screenshot:

This is the output when you push your git repository to Heroku. The source code is built on Heroku.

$ git push heroku master
Counting objects: 9, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (5/5), 514 bytes, done.
Total 5 (delta 3), reused 0 (delta 0)

-----> Heroku receiving push
-----> Fetching custom buildpack... done
-----> Mono app detected
-----> Fetching Mono binaries
tar: Ignoring unknown extended header keyword `SCHILY.dev'
tar: Ignoring unknown extended header keyword `SCHILY.ino'
tar: Ignoring unknown extended header keyword `SCHILY.nlink'
tar: Ignoring unknown extended header keyword `SCHILY.dev'
tar: Ignoring unknown extended header keyword `SCHILY.ino'
tar: Ignoring unknown extended header keyword `SCHILY.nlink'
tar: Ignoring unknown extended header keyword `SCHILY.dev'
tar: Ignoring unknown extended header keyword `SCHILY.ino'
tar: Ignoring unknown extended header keyword `SCHILY.nlink'
tar: Ignoring unknown extended header keyword `SCHILY.dev'
tar: Ignoring unknown extended header keyword `SCHILY.ino'
tar: Ignoring unknown extended header keyword `SCHILY.nlink'
tar: Ignoring unknown extended header keyword `SCHILY.dev'
tar: Ignoring unknown extended header keyword `SCHILY.ino'
tar: Ignoring unknown extended header keyword `SCHILY.nlink'
tar: Ignoring unknown extended header keyword `SCHILY.dev'
tar: Ignoring unknown extended header keyword `SCHILY.ino'
tar: Ignoring unknown extended header keyword `SCHILY.nlink'
tar: Ignoring unknown extended header keyword `SCHILY.dev'
tar: Ignoring unknown extended header keyword `SCHILY.ino'
tar: Ignoring unknown extended header keyword `SCHILY.nlink'
tar: Ignoring unknown extended header keyword `SCHILY.dev'
tar: Ignoring unknown extended header keyword `SCHILY.ino'
tar: Ignoring unknown extended header keyword `SCHILY.nlink'
tar: Ignoring unknown extended header keyword `SCHILY.dev'
tar: Ignoring unknown extended header keyword `SCHILY.ino'
tar: Ignoring unknown extended header keyword `SCHILY.nlink'
-----> Vendoring mono 2.10.8
-----> building via /tmp/mono-GssQ/bin/mono /tmp/mono-GssQ/lib/mono/4.0/xbuild.exe /tmp/build_19y6xv4nr43ic/Nancy.Demo.Hosting.Self.sln
XBuild Engine Version 2.10.8.0
Mono, Version 2.10.8.0
Copyright (C) Marek Sieradzki 2005-2008, Novell 2008-2011.

Build started 01/03/2012 16:28:24.
__________________________________________________
Project "/tmp/build_19y6xv4nr43ic/Nancy.Demo.Hosting.Self.sln" (default target(s)):
Target ValidateSolutionConfiguration:
Building solution configuration "Debug|Mixed Platforms".
Target Build:
Project "/tmp/build_19y6xv4nr43ic/src/Nancy.Demo.Hosting.Self.csproj" (default target(s)):
Target PrepareForBuild:
Configuration: Debug Platform: x86
Created directory "bin/"
Created directory "obj/x86/Debug/"
Target CopyFilesMarkedCopyLocal:
Copying file from '/tmp/build_19y6xv4nr43ic/lib/Nancy.dll' to '/tmp/build_19y6xv4nr43ic/src/bin/Nancy.dll'
Copying file from '/tmp/build_19y6xv4nr43ic/lib/Nancy.Hosting.Self.dll' to '/tmp/build_19y6xv4nr43ic/src/bin/Nancy.Hosting.Self.dll'
Target GenerateSatelliteAssemblies:
No input files were specified for target GenerateSatelliteAssemblies, skipping.
Target CoreCompile:
Tool /tmp/mono-GssQ/bin/dmcs execution started with arguments: /noconfig /debug:full /debug+ /optimize- /out:obj/x86/Debug/Nancy.Demo.Hosting.Self.exe Program.cs TestModule.cs /target:exe /define:"DEBUG;TRACE" /main:Nancy.Demo.Hosting.Self.Program /platform:x86 /reference:/tmp/mono-GssQ/lib/mono/4.0/System.dll /reference:../lib/Nancy.dll /reference:../lib/Nancy.Hosting.Self.dll /reference:/tmp/mono-GssQ/lib/mono/4.0/System.Xml.Linq.dll /reference:/tmp/mono-GssQ/lib/mono/4.0/System.Data.DataSetExtensions.dll /reference:/tmp/mono-GssQ/lib/mono/4.0/Microsoft.CSharp.dll /reference:/tmp/mono-GssQ/lib/mono/4.0/System.Data.dll /reference:/tmp/mono-GssQ/lib/mono/4.0/System.Xml.dll /reference:/tmp/mono-GssQ/lib/mono/4.0/System.Core.dll /reference:/tmp/mono-GssQ/lib/mono/4.0/mscorlib.dll /warn:4
Target _CopyDeployFilesToOutputDirectoryAlways:
Creating directory '/tmp/build_19y6xv4nr43ic/src/bin/Views'
Copying file from '/tmp/build_19y6xv4nr43ic/src/Views/staticview.html' to '/tmp/build_19y6xv4nr43ic/src/bin/Views/staticview.html'
Target _CopyAppConfigFile:
Copying file from '/tmp/build_19y6xv4nr43ic/src/app.config' to '/tmp/build_19y6xv4nr43ic/src/bin/Nancy.Demo.Hosting.Self.exe.config'
Target DeployOutputFiles:
Copying file from '/tmp/build_19y6xv4nr43ic/src/obj/x86/Debug/Nancy.Demo.Hosting.Self.exe.mdb' to '/tmp/build_19y6xv4nr43ic/src/bin/Nancy.Demo.Hosting.Self.exe.mdb'
Copying file from '/tmp/build_19y6xv4nr43ic/src/obj/x86/Debug/Nancy.Demo.Hosting.Self.exe' to '/tmp/build_19y6xv4nr43ic/src/bin/Nancy.Demo.Hosting.Self.exe'
Done building project "/tmp/build_19y6xv4nr43ic/src/Nancy.Demo.Hosting.Self.csproj".
Done building project "/tmp/build_19y6xv4nr43ic/Nancy.Demo.Hosting.Self.sln".

Build succeeded.
0 Warning(s)
0 Error(s)

Time Elapsed 00:00:01.0730530
-----> Discovering process types
Procfile declares types -> local, web
-----> Compiled slug size is 78.3MB
-----> Launching... done, v18
http://deep-moon-1452.herokuapp.com deployed to Heroku

To git@heroku.com:deep-moon-1452.git
4d9e07b..6ce44e4 master -> master

2011 - An Unexpected Journey

2012-01-01T15:31:00.003+00:00

When I started this post I wanted to reflect on the interesting year I've had. I wanted to talk about how launching a start-up had taught me so much about business, technology and myself. I wanted to go on and cover how it has also meant making sacrifices, such as missing Glastonbury even when I had a ticket and cutting back on spending, blogging, open source and conference speaking - the things I really enjoy. Finally, I wanted to end with how start-up life has sometimes meant working in isolation and at times being demotivated and stressed with how everything was going.

However, while it's important for me to look back I also want to focus on looking forward which can be summed up by this tweet:

#2012 Release more. Blog more. Drink more. Earn more.
— Ben Hall (@Ben_Hall) January 1, 2012

To get the ball rolling, interested in hiring me? Send me an email at blog@benhall.me.uk

Slides on Amazon Web Services (AWS) from DDD North

2011-10-10T12:11:00.000+00:00

This weekend I had the pleasure of presenting on Amazon Web Services at DDD North. Please find my slides below which discusses S3, EC2, some architecture and other services provided by AWS.

If you have any questions, please ping me an email or tweet.

Taking advantage of the Amazon Web Services (AWS) Family

View more presentations from BenHalluk

Join @maydayhq as a start-up co-founder

2011-08-31T11:14:00.001+00:00

Cross-posted with http://blog.maydayhq.com/2011/08/31/join-mayday-as-a-start-up-co-founder/

We currently have an exciting opportunity to join the founding team of Mayday. Ideally, the person would bring a level of visual design / UI / UX expertise in the role of a front-end developer and help us build an amazing product. As a co-founder, you’ll be a key member of the company at an early stage, involved in building the vision, product and the company.

The product is SaaS-based, enabling clients to improve their website by alerting them when and where users experience problems. Combining lightweight integration with ‘Big Data’ technologies, we monitor users engagement, interaction and client details allowing us to identity why they’re having problems.
Our concept is still evolving, and you’ll be in a position to make your mark. Mayday has the potential to have a huge impact on development, testing and production systems and impact how we handle modern software issues and their related testing concerns.

Mayday recently graduated from the Springboard Accelerator Programme in Cambridge and has been covered by sites including TechCrunch and TheNextWeb. The next stages for the company involve a funding round, building an amazing team to allow us to change the world and then achieving profit. The product is currently an early alpha, with a small number of users providing valuable feedback before opening it up to the private invite list. A demo is available at http://app.maydayhq.com/demo

Key characteristics for the role include:
- Passionate about startups. Building a company is amazing and extremely rewarding, but it takes a certain type of person.
- Passionate about building amazing products and changing the world.
- Capable of producing amazing visual design.
- Capable of turning an amazing visual design into reality via HTML, CSS, Javascript and any other cool tools you fancy.
- Understands the importance of UX, workflows and the key metrics required to monitor and improve.
- Self motivated, proactive while taking pride in their work and looking for ways to learn / improve.

Remuneration: Variable; let’s discuss. As a co-founder you’ll be a key part of the business with significant founder share options available.

Location: Old Street, London (open minded to other locations.)

Interested?
Email workwithus {at} Maydayhq.com and tell us about yourself. It would be great to hear about recent projects you’ve been excited about, examples of your work and links to github, Twitter, LinkedIn, blog etc.

Why not add something different to the mix by telling us what features you would love to see in Mayday and how you see it growing over the next few years? What other startups currently excite you and why?

If you don’t fit the role of front-end developer, but feel you could make a difference to the company and add value in a different role, please email and tell us – we’d love to hear from you!

Count number of times a file has been download via HTTP logs

2011-08-17T15:03:00.000+00:00

Today, I wanted to find out how many times a file had been downloaded from a web server. Without Google Analytics, it was time to pull out the big guns - grep.

The command I used was:

grep -o /e.js /var/log/lighttpd/access.log | wc -w

/e.js is the file I'm interested in, while access.log is the file.

The output of grep will be a long list of the string /e.js to indicate the matches. wc counts number of words. With the use of the pipe, you have a lovely quick command to find out how many times a file has been downloaded via your logs.

SXSW 2012 Proposal: Lean Testing

2011-08-15T19:31:00.001+00:00

For the past few years I've been kicking myself for not attending South-by-Southwest (SXSW). Every year it sounds like an amazing conference, so this year I decided to submit a talk - what's the worse that could happen?

The session is titled "How Lean Testing removes waste and software bugs" where I'll discuss how to take a lean approach to testing and how it can enable you to remove the waste causing delays to releasing and delays to customer feedback among other things. Over the next few months I'll be blogging related items here and on the Mayday blog.

As such, I have a request. SXSW receive a huge amount of sessions every year. To help, they ask for the community to vote. If you could take a moment to vote for my session proposal then I would be most grateful!! http://bit.ly/leantestingsxsw2012

Every vote helps! Thank you!

First play with the Cloud-Based Amazon Reader

2011-08-10T12:57:00.003+00:00

In the last few hours Amazon announced their "Cloud Reader", enabling you to read your Kindle collection from a browser. Sounds great, but what's the implementation like?
Webkit based means no Firefox
Firstly they're promoting it as a web-based reader, however with their use of WebSQL database they only support Chrome and Safari. Attempting to use the service using the FireFox 8 Nightly builds results in the message "Your web browser isn't supported yet. Download Chrome or Safari below.".
Initially I was surprised they didn't use an abstraction layer allowing them to support IndexDB (Firefox \ IE) and WebSQL (Webkit). From reading articles, it looks like their target market is tablet devices meaning Webkit based browsers, making supporting only WebSQL an obvious choice. This is an important point for developers - understand your user-base when developing products. Release a product that works for your target audience, only add additional time and complexity if there is demand.
When I jumped into the product, there were a few more interesting surprises. I was shown was an option to setup Offline Reading. From a users point of view this was very surprising - most people would assume offline reading would be enabled by default.
Instead, it expects me to install a Chrome Extension. Amazon knows they will always have HTML5 and WebSQL available so I'm guessing this allows them to get around DNS lookups when no wi-fi is found.

Sadly, after clicking Install I was shown a blank tab with no explanations and nothing listed under my browser extensions. Not a great experience. I do have an Amazon Reader WebApp icon, however this process feels confusing and unnecessary mainly because of the time within the journey they asked me but also the lack of indication from Chrome about if it's an App or Extension.
Explicit Offline Mode
Within the reader the offline mode becomes even more confusing. They have made a separation between Cloud and Downloaded which makes sense with a large library but if your currently reading a book is that classed as cloud based or downloaded? From what I could tell it was cloud based which could become annoying. Having the separation of Cloud vs Available Offline would have made more sense based on other terms and phasing used in the application.

However, the reader is impressive. It shows how HTML5 and Javascript based applications can create compelling experiences, reaching beyond the web browser.

Yet, with functionality such as reading preferences not being used across different browsers\devices the reader still has a long way to go.
No wifi but still works
As proof of the potential, what actually happens when you don't have wifi? The first attempt was a complete failure.

After closing and reopening browsers it worked as expected and I was shown my offline book which was pretty impressive.
This is the start of an interesting future for HTML5 readers with fellow Springboard team Publification creating BrowserBooks which works without having to download extensions and where offline reading is implicit - just like you would have expected.

My blog has a few javascript errors... visualise them at MaydayHQ

2011-07-12T14:11:00.001+00:00

I knew my blog had a few issues, but I had no idea how many!

I'm not the only one. Every day users are encountering bugs and errors with websites that causes them confusion, creates a terrible experience and encourages them to a competitor resulting in companies losing revenue! Companies try and have a friendly "Oops - something went wrong" approach, but it doesn't hide the problem that the user had a problem.

To make matters worse, companies are unaware of the problems real users are experiencing and instead wait for them to contact customer support - this worked in physical stores, it doesn't work for the web. With more powerful javascript\HTML5 applications, the buzz around web stores and the wide range of devices pushing the boundaries then problems only going to get worse.

This is why we (Meerkatalyst) created Mayday (MaydayHQ.com). A product designed to capture the problems users experience allowing companies to fix problems before anyone even complains. This means more happy users and as such more revenue. With big plans for the future we're taking small steps and iterating quick. Currently we're alerting companies to javascript and in-browser errors and potential reasons for the problem.

You can visualise the errors affecting visitors of my blog at http://app.MaydayHQ.com/demo and you can sign up at http://www.MaydayHQ.com/

We'd love to hear your feedback on this, you can reach us at @MaydayHQ or if your in the London \ Cambridge area then we'd love to meetup for a coffee\beer.

Installing Hadoop 0.21.0 on Windows – Spaces in username gotcha

2011-01-18T00:39:00.001+00:00

A quick side note about installing Hadoop on Windows. My Windows username is “Ben Hall” – this will be used by hadoop in two places – one being ${user.name} and the other being $USER in shell scripts.

As hadoop is cross platform, it doesn’t expect to see spaces in the path names resulting in random errors.

The first one I received was while creating the DFS directory.

11/01/16 17:54:04 WARN common.Util: Path file:///tmp/hadoop-Ben Hall/dfs/name should be specified as a URI in 
configuration files. Please update hdfs configuration.
11/01/16 17:54:04 ERROR common.Util: Error while processing URI: file:///tmp/hadoop-Ben Hall/dfs/name
java.io.IOException: The filename, directory name, or volume label syntax is incorrect
        at java.io.WinNTFileSystem.canonicalize0(Native Method)
        at java.io.Win32FileSystem.canonicalize(Win32FileSystem.java:396)
        at java.io.File.getCanonicalPath(File.java:559)
        at java.io.File.getCanonicalFile(File.java:583)
        at org.apache.hadoop.hdfs.server.common.Util.fileAsURI(Util.java:78)
        at org.apache.hadoop.hdfs.server.common.Util.stringAsURI(Util.java:65)
        at org.apache.hadoop.hdfs.server.common.Util.stringCollectionAsURIs(Util.java:91)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStorageDirs(FSNamesystem.java:378)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNamespaceDirs(FSNamesystem.java:349)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1223)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1348)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368)

Not the most helpful error message. After some pondering and looking at the hadoop code-base, I realised it was due to the space in my username. To solve the problem I created an override of the default tmp path of the directory without the space. Within core-site.xml, I added the following property node:

<property>
  <name>hadoop.tmp.dir</name>
  <value>/tmp/hadoop-BenHall</value>
</property>

This allowed me to proceed until I hit the following error when starting the nodes:

C:\hadoop\hadoop-0.21.0\/bin/hadoop-daemon.sh: line 111: [: /tmp/hadoop-Ben: binary operator expected
C:\hadoop\hadoop-0.21.0\/bin/hadoop-daemon.sh: line 67: [: Hall-namenode-BigBlue7.out: integer expression expected
starting namenode, logging to C:\hadoop\hadoop-0.21.0\logs/hadoop-Ben
C:\hadoop\hadoop-0.21.0\/bin/hadoop-daemon.sh: line 127: $pid: ambiguous redirect
localhost: /cygdrive/c/hadoop/hadoop-0.21.0/bin/hadoop-daemon.sh: line 111: [: /tmp/hadoop-Ben: binary operator expected

Again – not an ideal exception. After looking within the hadoop-daemon.sh script, I found a usage of $USER to build up the log output path. Again, this resulted in a space in the path that caused the exception. To fix it, instead of using the $USER variable I hard-coded the value to BENH.

If you would prefer not to hard-code the value then you could use ‘sed’ to remove spaces dynamically as described here - http://mydebian.blogdns.org/?p=132

After fixing those two errors, I was able to run hadoop without any errors.

Installing Hadoop 0.21.0 on Windows – Installing Hadoop Map\Reduce

2011-01-18T00:31:00.001+00:00

After configuring your SSH server, you should be ready to install Hadoop core.

1) Download

Firstly, you will need to download the package. If you visit the following site then it will help you find your nearest mirror.

http://www.apache.org/dyn/closer.cgi/hadoop/core/

You should then be able to navigate and download Hadoop 0.21.0

hadoop-0.21.0.tar.gz      17-Aug-2010 06:10   71M

2) Unpack

Once downloaded, simply unpack the tar into a directory of your choice. As Hadoop is cross platform, ensure you extract it to directory without spaces (not Program Files) as otherwise you may experience random errors.

The location I picked was C:\hadoop. To extract, run the following command from cygwin:

$ tar xvfz hadoop-0.21.0.tar.gz

3) Configuration

Once extracted, you need to customise three configuration files - core-site.xml, hdfs-site.xml and mapred-site.xml. I used the settings from the quick start guide below, the settings specify which port different subsystems should run on.

http://hadoop.apache.org/common/docs/current/single_node_setup.html#Configuration

We can now start to test our setup. Personally, with any new tool I attempt to output the version first to check the basic system is happy.

In this example, execute:

$ bin/hadoop version

In typical fashion, my machine wasn’t configured correctly and I received the following error:
Error: JAVA_HOME is not set.

This indicates that I had to set JAVA_HOME to my JDK installation.

$ export JAVA_HOME=/cygdrive/c/PROGRA~1/Java/jdk1.6.0_23/

When I tried to output the version again I received a nice java stacktrace – at least it meant something had been executed. To solve this there requires some windows only modifications.

Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/util/PlatformName
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.util.PlatformName
       at java.net.URLClassLoader$1.run(Unknown Source)
       at java.security.AccessController.doPrivileged(Native Method)
       at java.net.URLClassLoader.findClass(Unknown Source)
       at java.lang.ClassLoader.loadClass(Unknown Source)
       at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
       at java.lang.ClassLoader.loadClass(Unknown Source)
       at java.lang.ClassLoader.loadClassInternal(Unknown Source)

4) Modifications for Windows

Personally, this was the most frustrating section however key to solving the above exception. Thankfully I found the answer on this blog post.

http://juliensimon.blogspot.com/2011/01/installing-hadoop-on-windows-cygwin.html

The problem is that the java classpath can’t find the required Hadoop libs\jars due to the way cygwin handles windows file paths.

As described in the post, you need to modify hadoop-config.sh. At around line 181, there is an if statement with the comment "# cygwin path translation"

Within this block, add

CLASSPATH=`cygpath -wp "$CLASSPATH"`

Pitfall: It won’t have any affect if you don’t add it within the if block. This took me a good hour to work out before finding the post.

Attempting the command again, you should hopefully see version output similar to below.

$ hadoop version
Hadoop 0.21.0
Subversion https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21 -r 985326
Compiled by tomwhite on Tue Aug 17 01:02:28 EDT 2010
From source with checksum a1aeb15b4854808d152989ba76f90fac

5) Starting Hadoop

Now is the time to start running Hadoop.

Firstly, we need to setup a temporary directory on our harddrive for hadoop to use while running jobs. By default this will be /tmp/hadoop-${user.name} which maps to C:\tmp\hadoop-${your login name}. If you wish use another directory, modify the hadoop.tmp.dir property in your core-site.xml.

To setup the directory, execute the command.

$ bin/hadoop namenode -format

You can then start hadoop.

$ ./start-all.sh

This will output some information about which nodes have been started. From a developers viewpoint, the most important part will be two new accessible websites.

HDFS Information http://localhost:50070 – This exposes details about the directory we just created

JobTracker http://localhost:50030 – This exposes details about any running or finished jobs

That’s it. Hadoop is now ready to start executing jobs which I’ll discuss in my next post.