Geoff on..

Tailrec in Scala

2012-09-30T12:52:00.000+01:00

One of the more tricky aspects to functional programming is optimising your code. This is because you are typically working at a higher level of abstraction than procedural languages. The code you write isn't necessarily a natural fit to the way the processor works. For example, if you want to compare the factorial function in C# vs Scala:

C#:

public void Factorial(int n) 
{
    var result = 0;
    for (int i = n; i > 0; i--)
    {
        result *= i;
    }
    return result;
}

Scala:

def factorial(n: Int): Int = 
    if (n == 0) 1
    else n * factorial(n - 1)

The big difference here is that in a procedural language you'll tend to write a loop that side effects on a variable where as in a functional language you'll tend to use recursion or a higher order function to write side-effect free code. Processors are really good at the former.

What we see here is a compromise between performance and readability. The C# code will be quicker because the code will do all its work in the same stack frame. The Scala version will create n stack frames while calling itself recursively. However, the Scala version is easier to reason about because there are no side effects, i.e. you're not rewriting result in a loop. You may disagree for such a simple example, but once you have a couple of loops and several variables, procedural code becomes very difficult to follow.

Most functional languages address this issue by performing an optimisation called tail-call optimisation. However, some languages are better at it than others. In Scala you need to make sure that your recursive call is in the tail position (i.e. the recursive call is the last expression). In the factorial example above, it isn't the tail expression is actually n * factorial(n - 1). (You can read about this in more detail here and here.)

Reasoning about whether recursive functions will or won't be optimised is a problem for beginners because it isn't always obvious what qualifies for optimisation. There are other criteria to take into account such as the call must be a local function and the function must be final. A great feature of Scala is an annotation, @tailrec, that raises a compile error you if your function won't be optimised:

import scala.annotation.tailrec

@tailrec
def factorial(n: Int): Int = 
    if (n == 0) 1
    else n * factorial(n - 1)

This will generate a compile error in this case because the recursive function can't be tail-call optimised.

To allow the compile to optimise this, you will need to find a way to have the tail call be the recursive call, for example:

def factorial(n: Int): Int = {
    @tailrec
    def recFac(n: Int, acc: Int): Int = 
        if (n == 0) acc
        else recFac(n - 1, acc * n)
    recFac(n, 1)
}

I learnt about @tailrec through Martin Odersky's Scala course on Coursera. The course is fantastic, Odersky's lectures are very interesting and tie theory into practical applications very neatly. The practical assignments really take me back to my first year of university.

Odd MSBuild platform behaviour

2012-09-15T14:10:00.000+01:00

I came across some odd behaviour with msbuild this week. I was building a solution from the command line and setting the platform to Any CPU. This worked find and the whole solution built. I then built one of the projects in the solution on its own (to create a clickonce installer), again with the platform set to Any CPU, and this time it failed. The error I got was that I was trying to build for an unknown platform.

It turns out that the actual platform described in the .csproj is AnyCPU (i.e. without a space) and that when building a solution Any CPU gets translated to Any CPU. So to build a solution you need to use Any CPU and to build a project you need to use AnyCPU. How bizarre.

Alternative dependency injection in F#

2012-09-08T14:40:00.000+01:00

Dependency injection is a pattern where a dependency of a class is passed to an instance of the class at runtime and is not created by the class itself. This is very useful for decoupling code, especially when you are unit testing and want to mock a dependency.

In OO languages like C# and Java, this is mostly often done with constructor injection, i.e.:

interface IB
{
        void DoB();
}

class A
{
        public A(IB b)
        {
                this.b = b;
        }

        public DoA()
        {
                this.b.DoB();
        }

        private IB b;
}

The same thing can be done in F#, but there's an alternative. You could have your dependency be an abstract member and use an object expression to create a concrete instance that has the correct dependency.

type IB =
    abstract DoB : unit -> unit

type B() =
    interface IB with
        member this.DoB() = ()

[<AbstractClass>]
type A() = 
    abstract b : IB
    member this.doA() = this.b.DoB()

let implA = 
    { new A() with 
        member x.b = B() :> IB
    }

This isn't a better way of doing dependency injection. It's a more complicated way of achieving pretty much the same thing, but it does remind me a little of the Scala cake pattern.

MarkdownPreview 0.2.0

2012-09-02T18:54:00.001+01:00

I've uploaded a new version of MarkdownPreview. You can pick it up here. This includes:

A better HTML render. The main benefit is that scrolling doesn't go back to the top when the Markdown source is reloaded.
Syntax highlighting. This uses Google prettify. You can add syntax to a code block by putting {{cs}} (for csharp, see here) at the top of your code block.

MarkdownPreview

2012-09-01T21:11:00.000+01:00

MarkdownPreview is a small Windows utility to watch a text file written in Markdown for changes and to render it to HTML. You can download it or visit the project page.

I write my blog posts in Markdown using Vim. I wanted to be able to preview what I was writing as I was writing it. While there are some Markdown enabled editors out there, I wanted to stick to Vim, so I wrote a small utility to watch my markdown file for changes and re-render to HTML whenever I save it.

I think this is my first open-source effort (MIT license)! It's only about an hours worth of work, so there may well be problems. Let me know if you find any. The main problems now are:

The page scrolls back to the top when it reloads
The links are followed in MarkdownPreview, not externally
There is annoying sound-effect when the page reloads
No syntax highlighting

I'll take a look at these soon.

The object-oriected TDD journey

2012-08-27T18:11:00.000+01:00

When you start practicing TDD for real, one of the first problems you'll come across is dependency management. Chances are, that before you were TDDing, whenever you needed to use class A from class B, B would create it's own instance of A or A might be a singleton.

class B
{
    private A instanceA = new A();
}

You soon find that you want to mock A and the best way to get the mock of A into an instance of B (your SUT), is to pass it into the constructor. This is called Dependency Injection:

class B
{
    public B(A a)
    {
        instanceA = a;
    }

    private readonly A instanceA;
}

You also find that you don't want a dependency on A because creating an instance of the mock of A is going to call the constructor of A. So you solve that by having A implement an interface IA and pass IA to B instead.

class B
{
    public B(IA a)
    {
        instanceA = a;
    }

    private readonly IA instanceA;
}

Your next problem comes along when you actually want to create an instance of B. Say the constructor of A takes an instance of IC, you will need to do the following when you want a new B:

B instanceB = new B(new A(new C))

For a large application, this step will be a horror of nested news. So you will then learn about Inversion Of Control and this will help you manage your dependencies:

IocContainer container = new IocContainer();
container.Register<B>();
container.Register<IA>().ImplementedBy<A>();
container.Register<IC>().ImplementedBy<C>();
B instanceB = container.Resolve<B>();

Here the IOC framework is responsible for figuring out the dependencies of B and creating the required instances recursively.

Now you find that you're making a mess becase your IOC framework is letting you create a dependency from any class in your system to any other. You now pay attention to group your classes into modules with small, well-defined interfaces between them.

When working on a piece of code you will try and break it down into small coherent modules that look like stand-alone libraries. They will ideally have a very small set of public interfaces. The IOC registration will be done by the module itself and classes external to the module should only use the public interfaces. This prevents a spaghetti of inter-class dependencies.

Rich Hickey on Debugging

2012-08-26T14:19:00.001+01:00

I was reading Micheal Fogus' interview with Rich Hickey, the man behind Clojure. When asked why he is considered an excellent debugger, his answer was fantastic. It often surprises me that this isn't how many programmers approach debugging.

Fogus: I’ve spoken with a few of your former co-workers, and they described you as a trouble-shooting and debugging master. How do you debug?

Hickey: I guess I use the scientific method. Analyze the situation given the available information, possibly gathering more facts. Formulate a hypothesis about what is wrong that fits the known facts. Find the smallest possible thing that could test the hypothesis. Try that. Often this will involve constructing an isolated reproducing case, if possible. If and only if the hypothesis is confirmed by the small test, look for that problem in the bigger application. If not, get more or better facts and come up with a different idea. I try to avoid attempting to solve the problem in the larger context, running in the debugger, just changing things to see effects, etc.
Ideally, you know you have solved the problem before you touch the computer, because you have a hypothesis that uniquely fits the facts.

Scala first impressions

2012-08-04T14:31:00.000+01:00

In the last year or two I've been practicing TDD and the things that come naturally from it: dependency injection, inversion of control and data / behaviour separation. All these techniques complement each other and encourage you to write code that is actually quite functional. This lead me to start looking at functional languages to see if there was a better way to build software; OO paradigms were starting to look a little unnecessary. I started with Haskell, it being one of the most pure functional languages. I then tried F# and now I'm playing with Scala.

Scala is interesting because it is a full object-oriented language, in some ways going further than Java, and is also a functional language providing most of the things you'd expect from a functional language. For example:

Pattern matching
Higher order functions
Anonymous functions / lambdas
Type inference
Currying

Traits

One of the more interesting features of Scala is traits. These are similar to interfaces in Java and C# except that they allow implementations of the methods they describe. I find this a bit scary but also fascinating because a very common code smell I see is people deriving from base classes in order to re-use code between classes and not because of an inherent 'is-a' relationship. Traits let you re-use code between classes without a cumbersome inheritance relationship. I'm only starting to get the hang of them though, for example they can be stacked.

Pattern matching

Pattern matching in Scala is very similar to F# and Haskell. It's one of those features that you find yourself missing from other languages once you've learnt about it.

Type inference

The type inference in Scala is not as good as in F#, this is apparently because of Scala's OO features. That said it's still much better than Java or C#, although that's probably an unfair judgement).

Anonymous functions

This isn't really exclusively the domain of functional languages anymore, C# and even C++ support it now. I think the Java committee have really let themselves down by not including this in Java 7. They are extremely useful and pop up in all sorts of places. They also allow library writers to design great APIs.

JVM

I really like that Scala compiles to bytecode and can be run on lots of platforms. I love C# and the .NET framework, but I hate that it doesn't have first class support on non-windows platforms. The mono framework is an amazing piece of kit, but I'm always going to trust the JVM on Linux over Mono. That's perhaps unfair, but you really don't want to worry about bugs in your runtime environment and there's always a manager you're going to need to convince.

Tooling

One of the things that I found a little disappointing about the Haskell eco-system is that there is a distinct lack of decent tools. The only proper Haskell IDE is Leksah which is a step in the right direction, but is still in its infancy. Haskell is the perfect target for auto-refactoring tools because it has proper strong typing, but there isn't anything serious, no re-sharper or eclipse for Haskell. F# is better but still limited (you can't organise your code in folders in Visual Studio for example).

I've been really impressed with the tools available for Scala. There is decent support available in Intellij and Eclipse (through plugins). They're still a little rough, but are definitely usable.

I think Maven is fantastic and being able to use it to build Scala and pull in maven dependencies is wonderful. Also, SBT is an alternative to Maven for Scala which is easier to get started with (not being XML based helps), which I also quite like. SBT can pull in maven dependencies which means getting an SBT project started is very quick.

Conclusion

I'm clearly becoming a fan of Scala, but I can't help worry that the reason I'm getting on with it is because it's not actually very different from the languages I'm comfortable with (C#, C++, Python, Java etc). I think I still need to keep learning Haskell to try and get around some of my problems with it (like how to build a large system TDDing as I go along - I can't rely on IOC to help me out like I do with C# and Java). However, right now I would be happy to start a production project using Scala. Though I would love to do one in Haskell as an experiment, it would be a risky proposition.

Scala, Maven and Eclipse

2012-07-29T20:54:00.001+01:00

You might be having trouble getting Scala IDE, Maven (m2e) and Eclipse to play nicely. In particular the

More than one scala library found in the build path [...] At least one has an incompatible version. Please update the project build path so it contains only compatible scala libraries.

Try the following:

Generate a scala project using the maven archetype scala-archetype-simple
Edit the pom.xml with the following:

Change the scala.version to 2.9.2
Remove the unit testing dependencies (add the ones you need back later once you have the other stuff working - the default versions won't be compatible with 2.9.2)
Change the maven-surefire-plugin to 2.12

Remove the sample test classes in src/test/scala/sample
Import the maven project into eclipse: Package Explorer -> Import... -> Existing Maven Projects

This should now give you a mavenised Scala project loaded into Eclipse. If you have trouble with eclipse not updating with your pom changes, delete the project, clear out the eclipse stuff from the project dir (.project, .settings, .cache, etc) then re-import the maven project.

Also note that I'm using:

Eclipse Indigo
The nightly builds of Scala IDE
The normal m2e

Give your people time to innovate

2012-07-28T15:27:00.000+01:00

Developers who are too busy to experiment are wasted talent. Software developers are smart people who need to be creative to be good at their jobs. In all but the most lax environments, team members will have their work set out for them. Even in Agile environments, there might be flexibility in who does what work and developers will be involved in the work estimation process. Ultimately, it is management and customers who decide what work is done.

Google has made the concept of 20% time famous. However, 3M have been encouraging their employees to experiment for 15% of their time since 1948:

"[..] workers often use 15 percent time to pursue something they discovered through the usual course of work but didn't have time to follow up on." Kaomi Goetz - www.fastcodesign.com

The idea is very simple. Give your teams a little slack so that they can try out new things. For example, a programmer might want to try out a new library, learn a new language or build an app. There are some solid benefits:

They can work on an idea that they think will be great, but would never be able to convince management without a prototype. Products like Google Mail and the Post-It note came out of these kinds of schemes.
Fix something in the product won't ever be a high priority, but these kinds of things tend to add quality to the product over time.
On the smaller scale improvements to processes and smaller innovations that can be added to existing products.
It is great motivation for employees. All projects will have periods of grind where developers are just cranking out repetitive work. A bit of a break when they can let their hair down does wonders for morale.
It's a great form of training. No matter what people work on in this time, they'll be learning while doing it.

Ok, so assuming that you're now convinced, how would you implement a scheme like this? Mike Cannon-Brookes of Atlassian says:

"You see, while everyone knows about Google’s 20% time and we’ve heard about all the neat products born from it (Google News, GMail etc) – we’ve found it extremely difficult to get any hard facts about how it actually works in practice."

It will ultimately depend on your organisation. I think the key is to keep it as open as possible. Any constraints on the developers are going to limit any possibly innovation. However, a few constraints will help keep everyone's goals aligned:

"Self-organizing systems are able to create their own rules. All is needed for such a system to work is a set of simple constraints, sometimes called a boundary. It is important for managers to tune these constraints, and not to try and design all the rules. This means the job of a manager is to manage the system, and not the people in it." Management 3.0 - Jurgen Appelo

For example:

Your project should be something that would improve the company
Your project should have short, achievable goals
Your project must be visible to everyone

The first point is designed to steer people to work on something relevant to the company but without limiting them so much that they play it safe.

The typical scheme will be between 10% - 20% of the employee's time. Any less and they won't have time to produce anything of substance, any more and it will interfere with their jobs. However, this is still not much time, so it's important to make sure people aren't being too ambitious and setting themselves unachievable goals (e.g. write an operating system in 4 hours). This also helps with the other two goals.

Lastly, people should be talking about what they are working on. This is important for stakeholders to see that people aren't just wasting time. More importantly, however, it means that people are sharing ideas which is going to generate more ideas and collaboration. A wiki and small demos might be a nice way to kick start this.

So why not try it and see what happens. Friday afternoons are typically productivity sinks anyway. Why not use that slow time to stir things up a bit?

Teaching TDD

2012-07-20T21:52:00.000+01:00

I've been evangelising TDD at work and have been quite surprised by the how it has been taken up. In general, people are excited to learn about TDD and there was an excellent turnout to talks a colleague and I have given. We ran through the full test driven cycle:

Write a test
Watch it fail to compile
Fix the compile errors
Run the test and watch it fail
Write just enough code to make the test pass
Watch the test pass
Refactor
Go to step 1

We demonstrated each one of these steps with 3 or 4 unit tests for an example class. However, no matter how much I repeat that you shouldn't write code before a test and that you should only write enough code to make your test pass, people struggle to get their heads around that idea. I still often come across people who are trying to learn TDD, but are not following this pattern.

I was prompted to start running the mini TDD courses when I saw a piece of code a junior member of my team was working on. He was still not used to the idea of writing tests first, so I sat with him and helped him write a bit of code in the TDD style. While he coded in front of me, I was able to point out when he'd skipped too far ahead and had written code that he didn't have a test for yet and it was only after a few round of this that he finally got what TDD was all about.

I think that a practical one-on-one session like this is the only way to really show someone how to do TDD. Sit with them and have them start writing code and just gently nudge them when they've skipped a step. I've done this a couple of times now and it only takes a couple of tests before it sinks in.

Using S3 With dm-paperclip

2008-12-18T15:51:00.000+00:00

dm-paperclip is a port of Paperclip to Datamapper. It's a pretty straightforward port so most of the Paperclip documentation is valid for dm-paperclip. One of the most attractive features of Paperclip is it's ability to host your files on Amazon S3 instead of locally. Here's how you do it:

You will need 3 extra parameters on your has_attached_file declaration in your model:


has_attached_file :photo,
:storage => :s3,
:s3_credentials => Merb.root / "config" / "s3.yml",
:path => ':attachment/:id/:style.:extension'
validates_attachment_presence :photo
validates_attachment_size :photo, :in => 1..200000

The config/s3.yml file stores your S3 credentials and bucket name. This is very handy because you can have different configs and inheritance, just like database.yml:


development: &defaults
access_key_id: ...
secret_access_key: ...
bucket: project_images_development

production:
<<:       *defaults      bucket: project_images_production

That's it!

"Update All" Has Returned To The App Store

2008-11-30T21:08:00.000+00:00

I updated my iPhone to 2.2 today and was very pleasantly surprised to see that the ability to update all the applications at once has returned.

Upgrading A Jailbroken iPhone From 2.1 to 2.2

2008-11-30T17:03:00.000+00:00

To upgrade an iPhone that has been jailbroken on version 2.1 to version 2.2:

Open up iTunes and plug in your iPhone
Sync and backup your iPhone
Open up iTunes and click Restore to restore the factory default, this should also ask you to upgrade to 2.2
Use QuickPwn to jailbreak your fresh 2.2 install
Restore your backup

NOTE: You can also use AptBackup from Cydia to backup your jailbroken apps.

iPhone Tethering Done Right: PdaNet

2008-11-30T14:39:00.000+00:00

Tethering an iPhone to another device, usually a laptop, is not allowed. AT&T contracts ban the practice (though this is rumoured to be changing soon). However, when you're stuck without a decent WiFi connection, an iPhone tether is a tempting proposition. Sadly, since it is banned, Apple have not made it easy.

Most solutions revolved around something called a SOCKS proxy. A proxy is just an application that runs on the iPhone that the tethered device communicates with in order to retreive data from the web. Despite it's simplicity, there is a major drawback: every application that needs to use the internet must have SOCKS compatibility built in. Many applications do, but the implementations are usually hasily added and tend to be buggy (even Firefox has issues).

If you were lucky enough to acquire the NetShare iPhone application (a SOCKS proxy that was available on the App Store for a very short period before being removed by Apple) you do not need to jailbreak your iPhone to be able to tether a device. However, most people will need to do this. However, this is really easy these days and quite safe. A tool such as QuickPwn will do the job with very little fuss.

Once your iPhone is jailbroken, you have a number of proxy server choices. From the NetShare style 3proxy to the old school SSH server. However, there is a much better solution: PdaNet. This is not a SOCKS proxy, it acts more like a software router, passing all your network traffic over the iPhone's internet connection. This means that your applications don't need to have SOCKS built in.

There are a number of issues with NetShare. Sometimes the DNS setup doesn't work very well (especially using an Ad-Hoc network on a Mac). It will also heavily drain your battery. You will need to plug it in for it to last for any long period (i.e. over an hour). You also have to pay to use it with encrypted websites (i.e. HTTPS) - but there is a way around that. Just visit the link printed at the bottom of the PdaNet screen on the iPhone, pretend to buy the software, but stop when you hit the payment screen. You can then browse any site you want without paying!

However, it's definitely the best tethering solution at the moment. Though you might also like to try solutions like proxifier for making SOCKS proxies easier (and more reliable).

Facebook Group Email Limits

2008-11-30T14:22:00.000+00:00

Facebook do not allow group owners with more than 5000 members to message all their members at once. Facebook have made this decision to stop people from acquiring large Facebook groups in order to spam the members. A group with over a million members would be incredibly valuable as there is a very direct and clutter free communication channel with the members.

However, I think that Facebook have missed a trick here. They have been struggling to find a revenue model for a while now and allowing this kind of marketing might give them a stready income stream by charging companies who want access to their group members. For example, say a million people sign up to the 'I Love Sprite' group. Sprite would have no way of sending a message to these users, despite them all having expressly shown their interest in Sprite.

Facebook could charge Sprite to message their users. This fee could also help to pay for decent quality control and to enforce any rules they might have. Facebook is already proficient at rejecting low quality ads (though they do have a inventory problem), this shouldn't be difficult for them.

Facebook Loses Data

2008-11-29T20:31:00.000+00:00

A schoolboy mistake from the king of social networks today. They sent me this email:

Please reset your email notification settings

Unfortunately, the settings that control which email notifications get sent to you were lost. We're sorry for the inconvenience.

To reset your email notification settings, go to:

http://www.facebook.com/editaccount.php?notifications

Thanks,
The Facebook Team

Made me chuckle

Spotify Rocks

2008-11-23T11:40:00.000+00:00

Spotify is a new music service that lets you listen to unlimited amounts of music for free. They have an impressively complete collection and you have complete control over what you listen to. The only caveat: it's ad supported. However, the ads are sparse (about one 30 second add for every album you listen to).

I was initially skeptical about having ads. I hate radio stations that will have you listening to adverts all day long or TV stations that play a few minutes of ads every 20 minutes. But adverts in Spotify are rare enough that it doesn't bother me. If you don't like the adverts, you can pay a subscription at £9.99 a month or a day pass for £0.99 (for parties perhaps?). I think the £9.99 a month is a little steep, but luckily I'm happy with the ad-supported version.

They have most of the music I like to listen to. However, they often don't have the obscure first albums (produced by tiny labels), which is a real shame. I expect this will change if Spotify gets big. There are a couple of big names who are not available. The only one I have really missed is Metallica's new album.

Each artist has links to similar artists. This is nice for finding new music, but doesn't appear to be particularly clever. That said, exploring is effortless because songs are played instantly and the browsing interface is very well designed. Songs and playlists can be sent to friends using special URLs, but these have to be emailed or IMed, it would be much better to have a more tightly integrated social network. I'm sure this will follow later on.

At the moment, Spotify is invitation only. I have a few spare, leave me your email and I'll send you one if I have one left.

Doing EC2 Without Scalr

2008-10-10T22:00:00.000+01:00

It is important that your website can scale. You can spend all your energy promoting it and adding features, but that is all wasted if it cannot deal with all those users who are desperate to use your site. The architecture is at least half the battle. We are spoilt these days with affordable options to be able to deal with scale. Cloud computing has become a buzz word and big tech companies are scrambling to carve out their chunk of the market. So far we have EC2, Google App Engine, Joyent, VPS services (such as Slicehost) and a number of others.

EC2 was one of the first and most mature of the offerings. Though it does not stray far from the traditional concept of a machine (with fixed CPU and memory), what it does do is give you access to as many of these as you need at any one point and the (very) basic tools to manage them. They provide the basic infrastructure: machines, network connections and data storage. It is up to you to do the rest: load balancing, application servers, relational databases, backups, replication, redundancy, fail-over, etc.

Scalr promises to help with some of these issues, notably load balancing, and backups / replication / fail-over for MySQL. The idea is great but the project is not mature (despite the claim to v1.0) and the architecture is not very solid. To be fair, this is acknowledged to some degree in that v2 is to be a complete redesign. After a period of two months using Scalr, I decided to move on because of these issues:

Ad-hoc design: a strange combination of PHP, bash, MySQL and CRON. The interaction between these components was overly complicated, which means debugging was tricky and the system itself was prone to getting mixed up (for example: CRON calling PHP to do a backup but the MySQL state was stuck so backups were being skipped, silently, for weeks). V2 is to be Java.
Buggy: rebundling an instance would break the load balancer. It would get confused about the new instances and refuse to update. This was only solved by restarting the whole farm.
Scalr makes the decision to start new instances based on the load average. This is over simplistic really and can mean that your machines are under heavy load for a little too long which new instances are being brought up.
Scalr is built on Ubuntu 7.04 AMIs, upgrading the distro breaks it.

I do like what Scalr are trying to do. Just having pre-built load balancing servers, application servers and MySQL servers that are able to replicate and reconfigure themselves based on load is great. However, Scalr v1 is experimental really, I'm very keen to see how the next version evolves.

One alternative is to build the architecture yourself. It's actually not very hard (and quite quick). All you need is a decent scripting language that has good SSH and EC2 libraries and some experience with MySQL and linux admin. A few days work (4 in my case) will get you:

Run and stop instances with a single command, including application servers, master and slave MySQL instances.
MySQL slave replication (automatically configured against the current master)
NGINX load balancer, always up to date with the application servers.
Automatic backup and restore facilities.
Various other tricks that your setup might benefit from.

All that would be needed to catch up with Scalr's features (albeit not generically) is to automatically bring up the required instances as demand changes and automatically fail over MySQL instances. Not much more work and with the added benefit that you can tailor your load algorithm to your site (i.e. use response times instead of load averages). I would recommend getting your hands dirty!

Spotlight Goes Dotty

2008-09-05T07:53:00.000+01:00

The last 3 times I have had a new install of OS X, I've noticed that it gets sluggish after a few weeks. Loading up the monitor app shows a process called mdworker being heavy on both CPU and memory.

My first reaction was that it was some sort of virus or at best a rogue app, but googling mdworker revealed that it was part of OS X. More precisely it is the Spotlight indexing daemon (keeps track of all your files to make searching faster).

My first reaction was to look for a way to turn off spotlight. I don't really use it very much and Quicksilver covers the needs I do have. However, Spotlight is also needed to search your mail, which I can't do without.

The problem appears when Spotlight is trying to keep up with a large number of new files on the system. In my case copying some music onto a new machine. The most straightforward solution is to tell Spotlight to ignore those files. This is easily done in System Preferences -> Spotlight -> Privacy -> Add ( + ). I added my Music directory which is where iTunes stores it's library.

It is odd that this should be a problem. Indexing is something that should happen in the background. There is no reason for it to affect the user. It appears that Apple decided that it was more important for files to be searchable than for the operating system to be responsive.

A Good Reason To Use IFrames For Facebook Apps

2008-07-19T18:04:00.000+01:00

If you use canvas pages to display your Facebook apps, dealing with scale suddenly becomes quite a bit harder. A request must be served in less than 4 seconds otherwise it fails completely. This is fine when your traffic is constant, but can catch you off your guard if you get a sudden spike.

I'm currently building a Facebook app using Scalr to deal with scaling. This automatically creates new Amazon EC2 instances to deal with increases in load and kills them again when traffic dies down. The problem arises in deciding when to start a new instance.

Scalr keeps track of the server load average. This is a moving average over 15 minutes. It's a good idea to use a slow moving average like this so as to not start instances prematurely. It will only do so if the existing instances do actually need some help (especially considering Amazon charge by the hour). However, this introduces lag in starting a new server, say about 5 minutes.

If I weren't serving pages to users via Facebook this wouldn't be such a problem. There would be small window when the site would be a little sluggish. For a canvas page this means pages are considered to have timed out, not good!

Of course, using IFrame's is generally faster because it doesn't have the proxy overhead (and sometimes sluggish Facebook servers). There are some drawbacks in losing some cool FBML functionality (e.g. fb:can-see) and you will have to make a lot more calls to the API, but I now think IFrames are the way to go.

Is SQLAlchemy Ready For Production?

2008-05-28T20:02:00.000+01:00

I have built my most recent project using SQLAlchemy (SQLA) 0.4. It is a reasonably young library, so I thought others considering using SQLAlchemy might benefit from my experiences.

The most recent version is 0.4, so it obviously does not claim to be 'finished' (can software ever be?) However, there are plenty of benefits in using it already:

Ridding your beautiful Python code of any SQL. It's untidy and pain to work with (which means bugs).
ORM (Object Relational Model) to fit in with a nice OO architecture.
Free sharding to split your database horizontally.
DB independant code, so you can switch if you feel the need.
Hassle free transactions and DB connection management.

Sounds great doesn't it? In general this is exactly what you get. However, there are a few points which are very important to consider before committing yourself to using SQLA.

Forcing Indices

Anyone who has had to do something reasonably involved in MySQL has spent time optimising their indices. Sadly, MySQL is not very clever about how it chooses the index to use for a particular query. In some cases the only option is to force it with FORCE. This is not available in SQLA, so anytime you need to do this, you'll have to manually specify the SQL. Not only does this partly defeat the point of SQLA, but it also happens that the place you most need the SQL abstraction is often the same place you need to force an index. Consider a complicated search page. You are searching over a large data set using a number of filters. MySQL is probably going to get the index wrong and you're going to have to generate a complicated SQL query without SQLAlchemy's help.

Commit ORM Objects

SQLAlchemy can be a little silly about detecting a change in an ORM object. If you assign to a member variable which is part of the schema, it will be marked as dirty even if the value doesn't actually change. The solution is to check for a change before assigning, which does not make for neat code!

Sharding

It's a good idea to plan for scaling as early as possible. It can be very difficult to build the necessary bits in later when you need it (and will also under pressure to fix things fast). One common way to deal with scale is to split large tables over several machines, this is known as sharding (each independant chunk of table is known as a shard).

SQLAlchemy has some code to help you there. All you need to do is to define 3 functions which tell SQLA which shard a particular row is in. Really simple. Sadly, this part of the code is not very mature at all:

query.count() doesn't work (nor any scalar query). I had to write a function to query each shard in turn and sum the result. The real problem here is that it was not clear at all which bits of functionality will or won't work with sharding (expect long debugging sessions, digging into the SQLA code).
The ORM caches objects and identifies them by their primary key. However, a common MySQL trick when sharding is to have an auto_incremented INT as the primary key of each shard but use something like a UUID as the 'real' primary key recognised by the code (this speeds things up quite a bit). Of course, the auto_incremented key will not be unique across shards and this will confuse SQLAlchemy. I think the best solution here (suggested by someone on the SQLAlchemy group) is to have a 2 column primary key with an INT and another integer shard identifier, making it unique.

No Server Side Cursors

A minor point, but might be important for some. Server side cursors come in handy sometimes when you are dealing with large amounts of data.

Tips For Optimising

Periodically check the SQL queries being made with the echo option. There might be some surprises in there (though usually easily fixed). This kind of thing usually pops up because you will use the ORM instance and forget / not realise that it will result in a query (from a software design point of view, great. From an optimisation point of view, awful).
Use set_shard on a query whenever you are able. If you know which shard the column you want is, no need to go checking the other ones. A common example is when the shard identitifier is in one of the query parameters.
Design for scale right from the beginning and develop / test on a distributed architecture (i.e. have at least 2 shards). This doesn't need to be difficult, for example: just create two databases on your developement box to simulate two machines.

Conclusion

I still think that SQLAlchemy is worth using. Overall it will save time and effort as long as you are careful and not afraid to get your hands dirty when the going gets tough. I expect most of these problems will be addressed in (near) future releases.

C# BHO Tutorial

2008-05-27T23:12:00.000+01:00

I get a fair bit of traffic from people looking for help with C# and BHOs (my event handling post). There isn't much information about and the only beginners tutorial went missing a few months back. A new one has appeared, anyone wanting to get going should check it out.

Vim: The Word Processor

2008-05-14T21:31:00.000+01:00

I love Vim. It took me a while to get there, but I've been using it exclusively for coding for a couple of years now and it has become second nature. I first learnt to use it during a practical exercise for an operating systems course at uni. We had to write a Minix driver so all the work had to be done on the command line. A powerful text editor was a must. I think I used Vi (rather than Emacs) because it was available and I had been told it was great. It was a steep learning curve, but I got a hang of the basics after a few days.

It was a couple of years before I started to use it again. In the meantime I'd mostly been working with .NET and Java so I'd been using some pretty decent development environments and it didn't seem necessary to use anything else (especially considering the power of their debuggers). But I moved into the world of the web and started writing PHP and then Python, these did not really have especially good IDEs and so it was back to a text editor and my choice was Vim. It also coincided with worsening RSI, for which Vim is great.I tend to learn a couple of features in a burst every few months when something really bugs me. This is probably not ideal, but Vim is so powerful I will never learn it all. I've been keeping a to do list in a text file recently (rather than on paper - go planet!) but Vim's defaults are not great for editing prose:

Vim's word wrap is by character, not word.
k and j (up and down) work on a line basis. If you have a wrapped line, you cannot move inside it with j and k (like you would with a normal text editor).

Of course, Vim is hugely powerful and can be tweaked to be much more useable when writing prose. These .vimrc commands:

Wrap lines by cutting lines off at word boundaries.
The word wrap is virtual, no extra line break is inserted (so that it's easy to edit afterwards).
j and k are replaced with gj and gk which allow you to move up and down inside a wrapped line.
I've also added the spell checker (I've not played around with it properly yet, but it looks a little weak).
smartindent for bullet points.


 autocmd BufRead *\.txt setlocal formatoptions=l
 autocmd BufRead *\.txt setlocal lbr
 autocmd BufRead *\.txt map  j gj
 autocmd BufRead *\.txt  map  k gk
 autocmd BufRead *\.txt setlocal smartindent
 autocmd BufRead *\.txt setlocal spell spelllang=en_us

On a slightly different note: if you're using OS X, get this port of Vim. It's aim (and it does) is to integrate better into Macs. It's worth getting just because it has pretty Carbon tabs ;-)

Facebook Chat Firefox Plugin

2008-04-23T21:53:00.000+01:00

Facebook finally released their chat functionality to all their users today. It's fantastic! I quickly came to the conclusion that it did have one major flaw: you have to be on the Facebook page to see any incoming messages. So to solve this, I quickly knocked up a Firefox add-on to alert you if you have any new chat messages. You can download it here.

It uses the built-in Firefox alert system, so you'll need Firefox 3 on OS X (for Growl) and 2 for other OSs. At the moment you also have to leave the Facebook page open, if I have time later, I'll deal with this so that you can close the Facebook page.

PLUG: Donate 2 Date is my Facebook app. It helps raise money for charity by setting people up on dates. Check it out!