The solution is simple, add some information about the instance to the HTTP response headers!

It doesn’t really matter which load balancing or proxy solution you use, Amazon ELB, HAProxy, Varnish, Pound, Nginx, this would just work with most of them without any modification.

Here is a simple example of just such a middleware:
https://gist.github.com/2393220#file_hostname_middleware.py

So as the first step in making it more transparent to use the original Amazon AWS API reference documentation in a copy-and-paste fashion, I had to write the same reinvent-the-wheel piece of boiler plate that everyone wrote a hundred times before (google for it, its there). But mine is shinier and prettier, or so I would like to think.

I present to you, in all its glory — The Python Amazon AWS API queries class!

There are several “solutions” that provide stubs that can be used in unit testing Google AppEngine applications, including something called a “testbed” which is part of the API itself. But the problem with these is that they provide functional bits of API implemented on your local environment just like it would work on a deployed AppEngine application.

It sounds quite good to have a local personal instance of something similar to the datastore you get in deployed applications, but unfortunately for the urlfetch service it is not exactly what I was looking for in tests.

The thing I need is an object that will not urlfetch anything, will not access the network at all. The requirement in this case is an object that I can tinker with its state before and after my own methods have used the urlfetch facility. After a lot of digging in the current implementation of the stub, I ended up writing a very simple mock for this myself. It is far from perfect, but its a start.

> irb
irb(main):001:0> a = [1,2,3]
=> [1, 2, 3]
irb(main):002:0> a[10] = 20
=> 20
irb(main):003:0> a
=> [1, 2, 3, nil, nil, nil, nil, nil, nil, nil, 20]

Perl:

> perl
@a = (1,2,3);
$a[10] = 20;
use Data::Dumper;
print Dumper(@a);
$VAR1 = 1;
$VAR2 = 2;
$VAR3 = 3;
$VAR4 = undef;
$VAR5 = undef;
$VAR6 = undef;
$VAR7 = undef;
$VAR8 = undef;
$VAR9 = undef;
$VAR10 = undef;
$VAR11 = 20;

Python:

> python
>>> a = [1,2,3]
>>> a[10] = 20
Traceback (most recent call last):
  File "", line 1, in ?
IndexError: list assignment index out of range
>>> a
[1, 2, 3]

I really love the DSL that CruiseControl is using for it’s configuration, it’s extremely powerful at describing how to build projects. Especially powerful are the variables, that unlike in Ant are not immutable, and the way plugins can be pre-configured with your own defaults, as well as renamed to other names. It’s really easy to configure it in such a way that adding a new version for a project is just 1-3 lines of XML, for example

<xxx-project name="XXX v6.66">
  <property name="version" value="6.66"/>
</xxx-project>

Just in those 3 lines, the pre-configuration already includes all the information about the project. Where it is at, who to send e-mail to, where is the version control, EVERYTHING! If the only variable that changes over time is the version number, then that is all you need to leave as a variable … everything else is just a template that can be re-used. And these templates are extremely easy to combine from smaller templates, it’s a template-in-the-template kind of configuration.

IMHO this would very much apply to configuration of monitoring software, like nagios for example. And the way the (CC) plugins are written in java – adding new plugins that check all kinds of esoteric things is really easy to do.

If it would also have the XML/XSLT configuration of how the web-interface looks like (the way CruiseControl does), and the super-easy installation (again like in CruiseControl). It would be a really really really great product, extremely powerful, easy to configure, and potentially great looking.

If only ThoughtWorks would write such a thing … I would be thrilled!

Actually nagios is already extremely similar to what I described, but for some strange reason I find the rigid configuration of nagios a large PITA. Maybe some-day when time stops and I will have unlimited time to code, I will do it myself.

I am reading the book “Refactoring” by Martin Fowler, and just reek with ideas about improving software, as well as solving problems I head-banged during my “software” development career.

On my last job, there was this huge messy heap of bash scripts that were “The Installation” of their main software product. It was a remarkable amount of bad smelling bash code, however it somehow managed to work. My work, at the time, was from-scratch-rewrites of this or that functionality and then somehow plugging it into the existing framework (damn, i call it a framework now).

It could have been just amazing if I could take the existing pile of dirt dung bash scripts – and refactor it into something that is readable, sort of.

Today, while I was riding the bus, reading “Refactoring” pg.110, it struck me. It’s actually can be extremely easy to test bash scripts! All it takes is a collection of all the familiar commands, like “cat”, “rm”, etc … and one sneaky PATH environment variable. These commands would be fakes, stubs – they all just print their name and parameters into a log file. In fact, there is just one actual script and the rest are links to that single one.

That log file can be compared before and after a refactoring. It takes less time to output names and parameters to a file than to execute the actual commands. So while running the actual script might take hours, with the stubs it should run in less time, much less. Finally, by comparing your pre and post refactoring log files, you get a really nice test-suite that can help you refactor. I might even call it replacing unreadable code with readable one without breaking much of anything.

In the particular case of the scripts that I’ve mentioned earlier, I guess that the most used refactoring to improve readability would be Inline Method. Whoever wrote the original scripts was very fond of wrappers for commands like “ln”. Even though commands like “ln” don’t really need a wrapper. The wrapper with at least two echo commands and a very long name is quite redundant and adds unnecessary complexity.

There are several problems with this simplistic approach though. One of the problems might be that the script is using full path in names (like /bin/ln), instead of relying on the PATH environment variable. Such things can be relatively easily taken care of until the testing solution is perfect. (I guess). One of the things to try, for example, is running bash in restricted mode, if I remember what that is correctly.

I’ll try that on some new messy scripts that I got on my new job!

YEY!

PS: Google docs rocks! (even publish-to-blog kinda works)

Is this GMail 2.0?
Fuck 2.0, I want the working GMail back!
And stop it from stealing focus to it’s own tab, that is SO annoying.

Thankfully, Sun realizes that not everyone is looking for a 3,456 port InfiniBand switch. So, it has prepped a 72-port InfiniBand switch that will sit on top of the Sun Blade 6048 rack.

I just can’t stop being amazed.

Imagine my surprise when today I find that box at the same place, with 50 new invitations to give away.

No, really – I don’t have any friends who don’t have a GMail account. I don’t want to use this “feature”, ever. And it does not contribute a thing to my GMail experience – get rid of it Google! Put some AdSense there or something.