However, being hit by this problem, I managed to work around it by simply using stuff that’s already part of git itself. No need for going in panic mode for every GitHub hiccup. git is distributed, stupid.

I’ve seen a lot of solutions or proposals, but none of them were KISS compliant (or should I say: blog post title compliant), like using a different remote for pushing to a secondary service, or using hooks. Found out that git supports multiple url entries per remote, but the functionality isn’t exposed into the interface itself. You need to actually edit the config file.

Fortunately, git exposes a config edit shortcut: “git config -e” which opens the repository configuration file “.git/config” with the default editor. Found out that “git config -e” is easier to remember, but YMMV.

A real world example from one of my projects:

[remote "origin"]
        fetch = +refs/heads/*:refs/remotes/origin/*
        url = git@bitbucket.org:SaltwaterC/aws2js.git
        url = git@github.com:SaltwaterC/aws2js.git

This means that every time I issue a “git push [--tags] [remote branch]” everything is automatically synced in multiple remote repositories, removing the single point of failure.

The ordering of the url entries is important as only the first is use for pulling the changes. If a specific url fails to accept the changes, then the rest of the url entries are ignored. Sure, some things may go out of sync for a while, but “eventually consistent” is the term you’re looking for in this scenario. You may pull changes between team members, but that’s not always applicable, therefore it doesn’t hurt to have some failover option.

I found out that Bitbucket is a little bit more stable that GitHub. It defaults to that. Used to be the other way though.

However, it took me a while to write this second part.

Please have a look at the above video. Besides the general common sense guidelines about how to scale your stuff, and the Postgres typical stuff, there’s a general rule: cache, cache, and then cache some more.

However, too much caching in memcache (whatever implementation) may kill the application at some point. The application may not be database dependent, but it is cache dependent. Anything that affects the cache may have the effect of a sledgehammer on your database. Of couse, you can always scale vertically that DB instance, scale horizontally by adding read-only replicas, but the not-so-fun part is that it costs a lot just to have the provisioned resources in order to survive a cache failure.

The second option is to have a short lived failover cache on the application server. Something like five minutes, while the distributed cache from memcache may last for hours. Enough to keep the database from being hit from live traffic, while you don’t have to provision a really large database instance. Of course, it won’t work with stuff that needs some “real time” junk, but it works with data that doesn’t change with each request.

There are a lot of options for a failover cache since there’s no distributed setup to think about. It may be a memcached daemon, something like PHP’s APC API, or, the fastest option: the file based caching. Now you may think that I’m insane, but memcached still has the IPC penalty, especially for TCP communication, while if you’re a PHP user, APC doesn’t perform as expected.

I say file based caching, not disk based caching, as the kernel does a pretty good job at “eating your RAM” with the disk caching stuff. It takes more to implement it since the cache management logic must be implemented into the application itself, you don’t have stuff like LRU, expiration, etc. by default, but for failover reasons, it is good enough to worth the effort. In fact, it ran for a few days on the failover cache without any measurable impact.

The next part for not using the same basket for all of your eggs is: cache everywhere you can. For example, by using the nginx FastCGI cache, we could shave off 40% of our CPU load. Nothing experimental about this last part. It is production for the last 18 months. If you get it right, then it could be a really valuable addition to a web stack. However, a lot of testing is required before pushing the changes to production. We hit a lot of weird bugs for edge cases. The rule of thumb is: if you get the cache key right, then most of the issues are gone before going live.

In fact, by adding the cache control stuff from the application itself, we could push relatively shortly lived pages to the CDN edges, shaving off a lot of latency for repeated requests as there’s no round trip from the hosting data center to the CDN edge. Yes, it’s the latency, stupid. The dynamic acceleration that CDNs provide is nice. Leveraging the HTTP caching capabilities is nicer. Having the application in a data center closer to the client is desirable, but unless your target market is more distributed than having a bunch of machines into the same geo location, it doesn’t make any sense to deploy into a new data center which adds its fair share of complexity when scaling the data layer.

Now I know that there’s apt-cache rdepends, but it lists all the reverse dependencies of a specific package. I needed just the reverse dependencies of the installed packages. With a little bash-fu, here it goes:

#!/bin/bash
 
function package_rdepends
{
	for package in $(apt-cache rdepends $1 | grep -Ev "^$1$" | grep -v 'Reverse Depends:')
	do
		apt-cache policy $package | grep 'Installed: (none)' > /dev/null 2>&1
		if [ $? -eq 1 ]
		then
			echo $package
		fi
	done
}
 
package_rdepends $1 | sort -u

Saved as installed-rdepends. Made executable.

./installed-rdepends libpcre3
grep
libglib2.0-0
libglib2.0-dev
libpcre3-dev
libpcrecpp0

The above script may be slow for packages with many reverse dependencies due to the fact that each package has an individual lookup. Didn’t have the patience to measure the time it takes to do a lookup for libc6. Some benchmarks for the package lookup:

time apt-cache policy libpcre3 | grep 'Installed: (none)' > /dev/null 2>&1
 
real	0m0.006s
user	0m0.005s
sys	0m0.003s
 
time dpkg -L libpcre3 > /dev/null 2>&1
 
real	0m0.017s
user	0m0.012s
sys	0m0.005s
 
time dpkg -l libpcre3 > /dev/null 2>&1
 
real	0m0.667s
user	0m0.600s
sys	0m0.067s
 
time dpkg -s libpcre3 > /dev/null 2>&1
 
real	0m0.587s
user	0m0.533s
sys	0m0.054s
 
time cat /var/lib/dpkg/available | grep -E "Package: libpcre3$" > /dev/null 2>&1
 
real	0m0.034s
user	0m0.015s
sys	0m0.048s

However, I didn’t try these results on a bare metal installation.

// curl -k https://localhost:8000/
var https = require('https');
var fs = require('fs');
 
var options = {
  key: fs.readFileSync('test/fixtures/keys/agent2-key.pem'),
  cert: fs.readFileSync('test/fixtures/keys/agent2-cert.pem')
};
 
https.createServer(options, function (req, res) {
  res.writeHead(200);
  res.end("hello world\n");
}).listen(8000);

All fine and dandy as the sync operation doesn’t penalize the event loop. It is associated with the server startup cost. However, jslint yells about using sync operations. As the code is part of the boilerplate for testing http-get, refactoring didn’t make enough sense. Making jslint to STFU is usually the last option. The content of the files never changes, therefore it doesn’t make any sense to read them from the disk either. Inlining is the obvious option.

Couldn’t find any online tool to play with. Therefore I fired a PHP REPL, then used my PCRE-fu to solve this one. The solution doesn’t look pretty, but it gets the job done:

php > var_dump(preg_replace('/\n/', '\n\\' . "\n", file_get_contents('server.key')));
string(932) "-----BEGIN RSA PRIVATE KEY-----\n\
MIICXAIBAAKBgQCvZg+myk7tW/BLin070Sy23xysNS/e9e5W+fYLmjYe1WW9BEWQ\n\
iDp2V7dpkGfNIuYFTLjwOdNQwEaiqbu5C1/4zk21BreIZY6SiyX8aB3kyDKlAA9w\n\
PvUYgoAD/HlEg9J3A2GHiL/z//xAwNmAs0vVr7k841SesMOlbZSe69DazwIDAQAB\n\
AoGAG+HLhyYN2emNj1Sah9G+m+tnsXBbBcRueOEPXdTL2abun1d4f3tIX9udymgs\n\
OA3eJuWFWJq4ntOR5vW4Y7gNL0p2k3oxdB+DWfwQAaUoV5tb9UQy6n7Q/+sJeTuM\n\
J8EGqkr4kEq+DAt2KzWry9V6MABpkedAOBW/9Yco3ilWLnECQQDlgbC5CM2hv8eG\n\
P0xJXb1tgEg//7hlIo9kx0sdkko1E4/1QEHe6VWMhfyDXsfb+b71aw0wL7bbiEEl\n\
RO994t/NAkEAw6Vjxk/4BpwWRo9c/HJ8Fr0os3nB7qwvFIvYckGSCl+sxv69pSlD\n\
P6g7M4b4swBfTR06vMYSGVjMcaIR9icxCwJAI6c7EfOpJjiJwXQx4K/cTpeAIdkT\n\
BzsQNaK0K5rfRlGMqpfZ48wxywvBh5MAz06D+NIxkUvIR2BqZmTII7FL/QJBAJ+w\n\
OwP++b7LYBMvqQIUn9wfgT0cwIIC4Fqw2nZHtt/ov6mc+0X3rAAlXEzuecgBIchb\n\
dznloZg2toh5dJep3YkCQAIY4EYUA1QRD8KWRJ2tz0LKb2BUriArTf1fglWBjv2z\n\
wdkSgf5QYY1Wz8M14rqgajU5fySN7nRDFz/wFRskcgY=\n\
-----END RSA PRIVATE KEY-----\n\
"
php > var_dump(preg_replace('/\n/', '\n\\' . "\n", file_get_contents('server.cert')));
string(892) "-----BEGIN CERTIFICATE-----\n\
MIICRTCCAa4CCQDTefadG9Mw0TANBgkqhkiG9w0BAQUFADBmMQswCQYDVQQGEwJS\n\
TzEOMAwGA1UECBMFU2liaXUxDjAMBgNVBAcTBVNpYml1MSEwHwYDVQQKExhJbnRl\n\
cm5ldCBXaWRnaXRzIFB0eSBMdGQxFDASBgNVBAMTC1N0ZWZhbiBSdXN1MCAXDTEx\n\
MDgwMTE0MjU0N1oYDzIxMTEwNzA4MTQyNTQ3WjBmMQswCQYDVQQGEwJSTzEOMAwG\n\
A1UECBMFU2liaXUxDjAMBgNVBAcTBVNpYml1MSEwHwYDVQQKExhJbnRlcm5ldCBX\n\
aWRnaXRzIFB0eSBMdGQxFDASBgNVBAMTC1N0ZWZhbiBSdXN1MIGfMA0GCSqGSIb3\n\
DQEBAQUAA4GNADCBiQKBgQCvZg+myk7tW/BLin070Sy23xysNS/e9e5W+fYLmjYe\n\
1WW9BEWQiDp2V7dpkGfNIuYFTLjwOdNQwEaiqbu5C1/4zk21BreIZY6SiyX8aB3k\n\
yDKlAA9wPvUYgoAD/HlEg9J3A2GHiL/z//xAwNmAs0vVr7k841SesMOlbZSe69Da\n\
zwIDAQABMA0GCSqGSIb3DQEBBQUAA4GBACgdP59N5IvN3yCD7atszTBoeOoK5rEz\n\
5+X8hhcO+H1sEY2bTZK9SP8ctyuHD0Ft8X0vRO7tdt8Tmo6UFD6ysa/q3l0VVMVY\n\
abnKQzWbLt+MHkfPrEJmQfSe2XntEKgUJWrhRCwPomFkXb4LciLjjgYWQSI2G0ez\n\
BfxB907vgNqP\n\
-----END CERTIFICATE-----\n\
"
php >

This gave me usable multi line strings that don’t break the PEM encoding.

Update: shell one liner with Perl

cat certificate.pem | perl -p -e 's/\n/\\n\\\n/'

SpiderOak – backs up stuff, uses client side encryption, has optional sync between your machines. So far, so good. In the latest OS X client, at least, the possibility to paste the password is missing. Thanks, I’ll me use my password manager instead with services that don’t do such a braindead thing. Seriously, there’s a thing that improves the security of the password authentication. It is called two factor authentication. Dropbox has it. Google has it. In fact, any decent service has it. Disabling the possibility to paste the password, not so much.

Google Drive – you wouldn’t think I’m letting Google of the hook this time. As I don’t trust with my data these sync services, I always do client side encryption. Dropbox doesn’t choke on it, SpiderOak doesn’t choke on it. Google Drive must be a special kind of breed as it chokes on my encrypted files with “Upload Error – An unknown issue has occurred ”. Gee, let me fix the error message for you: “your piece of shit encrypted files aren’t of any use for us, there’s no personal info there”. Was it that difficult? Thanks, but the market is full of alternatives. Seriously Google, you could do better than this “not being evil” thing.

Tested frameworks:

• node.js – v0.6.17
• vert.x – v1.0 final + OpenJDK 7 installed from the Ubuntu repository – using the JavaScript bindings
• luanode – built from the master branch using the Ubuntu provided lua dependencies
• luvit – built from the master branch
• react – cloned the master branch

I also wanted to test node.native, but it kept crashing on me. You can see that it is a pretty old issue. I didn’t have the patience to make the v0.1.0 branch to work with the previously used code. But I’d like to give it a run for its money.

The system used for the testing is a modest Athlon II X2 240e (2.8GHz) with 4GB or DDR2 800MHz running the latest Kubuntu 12.04 LTS amd64. Since ab pretty much takes a CPU core for itself, the frameworks ran a single process that occupied a single CPU core. I tried running a node.js HTTP server wrapped with the cluster module. Or passing -instances 2 to the vertx framework. The results were pretty much the same, therefore using just a single CPU core is a fair comparison.

The ab command that I used to hammer the Hello World! output:

ab -r -k -n 1000000 -c 1000 http://127.0.0.1:{port_name}/

The command ran at least a couple of times before saving the results. Just to make sure that everything is properly warmed up.

The averages graph:

The test sources and full ab output is available on this gist. There’s interesting output in the results.txt file for the stats nerds.

PS: I have the impression (but did not test) that vert.x may be a little bit faster, but ab is the actual bottleneck.

Update: added React (node.php) to comparision. Too lazy to plot another graph. But at 1573.40 req/s, it is harly a match even for luanode. Used the PHP 5.3.10 from the Ubuntu repositories.

Update: added another React (node.php) to comparision, but with a custom build of PHP 5.4.3. This time, it managed to get 3727.49 req/s.

When it turns out that it is not keeping the most basic promises. The AWS marketing machine did it. Again.

When it turns out that the latency is absolutely crap. I mean, SDB vs. RDS, as shown by New Relic: 183 ms vs. 1.6 ms. And I’m only talking about averages. Plotting the whole stuff on a graph along with the standard deviation will drive insane a statistician.

I could go about this all day long. But why bother.

node.js:201
        throw e; // process.nextTick error, or 'error' event on first tick
              ^
RangeError: Maximum call stack size exceeded

Oops, I smashed the stack. You may reproduce it with something like this:

var foo = []
 
for (var i = 0; i < 1000000; i++) {
    foo.push(i)
}
 
var recur = function (bar) {
    if (bar.length > 0) {
        var baz = bar.pop()
        // do something with baz
        recur(bar)
    } else {
        // end of recursion, do your stuff
    }
}
 
recur(foo)

“Thanks, that’s very thoughtful. But you’re not helping.” Bear with me. The solution is the obvious tail call elimination. But JavaScript doesn’t have that optimization.

However, you may wrap the tail call in order to call the above recur() function in a new stack. The proper recur() implementation is:

var recur = function (bar) {
    if (bar.length > 0) {
        var baz = bar.pop()
        // do something with baz
        process.nextTick(function () {
            recur(bar)
        })
    } else {
        // end of recursion, do your stuff
    }
}

Warning: please read this carefully. I gave you the solution for recurring over such a large input, but the performance is poor. Using process.nextTick (or a timer function such as setTimeout for that matter, slower BTW) is an expensive operation. Didn’t test where’s the actual bottleneck (epoll itself under Linux, libuv | libev, etc).

time node recur.js
node recur.js  1.36s user 0.28s system 101% cpu 1.610 total

The cost of this method is high. Therefore, don’t attempt this in a web application. It kills the event loop. For instance, I don’t use node for writing web applications. It is a difficult task, while the cost of the event loop itself isn’t that negligible as you may think. It useful as long as the CPU time is negligible compared to the time spent doing IO. Therefore, please, don’t include me in the group of people that thinks about node as the hammer for all the problems you throw at it.

If you’re wondering why I won’t just simply iterate the object, the answer is simple: because that “do something with baz” involves some async IO that would kill the second data provider. Sequential calls ensure that everybody in the architecture stays happy. Besides, I don’t actually use bar.pop(), but something like bar.splice(0, 5000) for packing more data in less remote calls and less events. bar.shift() in a situation like this is as slow as molasses in January. In an async framework, the order of the items from a TODO list is not relevant, therefore use the fastest way.

If you’re still wondering why I still use a solution like this, the above technique is part of the cost associated with the start-up cost. The application fetches all the required data in RAM. Having the application to kill the event loop for 20-30 seconds before hitting the Internet pipe is negligible for a process that runs for hours or days. After the application hits the Internet, only then I can say that node is in use for the stuff where it shines. I know, before this, I listed all the wrong reasons for using node as a tool.

Let’s have a look:

var filename = process.argv[2];
var crypto = require('crypto');
var fs = require('fs');
 
var shasum = crypto.createHash('sha256');
 
var s = fs.ReadStream(filename);
s.on('data', function(d) {
  shasum.update(d);
});
 
s.on('end', function() {
  var d = shasum.digest('hex');
  console.log(d + '  ' + filename);
});

time node hash.js ubuntu-10.04.3-desktop-i386.iso
208fb66dddda345aa264f7c85d011d6aeaa5588075eea6eee645fd5307ef3cac ubuntu-10.04.3-desktop-i386.iso
node hash.js ubuntu-10.04.3-desktop-i386.iso 28.92s user 0.80s system 100% cpu 29.661 total

time sha256sum ubuntu-10.04.3-desktop-i386.iso
208fb66dddda345aa264f7c85d011d6aeaa5588075eea6eee645fd5307ef3cac ubuntu-10.04.3-desktop-i386.iso
sha256sum ubuntu-10.04.3-desktop-i386.iso 4.86s user 0.21s system 99% cpu 5.093 total

time openssl dgst -sha256 ubuntu-10.04.3-desktop-i386.iso
SHA256(ubuntu-10.04.3-desktop-i386.iso)= 208fb66dddda345aa264f7c85d011d6aeaa5588075eea6eee645fd5307ef3cac
openssl dgst -sha256 ubuntu-10.04.3-desktop-i386.iso 4.40s user 0.17s system 100% cpu 4.567 total

Edit: to sum up for those with little patience:

node hash.js – 29.661s
sha256sum – 5.093s
openssl dgst -sha256 – 4.567s

/Edit

That’s a ~6.5X speed boost just by invoking openssl alone instead of binding to its library. node.js does something terribly wrong somewhere since the file I/O is not to blame for the slowness:

var filename = process.argv[2];
var fs = require('fs');
 
var s = fs.ReadStream(filename);
s.on('data', function(d) {
 
});
 
s.on('end', function() {
 
});

time node read.js ubuntu-10.04.3-desktop-i386.iso
node read.js ubuntu-10.04.3-desktop-i386.iso 0.62s user 0.60s system 106% cpu 1.148 total

This little example that I hacked together shows that using child_process.exec is pretty fine:

var exec = require('child_process').exec;
exec('/usr/bin/env openssl dgst -sha256 ' + process.argv[2], function (err, stdout, stderr) {
	if (err) {
		process.stderr.write(err.message);
	} else {
		console.log(stdout.substr(-65, 64));
	}
});

time node hash2.js ubuntu-10.04.3-desktop-i386.iso
208fb66dddda345aa264f7c85d011d6aeaa5588075eea6eee645fd5307ef3cac
node hash2.js ubuntu-10.04.3-desktop-i386.iso 4.44s user 0.19s system 100% cpu 4.630 total

So you can have your cake and eat it too. The guys with the philosophy got this one right.

This first chapter is not about recursion. One member of the community wrote that certain inflammatory statements that I use may upset people. I replied with: “buzz marketing”. Neutral articles, with neutral titles, written by nobodies like me, gain zero traction, although I may write something that’s technically sound. Cheap journalism has more success. I even have graphs to prove it now.

The second stuff is the usefulness of my little experiment. I don’t know about others, but the curiosity was the main thing behind my whole benchmark. If it isn’t useful for some people, it doesn’t mean that it isn’t useful for others.

The other thing: the lack of tail recursion. I mean, do you need a “DUH” award, or something? The whole point of a “bad” algorithm that’s mathematically correct (well, almost, I stated that fibonacci(0) is wrong) is to prove how smart are specific compilers regarding recursion. The rest, simply do brute force.

Patterns that emerge

The numbers say something if you know how to read the page. There are runtimes that are optimized for doing proper recursion without bothering the programmer with it: C, D, PyPy, V8, LuaJIT, JVM. The rest aren’t: PHP, CPython, Ruby, Perl, Lua. PyPy and V8 could do better. LuaJIT is already close to the speed of unoptimized C and D. V8 isn’t the king of the hill if you take Ruby (MRI / KRI), CPython, plain Lua VM, and PHP (Zend Engine) out of the equation. This may be another opportunity to get bashed by the node.js benchmark police with “this is irrelevant” statements, although this wasn’t something that I wanted to prove.

Thing is, that for most of the web development, I rarely needed to actually solve purely recursive problems. At most a fairly simple tree. Sometimes even that simple tree didn’t actually require recursion. Therefore I get why some don’t optimize for this specific case, although they refer the thing as being “a general purpose language”.

For the “write better algorithms” crowd … WHY? The difference between C’s 0.6 seconds and Ruby’s 5 minutes doesn’t ring any bell that some things are fundamentally flawed regarding recursion?

As for the edge cases, there are 3rd party libraries that solve this issue without bothering the programmer. Or for other edge cases, such as applications that do complicated stuff, operating at Google-like scale, there are better tools that most mere mortals won’t use. The fact that some implementation do poor recursion is indeed irrelevant when the problems you’re trying to solve don’t include this.

In the end

Initially I wanted to try more stuff such as factorial, Euclid’s GCD, or the Ackermann function, for example. Try them on runtimes that don’t take longer than the next ice age to return a value. But why bother, except maybe to give the “one true way of doing recursion in functional languages” programmers a reason to bash stuff without returning any useful output. Not even an academic paper. It’s not productive.

The source code of all the tests is available here in order to make the tests to be reproducible. There wasn’t a high number of runs, particularly for the rutimes that take more than the next ice age. But the results are pretty consistent for specific runtimes. The relevant systems specs are: Ubuntu 10.04 amd64 (up to date), Q9400 CPU.

Now, less talk, more results.

JavaScript (node.js/V8)

node -v: v0.4.12
time node fib.js
node fib.js 6.40s user 0.02s system 99% cpu 6.423 total
node fib.js 6.39s user 0.02s system 99% cpu 6.410 total

It may seem slow, but for a language with dynamic typing, it puts the rest from the same category to shame. Or most of them. Bear with me.

PHP

php -v: PHP 5.3.8 (cli)
time php fib.php
php fib.php 77.57s user 0.06s system 99% cpu 1:17.66 total
php fib.php 78.05s user 0.07s system 99% cpu 1:18.18 total

Compared to the V8 runtime, PHP seems to take an eternity. It happens that PHP isn’t bad at recursion because it uses the stack, but because the lack of speed of the runtime. But we’re not even halfway there. Stay tuned. PHP isn’t the only thing that sucks at recursion.

C

gcc -v: gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5)
make fib
time ./fib
./fib 2.84s user 0.00s system 100% cpu 2.840 total
./fib 2.84s user 0.00s system 100% cpu 2.835 total

It wasn’t a surprise that C came up to this result. Which makes the V8 result even more interesting.

Edit:
Forgot about the compiler optimization. Caught by Jabbles on HN.

gcc -O4 fib.c -o fib
time ./fib
./fib 0.65s user 0.01s system 100% cpu 0.657 total
gcc -O3 fib.c -o fib
time ./fib
./fib 0.66s user 0.00s system 100% cpu 0.657 total
gcc -O2 fib.c -o fib
time ./fib
./fib 1.54s user 0.00s system 100% cpu 1.535 total
gcc -O1 fib.c -o fib
time ./fib
./fib 0.00s user 0.00s system 0% cpu 0.001 total
time ./fib
165580141
./fib 2.06s user 0.00s system 99% cpu 2.060 total

For some reason, the O1 flag hates this code. Printing fibonacci(40) yields a result closer to the result without any O flag. This brings it past the Java result, but only for O3+.
/End Edit.

Lua

lua -v: Lua 5.1.4
time lua fib.lua
lua fib.lua 28.02s user 0.03s system 99% cpu 28.081 total
lua fib.lua 28.86s user 0.02s system 99% cpu 28.883 total

./luajit -v: LuaJIT 2.0.0-beta8
time ./luajit fib.lua
./luajit fib.lua 10.59s user 0.00s system 99% cpu 10.591 total
./luajit fib.lua 10.58s user 0.01s system 99% cpu 10.610 total

Tested both of the implementations that I know of. I guess this article isn’t that funny for the generations of Lua coders that laugh about V8 in somebody’s face. Don’t get me wrong, I like Lua due to its simplicity, but in the speed realm, I still need to do some tests to verify some of those claims that sometimes appear to be overly inflated.

Edit:

With the following Lua script:

local function fibonacci(n)
	if n < 2 then
		return 1
	else
		return fibonacci(n - 2) + fibonacci(n - 1)
	end
end
fibonacci(40)

the results are getting better:

time lua fib.lua
lua fib.lua 24.17s user 0.08s system 99% cpu 24.281 total
lua fib.lua 24.24s user 0.01s system 99% cpu 24.307 total

[with LuaJIT v2.0.0-beta8 GIT HEAD]
time ./luajit fib.lua
./luajit fib.lua 2.02s user 0.00s system 99% cpu 2.026 total
./luajit fib.lua 2.02s user 0.00s system 99% cpu 2.023 total

Now, some of the Lua chops can have a lulz about V8. This project is getting more interesting, especially for pairing LuaJIT with luafcgid. I forgot about the local keyword since my Lua experience is limited to basic testing. Nice comeback!

/End Edit.

Python

python -V: Python 2.6.5
time python fib.py
python fib.py 59.42s user 0.02s system 99% cpu 59.494 total
python fib.py 59.27s user 0.05s system 99% cpu 59.375 total

./configure
make -j 4
./python -V: Python 2.7.2
time ./python fib.py
./python fib.py 61.29s user 0.03s system 99% cpu 1:01.35 total
./python fib.py 61.38s user 0.06s system 99% cpu 1:01.48 total

./configure
make -j 4
./python -V: Python 3.2.2
./python fib.py 71.23s user 0.08s system 99% cpu 1:11.33 total
./python fib.py 70.31s user 0.06s system 99% cpu 1:10.39 total

./pypy -V
Python 2.7.1 (d8ac7d23d3ec, Aug 17 2011, 11:51:19)
[PyPy 1.6.0 with GCC 4.4.3]
./pypy fib.py 4.61s user 0.07s system 99% cpu 4.708 total
./pypy fib.py 4.81s user 0.01s system 99% cpu 4.853 total

Happens to have 2.6.5 around because Ubuntu says so. But in order to make the potential trolls to STFU about not using the latest versions, I made some fresh builds of 2.7.2 and 3.2.2. It gets even suckier with recent versions. In fact, the CPython runtime is struggling to catch up the PHP runtime on the slowness realm. The only Python runtime that is actually very impressive about the recursion stuff is PyPy. Which brings me to the first statement: the language is just a bunch of text. The runtime is the piece that sucks or does not suck. PyPy proves that with talented people shepherding the project, the language of the runtime implementation is quite irrelevant. This is the first implementation of a JIT that passes V8 as well.

Ruby

ruby -v: ruby 1.8.7 (2010-01-10 patchlevel 249) [x86_64-linux]
time ruby fib.rb
ruby fib.rb 233.40s user 66.94s system 99% cpu 5:00.55 total
ruby fib.rb 231.75s user 68.08s system 99% cpu 4:59.99 total

./configure
make -j 4
./miniruby -v: ruby 1.9.3dev (2011-09-23 revision 33323) [x86_64-linux]
time ./miniruby fib.rb
./miniruby fib.rb 36.05s user 0.01s system 99% cpu 36.073 total
./miniruby fib.rb 35.92s user 0.04s system 99% cpu 35.978 total

Everytime a Ruby fan says that “thou shalt not care about the runtime speed” makes me laugh so hard up to the point of bursting into tears. Seriously, I couldn’t imagine that MRI sucks that hard at recursion. I barely had the patience to even run this code. KRI washes part of the shame though while it scores closely to the Lua implementation. If you’re asking why I used the miniruby binary, the reason is that the ruby binary complained about not having rubygems.rb. I am bad at figuring out what’s missing from a Ruby stack. But it made the fib.rb work.

Perl

perl -v: This is perl, v5.10.1 (*) built for x86_64-linux-gnu-thread-multi
time perl fib.pl
perl fib.pl 125.60s user 0.17s system 99% cpu 2:05.92 total
perl fib.pl 124.06s user 0.10s system 99% cpu 2:04.20 total

./Configure [accepted all defaults, specifically built without threading]
make
./perl -v: This is perl 5, version 14, subversion 2 (v5.14.2) built for x86_64-linux
./perl fib.pl 100.12s user 0.09s system 99% cpu 1:40.29 total
./perl fib.pl 100.38s user 0.05s system 99% cpu 1:40.63 total

At first I didn’t want to bother with Perl, but then I remembered the legions of Perl fans ranting about the PHP recursion. I know that this is an inflammatory statement, but next time, people, please keep up with the facts. I guess you aren’t that smug now.

D

gdc -v: gcc version 4.3.4 (Ubuntu 1:1.046-4.3.4-3ubuntu1)
gdc -o fib fib.c (same source as the C binary)
time ./fib
./fib 2.82s user 0.00s system 100% cpu 2.817 total
./fib 2.82s user 0.00s system 100% cpu 2.814 total

Predictable results from a language from the same family as C/C++. Slightly faster binary that the C version (although the same source code), but I guess most people won’t notice.

Java

javac -version: gcj-4.4 (Ubuntu 4.4.3-1ubuntu4.1) 4.4.3
java -version
java version “1.6.0_20″
OpenJDK Runtime Environment (IcedTea6 1.9.9) (6b20-1.9.9-0ubuntu1~10.04.2)
OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
javac fib.java
time java fib
java fib 0.86s user 0.02s system 99% cpu 0.882 total
java fib 0.86s user 0.02s system 100% cpu 0.872 total

I admit that sometimes I use to tell this joke: knock! knock!; who’s there?; [very long pause]; Java. I guess that now is a good time to swallow my own words. Not only that Java puts the other JIT implementations to shame, the rest of the VMs to shame, it also obliterates the statically compiled C and D binaries at their own favorite game aka the runtime speed. My first reaction was: WTF, there’s got to be a mistake! Printing some junk to STDIO confirmed the same results between C and Java. Newbie warning: this is my first Java application. No, really! Don’t bash me for the lack of understanding of the usage of the static keyword. I don’t understand if it actually helps the runtime. I managed to put together the code by reading how to write a simple HelloWorldApp. Experienced Java chops may explain it though. ]]> http://www.saltwaterc.eu/but-the-question-is-will-it-recur-part-1-fibonacci40.html/feed 16 http://www.saltwaterc.eu/use-the-cache-luke-part-1-from-memcached-to-membase-memcached-buckets.html http://www.saltwaterc.eu/use-the-cache-luke-part-1-from-memcached-to-membase-memcached-buckets.html#comments Wed, 21 Sep 2011 13:07:44 +0000 SaltwaterC http://www.saltwaterc.eu/?p=1321 I start with a quote:

Matt Ingenthron said internally at Membase Inc they view Memcached as a rabbit. It is fast, but it is pretty dumb and procreates quickly. Before you know it, it will be running wild all over your system.

But this post isn’t about switching from a volatile cache to a persistent solution. It is about removing the dumb part from the memcached setup.

We started with memcached as this is the first step. The setup had its quirks since AWS EC2 doesn’t provide by default a fixed addressing method while the memcached client from PHP still has issues with the timeouts. Therefore, the fallback was the plain memcache client.

The fixed addressing issue was resolved by deploying Elastic IPs with a little trick for the internal network, as explained by Eric Hammond. This might be unfeasible for large enough deployments, but it wasn’t our case. Amazon introduced ElastiCache since then which removes this limitation, but having a bunch of t1.micros with reservation is still way much cheaper. Which makes me wonder why they won’t introduce machine addresses which internally resolve as internal address. They have this technology for a lot of their services, but it is simply unavailable for plain EC2 instances.

Back to the memcached issues. Having a Membase cluster that provides a memcached bucket is a nice drop-in replacement, if you lower a little bit your memory allocation. Membase over memcached still has some overhead as its services tend to occupy more RAM. The great thing is that the cluster requires fewer machines with fixed addressing. We use a couple for high availability reasons, but this is not the rule. The rest have the EC2 provided dynamic addresses. If a machine happens to go down, another one can take up its place.

But there still is the client issue. memcached for PHP is dumb. memcache for PHP is even dumber. None of these can actually speak the Membase goodies. This is the part where Moxi (Memcached Proxy) kicks in. For memcached buckets, Moxi can discover the newly added machines to the Membase cluster without doing any client configuration. Without any Moxi server configuration as the config is streamed to the servers via the machines that have the fixed addresses. With plain memcached, every time there was a change, we needed to deploy the application. The memcached cluster was basically nullified till it was refilled. Doesn’t happen with Moxi + Membase. Since there no “smart client” for PHP which includes the Moxi logic, we use client side Moxi in order to reduce the network round-trips. There still is a local communication over the loopback interface, but the latency is far smaller than doing server-side Moxi. Basically the memcache for PHP client connects to 127.0.0.1:11211 aka where Moxi lives, then the request hits the appropriate Membase server that holds our cached data. It also uses the binary protocol and SASL authentication which is unsupported by the memcache for PHP client.

The last of the goodies about the Membase cluster: it actually has an interface. I may not be an UI fan, I live most of my time in /bin/bash, but I am a stats junkie. The Membase web console can give you realtime info about how the cluster is doing. With plain memcached you’re left in the dust with wrapping up your own interface or calling stats over plain TCP. Which is so wrong at so many levels.

PS: v2.0 will be called Couchbase for political reasons. But currently the stable release is still called Membase.

Today’s case was a debate about the Expires header timestamps as defined by the HTTP/1.1 specs (RFC 2616). If you read the 14.21 section regarding the Expires header, you can see the following statement:

The format is an absolute date and time as defined by HTTP-date in section 3.3.1; it MUST be in RFC 1123 date format:

Expires = “Expires” “:” HTTP-date

I made a newb mistake in thinking that the RFC 1123 dates are legal Expires timestamps. Actually, by proof reading 3.3.1 of RFC 2616 you may deduce the following: the dates in use by the HTTP/1.1 protocol are not the dates into the RFC 1123 format, but the actual format is a subset of RFC 1123. The debate started around the GMT specification which in the HTTP/1.1 contexts is actually UTC, but it must be specified as GMT anyway. Even more, +0000 which is valid timezone specifier as defined by RFC 1123 is not valid for Expires timestamps. Although some caches accept +0000 as valid timezone specifier for the HTTP timestamps, some of them don’t.

It isn’t that the RFCs are broken per se, but the language they use can be very confusing sometimes.

Days started to fly, queries started to fill our log, we started to fix the slow points of the application. The thing that didn’t change is the fact that the mysql.slow_log table kept growing. Then I took some time to apply all my MySQL-fu regarding the cleanup of the mysql.slow_log table. Imagine my surprise when none of it worked. Since the master user of a RDS instance doesn’t have all the privileges, it wasn’t quite unexpected though.

For the first time, the AWS Premium Support was actually useful by sending one email that actually provides a solution. Imagine my surprise. The RDS team implemented a couple of stored procedures that can be used for rotating the slow log and the general log.

CALL mysql.rds_rotate_slow_log;
CALL mysql.rds_rotate_general_log;

Basically they move the content to a *_backup table while the original is replaced by an empty table. The exact quote:

When invoked, these procedures move the contents of the corresponding log to a backup table and clear the contents of the log. For example, invoking rds_rotate_slow_log moves the contents of the slow_log table to a new table called slow_log_backup and then clears the contents of the slow_log table. This is done by renaming tables, so no data is actually copied, making this a very light-weight, non-blocking procedure. Invoking the same procedure twice effectively purges the log from the database.

They are present since March 22, 2010 but nobody took the time to document them, apparently. All I could find via online searches was utterly useless junk. I hope this saves some time for some poor chop into the same situation as I was.

The thing that sometimes pisses me off is stuff like this article written by somebody who understands little about the web security. Writing the title with upper case letters doesn’t bring any value to the argument. So let’s dig a little bit better.

a PHP application that has display_errors enabled should never be in production

If you don’t see anything wrong about the quoted sentence, then you’re into the wrong field. Three of the most basic rules of security are:

1. don’t rely onto the defaults
2. all input is evil
3. don’t rely onto a single layer of security

The application doesn’t have display_errors turned on. The PHP runtime does. Sometimes this may be onto a shared host that doesn’t give a crap about your application. Or the sysadmin had a bad day and simply forgot something. Or, even better, somebody defined display_errors = true with php_admin, so the application can’t do anything about its own security context. “Should not | never” is an idiom that should not be used in any text which is security related. In the end, an attacker gain access through that critical piece of code that should not fail.

Which brings us to the next rule: the input validation. People should validate their input but this doesn’t enforce the fact that mistakes don’t happen. PHP makes it quite easy to both develop fast an application and shoot yourself in the foot at the same time. Properly developed code takes a lot of effort. If you didn’t developed code that treats notifications as fatal exceptions, then you don’t have a clue what I’m talking about. I am not talking about hello.php applications, but large projects built for scale. Couple of years ago I was working on a project built on top of the Kohana framework that used this specific setup in development mode. The consequence was the fact that much of the errors that usually creep into the production code were discovered during the development phase while my team learned how to properly initialize all the stuff that floats around the application. Few people have the patience to work this way. Some of the stuff can be automatically handled via proper abstractions, some of the stuff can’t or it’s simply not practical. If you failed the input validation class, there’s the next point.

Finally the third rule which says that’s bad practice to depend onto a single layer of security aka in this case simply turning off the display_errors. If PHP allows you to shoot yourself in the foot by having display_errors = true onto a production machine, doesn’t mean that it should make a harmless mistake (dumping the error reporting content to the screen) to be more severe (having an XSS in the application). If you regard XSS as harmless, you have much to learn.

The next quote that’s also goes into the clueless realm is:

I don’t want to see anything to do with HTML if I’m not doing web-based programming with PHP (such as CLI).

Since PHP exposes the PHP_SAPI constant which tells exactly what SAPI is in use, I guess that the runtime itself is smart enough to know exactly where the output goes: down a network pipe as HTML or into STDOUT / STDERR. A smart enough runtime also knows when to encode its output as appropriate and when not to. The argument holds water maybe just for the embed SAPI which is seldom used. I dare you to find proper docs about using the embed SAPI in your C / C++ application.

I want my error messages to provide me with precise information. I do not want an error message to arbitrarily encode problematic code. If the user provided test then I want to see test. I do not want to see <s>test</s> since that is not what was submitted and, possibly, not what the problem is.

These sentences enter the realm of self parody. If the user entered <s>test</s> that’s what you should see into an error output. That’s what you can find into an error log. Rendering the HTML markup is a browser feature therefore the output should play by the browser rules. Would you also like to see <script> tags as HTML instead of “what the user actually entered”?

The data corruption is the specific topic that snapshots simply don’t care about, at least in Amazon’s way of doing things. This isn’t exactly Amazon’s fault for EC2. EBS actually stands for Elastic Block Storage. They provide you a block storage, you do whatever you want with it. For RDS they should do a better job though as it’s a managed service where you don’t have access to the actual instance. The issue is those ‘specialists’ that put emphasis onto the ‘easy, cloud-ish way’ of doing backups by using snapshots. If you’re new to the ‘cloud’ stuff as I used to be, you may actually believe that crap. As I used to believe.

A couple of real life examples:

• An EBS-backed instance suffered some filesystem level corruption. Since EXT3 is not as smart as ZFS if we’re talking about silent data corruption, you may never know until it’s too late. Going back through revisions in order to find the last good piece of data is a pain. I could fix the filesystem corruption, I could retrieve the lost data, but I had to work quite a lot for that. Luck is an important skill, but I’d rather not put all my eggs into the luck basket.
• An RDS instance ran out of space. There wasn’t a notification to tell me: ‘yo dumbass, ya ran out of space’. Statistically it wasn’t the case, but a huge data import proved me wrong. I increased the available storage. Problem solved. A day later, somebody dropped by accident a couple of tables. I had to restore them. How? Take the latest snapshot, spin up a new instance, dig through the data. The latest snapshot contained a couple of corrupted databases due to the space issue, one of them being the database I needed to restore. I had to take a bunch of time in order to repair the database before the restoration process. Fortunately nothing really bad happened. But it was a signal that the RDS snapshot methodology is broken by design.

Lesson learned. The current way of doing backups puts the data, not the block storage, first. If you’re doing EBS snapshots as the sole method, you may need to rethink your strategy.

Apparently this ought to be a relative easy task, right? Wrong! I don’t know if Amazon is to blame. Or Microsoft. Or both. Really don’t care. The annoying thing is the large pile of fail that destroys the productivity for the most basic tasks. However, besides cookies, we, the sysadmins of the Dark Side, also have solutions. We have to.

Some that know me better may think that I hate MS/Windows. I don’t, as long I don’t have to administer those machines. I do have a Windows 7 Pro box at home. While the Windows soft-RAID implementation blows for certain reasons, otherwise than this and some of the small quirks that every OS has, is a pretty dandy setup. Therefore no, I am not a pure Windows hater. Don’t get started with the wrong impression.

The first thing that’s really annoying is the long period that takes for the instance to be ready. In order to get the ‘Administrator’ password it takes ages. Amazon says it takes between 15-30 minutes. Come on EC2, you take a AMI off the S3 and spin up a new instance, then setup a new password. How hard can it be? A Linux instances is ready almost instantly after the console says it’s running. Even more, after the password is available via the console or the API (sometimes one of these gets it faster) the RDP still takes a few minutes before being available. Since Amazon & Microsoft don’t provide this for free, even more, we pay by the hour, STOP WASTING OUR TIME. Sincerely, the EC2 Windows Server users.

The second really annoying thing is the fact that Amazon recommends you to change the default password that they provide. While the recommendation is good, if you do this, you get locked out the instance. If you don’t have a second account with RDP access, you have one option: terminate the instance. The almost useless AWS Premium support probably will be able to do something about this. But if you’re in a hurry, going back to square one is probably the best option.

You have a workaround though that I discovered during my time boxing with EC2 and Windows Server: add the Administrator to the RDP allowed list, although the window says that ‘Administrator already has access’. If you change the password, the Administrator loses the RDP access, even though the password is valid. The welcome screen will tell you rubbish about using the wrong password, when clearly the input is good. I tried to reproduce this few times. Each time I got locked out. I even validated the new password by running something with runas. What? Wrong password? Listen a’holes, I asked you to change the f’ing password, not make me French fries. How hard is not to screw up such a basic task? Initially I though my Terminal Server Client is dumb, but the *nix guys don’t have any screw ups here. Using a Windows 7 client did the same.

If you want to create a new AMI off the existing instance, boy, more gotchas are waiting around the corner. Even if the Administrator password is successfully changed, the RDP access is preserved, the ‘Set Password’ options of the ‘Ec2 Service Properties’ application are unchecked, sysprep manages to tank the instance and the new AMI. Administrator completely loses the access on both instances, the existing and new new one. Tried with the ‘Set a random password’ option. Didn’t work as expected. Didn’t try the ‘SetPassword’ feature after sysprep, but I had enough anyway. I read somewhere around the Interwebs that sysprep somewhat affects the Administrator user, but I didn’t get any relevant information. Searching for it is a mess.

My workaround: create a new user part of the Administrators group. Set a strong password for that user. Amazon sets a joke anyway. I wouldn’t call that strong. Disable the Administrator account:

net user Administrator /active:no

Enjoy your newly created AMIs without surprises. Didn’t have any issues with other Administrators besides Administrator. Plus it is available as soon as the RDP service is available, unlike the ‘Amazon way’ of keeping you waiting 15 to 30 minutes, for nothing.

The third really annoying thing is the fact that the AMIs are awfully outdated. Since there’s an official partnership between Amazon and Microsoft I would expect a higher level of support. I know, there’s Windows Update, but even after all those years, it still doesn’t manage to do it into a brain dead-free way. I know I’m biased as I’m used to Debian and friends, but the OS upgrade could be a little bit faster. OK, a lot more faster. Downloading and installing 54 updates that have around 200MiB takes around a hour and half, on a c1.medium instance which is the best specs you can get from a x86 instance. It takes 3 reboots. It takes 4 times to click the ‘Check for updates’ link in order to actually be up to date in the end. I get that there are dependencies, but an OS that claims is ‘Datacenter’ grade should handle this properly: download all your junk, deal with it later. Even more, I used c1.medium because m1.small is even slower for the Windows Update part. Even when the update manager claims that it’s only downloading updates, the CPU usage won’t go under 50%. What? A download that takes a lot of CPU time? WTF? The actual update doesn’t do it nicer or faster anyway.

A fellow sysadmin suggested vbscript + AutoIt. Now why the hell should I write a script to fix issues that should not exist in the first place? Don’t know why, but Windows Server under EC2 still feels like a second class citizen, despite the fact that you pay extra for the privilege of running Windows. Anyway, in the end I managed to bake my own AMI that’s ready for spinning up new instances for our HA setup. But the experience wasn’t even remotely close to the advertised level.

> bashing mode off

However, the thing that applies differently is the fact that the maximum connection backlog on a OpenVZ VPS is typically 128, therefore you can not increase the listen.backlog parameter of PHP-FPM over that value. The kernel denies you that. Even more, due to the low memory setups of a lot of VPSes out there in the wild, you may be forced to use a low value for pm.max_children which translates this into a shorter cycle for the process respawn. A process that is respawning, can’t handle incoming connections, therefore the heavy lifting is done by the kernel backlog. The concurrency may not increase linearly with the number of available processes into the FPM pool because of this consequence.

Since the backlog is kept by the kernel, it is common for all services, therefore it may affect nginx as well if it can not accept() a connection for some reason. Load balancing over UNIX sockets doesn’t increase the possible concurrency level. It simply adds useless complexity. Using TCP sockets may increase the reliability of the setup, at the cost of added latency and used up system resources, but it would fail in the end. Some measures need to be taken.

I gave a thought about using HAProxy as connections manager since HAProxy has a nice maxconn feature that would pass the connections to an upstream up to a defined number. But that would add another layer of complexity. It has its benefits. But it also means that’s another point of failure. Having more services to process the same request pipeline is clearly suboptimal if the services composing the request pipeline won’t add some clear value to the setup, for example the way a proxy cache does.

Then I thought about node.js, but implementing FastCGI on top of node seems a hackish solution at best. One of the Orchestra guys did it, but I wouldn’t go into production with a node.js HTTP server for static objects and FastCGI response serving, no matter how “cool” this may sound. Then the revelation hit me: Ryan Dahl, the author of node.js, wrote a nginx module that adds the maxconn feature to nginx: nginx-ey-balancer. Hopefully, someday this would make it into upstream.

The module adds some boilerplate to the configuration since it requires a fastcgi_pass to an upstream unlike direct fastcgi_pass to an UDS path, but otherwise that this, it works as advertised. Although the module wasn’t actively maintained, or, at least this is how things look from outside, the v0.8.32 patch works even for nginx v1.0.2. Having nginx to act as connection manager instead of sending all the connections straight to the FPM upstream may have clear benefits from the concurrency point of view. It is recommended to set max_connections to the value of net.core.somaxconn. That guarantees the fact that no connection gets dropped because the FPM pool processes are respawning due to a short cycle policy.

By using this module, nginx could handle easily around 4000 concurrent connections for a 4 worker process setup, but increasing the workers number does not increase linearly the possible concurrency. Anyway, at that concurrency level, most probably the issues caused by the IPC between nginx and PHP-FPM would be your last concern. This setup simply removes an IPC limit which is ridiculously small most of the times.

Since then, I keep telling people that Apache2 + mod_php5 + prefork MPM is a memory hog, while most of the stuff comes from the brain-dead client connection management that Apache does. One process per connection is a model that ought to be killed. Probably the best bang for the buck is using nginx as front-end for serving the static objects and buffering the response from Apache in order to spoon-feed the slow clients. But here’s the kicker: for virtual hosting setups, Apache/prefork/PHP is pretty dull to configure safely aka isolate the virtual hosts. Packing more applications together over a production cluster is an obvious way for doing server consolidation.

There’s mod_fcgid/FastCGI … but nginx supports FastCGI as well. Therefore, cutting off the middle man was the obvious solution. By using this setup, you won’t lose … much. PHP-FPM was an obvious solution as the default FastCGI manager that used to come as the sole solution from the PHP upstream is pretty dumb. Instead of having a dedicated PHP service for each virtual host, one can have a dedicated process pool for each virtual host. Believe me, this method is much easier to maintain.

While nginx comes pretty well tuned by default, except the open file cache settings, for PHP-FPM you need to tune some essential settings. nginx has a predictable memory usage. I can say that most of the time the memory used by nginx is negligible compared to the memory used by the PHP subsystem. However, PHP-FPM has a predictable memory consumption if used properly. The process respawn feature ought to be used in order to keep the memory leaks under control. pm.max_requests is the option that you need to tune properly. One may use 10000 served request before respawning, but under very restricted memory requirements I even had to use 10 (very low memory VPS).

The pm.max_children option may be used to spawn an adaptive number of processes, based onto the server load, but IMHO that might overcommit the system resources into the worse case scenario. Having a fixed number of processes per pool is preferred. Usually I have a rough estimation of the memory consumption in order to keep all the runtime into the RAM. Thrashing the memory is not something you would want onto a loaded web server. For everything else, there’s Master … cough! munin.

The Inter-Process Communication is preferred over an UNIX Domain Socket. Unlike the TCP sockets, the UDS doesn’t have the TCP overload that the IPC has even for connections over the loopback interface. For small payloads, UDS might have a 30% performance boost due to lower latency than the TCP stack. Rememer: you can’t beat the latency. For larger payloads, the TCP latency has a lower impact, but it’s still there. Another nice thing about the UDS is the namespace. UDS uses the filesystem for defining a new listening socket. Under Linux, classic ACLs may be used for restricting what user can read / write to the UDS. BSD systems may be more permissive for this kind of stuff. The TCP sockets require a numerical port that can’t be used for something else while the management from an application that generates the configuration files is more difficult for this kind of settings. For UDS the socket name can be derived from the host name. As I said into a previous article, I don’t write the configuration files by hand. Having a decent way of keeping the IPC namespace is always a plus.

Another thing that you should take care of is the listen.backlog option of PHP-FPM. Using TCP sockets seem to be more reliable for PHP-FPM. However, that’s untrue if you dig enough. The IPC starts to fail around 500 concurrent connections, while for UDS this happens way faster aka for an 1 process pool, you can serve at most 129 concurrent clients. 129 is not a random number. PHP-FPM can keep 128 connections into its backlog while the 129th connection is the active process. The default listen.backlog for Linux is 128, although the PHP-FPM documentation may state -1 aka the maximum allowed value by the system. Taking a peek at the PHP-FPM source code reveals this (sapi/fpm/fpm/fpm_sockets.h):

/*
  On FreeBSD and OpenBSD, backlog negative values are truncated to SOMAXCONN
*/
#if (__FreeBSD__) || (__OpenBSD__)
#define FPM_BACKLOG_DEFAULT -1
#else
#define FPM_BACKLOG_DEFAULT 128
#endif

The default configuration file that is distributed with the PHP-FPM source tree states that the value is 128 for Linux. The php.net statement that it defaults to -1 gave me a lot of grief as I though the manual won’t give me rubbish instead of usable information. However, since PHP 5.3.5 you may debug the configuration by using the -t flag for the php-fpm binary. You can use it like:

php-fpm -tt -y /path/to/php-fpm.conf

The doube t flag is not a mistake. If you’re using NOTICE as the debug level, the double t testing level prints the internal values of the PHP-FPM configuration:

php-fpm -tt -y /etc/php-fpm/php-fpm.conf
[08-May-2011 20:53:26] NOTICE: [General]
[08-May-2011 20:53:26] NOTICE:  pid = /var/run/php-fpm.pid
[08-May-2011 20:53:26] NOTICE:  daemonize = yes
[08-May-2011 20:53:26] NOTICE:  error_log = /var/log/php-fpm.log
[08-May-2011 20:53:26] NOTICE:  log_level = NOTICE
[08-May-2011 20:53:26] NOTICE:  process_control_timeout = 0s
[08-May-2011 20:53:26] NOTICE:  emergency_restart_interval = 0s
[08-May-2011 20:53:26] NOTICE:  emergency_restart_threshold = 0
[08-May-2011 20:53:26] NOTICE:
[08-May-2011 20:53:26] NOTICE: [www]
[08-May-2011 20:53:26] NOTICE:  prefix = undefined
[08-May-2011 20:53:26] NOTICE:  user = www-data
[08-May-2011 20:53:26] NOTICE:  group = www-data
[08-May-2011 20:53:26] NOTICE:  chroot = undefined
[08-May-2011 20:53:26] NOTICE:  chdir = undefined
[08-May-2011 20:53:26] NOTICE:  listen = /var/run/php-fpm.sock
[08-May-2011 20:53:26] NOTICE:  listen.backlog = -1
[08-May-2011 20:53:26] NOTICE:  listen.owner = undefined
[08-May-2011 20:53:26] NOTICE:  listen.group = undefined
[08-May-2011 20:53:26] NOTICE:  listen.mode = undefined
[08-May-2011 20:53:26] NOTICE:  listen.allowed_clients = undefined
[08-May-2011 20:53:26] NOTICE:  pm = static
[08-May-2011 20:53:26] NOTICE:  pm.max_children = 1
[08-May-2011 20:53:26] NOTICE:  pm.max_requests = 0
[08-May-2011 20:53:26] NOTICE:  pm.start_servers = 0
[08-May-2011 20:53:26] NOTICE:  pm.min_spare_servers = 0
[08-May-2011 20:53:26] NOTICE:  pm.max_spare_servers = 0
[08-May-2011 20:53:26] NOTICE:  pm.status_path = undefined
[08-May-2011 20:53:26] NOTICE:  ping.path = undefined
[08-May-2011 20:53:26] NOTICE:  ping.response = undefined
[08-May-2011 20:53:26] NOTICE:  catch_workers_output = no
[08-May-2011 20:53:26] NOTICE:  request_terminate_timeout = 0s
[08-May-2011 20:53:26] NOTICE:  request_slowlog_timeout = 0s
[08-May-2011 20:53:26] NOTICE:  slowlog = undefined
[08-May-2011 20:53:26] NOTICE:  rlimit_files = 0
[08-May-2011 20:53:26] NOTICE:  rlimit_core = 0
[08-May-2011 20:53:26] NOTICE:
[08-May-2011 20:53:26] NOTICE: configuration file /etc/php-fpm/php-fpm.conf test is successful

This stuff is not documented properly. I discovered it by having a nice afternoon at work, reading the PHP-FPM sources. That could save me some hours of debugging the internal state of PHP-FPM by other means. Maybe, for the 1st time, saying “undocumented feature” doesn’t sound like marketing crap implying “undiscovered bug”.

You may use listen.backlog = -1 for the system to decide, or you may use your own limit. -1 is a valid value as the listen(3) man page says. I am planning for opening a new issue as -1 is a more appropriate default for Linux as well. However, please keep in mind that a high backlog value may be truncated by the Linux kernel. For example, under Ubuntu Server this limit is … 128. The same manual page for listen(3) states that the maximum value for the backlog option is the SOMAXCONN value. While reading the Linux kernel sources is not exactly toilet reading, I could find the exact implementation of the listen syscall (net/socket.c):

/*
 *      Perform a listen. Basically, we allow the protocol to do anything
 *      necessary for a listen, and if that works, we mark the socket as
 *      ready for listening.
 */
 
SYSCALL_DEFINE2(listen, int, fd, int, backlog)
{
        struct socket *sock;
        int err, fput_needed;
        int somaxconn;
 
        sock = sockfd_lookup_light(fd, &err, &fput_needed);
        if (sock) {
                somaxconn = sock_net(sock->sk)->core.sysctl_somaxconn;
                if ((unsigned)backlog > somaxconn)
                        backlog = somaxconn;
 
                err = security_socket_listen(sock, backlog);
                if (!err)
                        err = sock->ops->listen(sock, backlog);
 
                fput_light(sock->file, fput_needed);
        }
        return err;
}

In plain English: the backlog value can not be higher than net.core.somaxconn. In order to be able to queue more idle connections into the kernel backlog, you ought to inrease the SOMAXCONN value:

root@localhost~# sysctl net.core.somaxconn=1024

The sysctl utility however modifies this value till the system is rebooted. In order to make it persistent, you have to define it as a new file into /etc/sysctl.d/. Or at least, using sysctl.d is recommended as it keeps the configuration to be more structured. I used /etc/sysctl.d/10-unix.conf:

net.core.somaxconn=1024

for having 1024 queued connections per listening UDS + the number of active connections that equals the size of the process pool. Remember that you need to restart the PHP-FPM daemon for the new backlog setting to be enabled. You may increase the limit as the usage model seems fit. Since nginx doesn’t queue any FastCGI connections, you need to be very careful about this setting. All the requests go straight to the kernel backlog. If there’s no more room for new connections, a 502 response is returned to the client. I can safely assume that you would like to avoid this.

Another thing that you should take care of for the number of idle connections to the PHP upstream is the fact that nginx opens a file descriptor for each UDS connection. If you increase too much the SOMAXCONN limit without increasing the number of allowed file descriptors per process, you will run into 502 errors as well. By default, a process may open up to 1024 file descriptors. Usually I increase this limit by adding a ulimit -n $fd_value to the init script of a certain service instead of increasing this limit as system wide.

You may want to buffer the FastCGI response in nginx as well. Buffering the response doesn’t tie the upstream PHP process for longer than needed. As nginx properly does the spoon-feeding to slow clients, the system is free to process more requests from the queue. fastcgi_buffer_size and fastcgi_buffers are the couple of options that you need to tune in order to fit your application usage mode.

Update (Aug 24, 2011):

Increasing the SO_SNDBUF also helps. Writes to the socket won’t block as it would be the kernel’s job to stream the data to the clients. For a server with enough memory, nginx could be free to do something else. The socket(7) man page comes to the rescue in order to demystify the SO_SNDBUF concept. Basically net.core.wmem_max is the one to blame when writes to the socket are blocking. By default the net.core.wmem_max is 128k which is very small for a busy server. If the server has a fat network pipe available, then you can get some more hints here: Linux Tuning. It may not be the case for most EC2 scenarios where the networking is shared. Therefore smaller buffers will do just fine. But it may be the case if you’re playing like me with toys that have dual 1G network interfaces.

Apache (+mod_php5) knows the difference between a script and a PATH_INFO input request that ends in .php. It would be ridiculous not to do so since the PHP runtime is part of the webserver itself. Didn’t bother to try various Apache + mod_fcgid configurations since most of the time Apache simply wastes my own time. lighttpd binds the FastCGI proxying to the file extension. This is the part where nginx fails: it tries to use a one-size-fits-all configuration logic that actually doesn’t fits all the usage modes. The FastCGI pass is done into a location (not file / file extension!) directive that doesn’t tell anything about the nature of the input. Let’s take a look at this example:

/directory.php/file.php/pathinfo.php

Except nginx, both Apache and lighttpd won’t create a mess out of a path like this. A location directive is a little big vague and no amount of regex will ever fix this. Of course, you can fix it for every single damn virtual host, but then again, can you spell boilerplate? Having different configurations for every virtual host, when clearly for other web servers this isn’t a bundled “feature”, is not fun from the system administration point of view. Usually I generate the virtual host configuration from a bash script I wrote myself. I have configuration templates for all of the applications I administer, thus is all about flags and options, unlike manually writing configuration files. Working around nginx’s inabilities to tell which stuff is which could only mean that I have to write a whole bunch of configuration boilerplate for each type of application. Doesn’t sound like one-size-fits-all anymore. Is that fun? Let me rephrase that: that’s not fun. In case AWS disappears into a black hole, I can recreate everything from scratch in a matter of minutes onto a complete new hosting service while changing Cotendo origins is a child’s play.

Here are a bunch of proposed solutions for something that can turn out into a remote exploit. I’ve being using for quite a while the same solution as provided by one of the people commenting the article:

if (-f $request_filename)
{
    fastcgi_pass php_upstream;
}

Usually because this is more readable than try_files. I usually tend to understand code blocks better.

Of course, a proper PHP script won’t save any uploaded junk to a public accessible location, but what sysadmin trust his coders anyway? I usually don’t. That doesn’t mean that they don’t do a good job, but mistakes happen. I can’t made every living thing to be as paranoid about security as I am. This exploitable situation happens when people validate their upload via the $_FILES array. I have news flash for you: the MIME type defined by the $_FILES array is defined by the browser. The browser does a lousy job at providing a proper MIME. It matches a specific MIME based onto the file extension. PHP file with JPEG extension, anyone? fileinfo would be the proper alternative. PHP should deal with this junk by design, but that’s a whole other joke about the design of PHP.

Getting back to PATH_INFO. The juicy part is that basically you can extract the PATH_INFO from an input path by using fastcgi_split_path_info, but that directive uses … regex. Which brings us to the above statement: no amount of regex will ever fix this crap. Let’s take a look at $request_filename by throwing a custom debug logging configuration that places some stuff into the access_log. Guess what, the $request_filename for the above example is … /directory.php/file.php/pathinfo.php while it’s pretty clear that the actual request filename is /directory.php/file.php. Which is the other broken-by-design thing that nginx features. Q: What damn server side variable would ever lie to your face that a $request_filename is not exactly a file? A: duh!

This doesn’t mean that you should throw nginx and PHP-FPM away and go back crying to Apache. Just simply avoid the PATH_INFO junk. However, even by using my proposed configuration directive aka check if $request_filename is actually a file before doing the FastCGI pass, you can still use fastcgi_split_path_info for a limited amount of work. fastcgi_split_path_info can replace the need for doing URL rewrites by simply using:

if (!-e $request_filename)
{
    rewrite ^ /index.php last;
}

This works for a lot of stull like WordPress, Drupal, or Zend Framework. It works for certain stuff, except the stuff containing .php somewhere. I might want to use /%postname%.php as permalink structure in WordPress. Guess what … with a properly configured nginx (cough!) + the above rewrite rule replacement simply can’t. You have to go back to:

if (!-e $request_filename)
{
    rewrite ^/(.*)$ /index.php?q=$1 last;
}

Which is exactly what I did. For all my apps. Happens to be more deterministic by nature, while I tend to sleep better when I can predict the request pipeline, no matter which is the input junk.

I guess I could give Hiawatha a run which seems to be lightweight enough to support PHP with a threaded architecture. PHP is process based, doing blocking I/O by design, therefore the web server is rarely the actual bottleneck.

For quite a while I’ve being using my own implementation of phpass. There are a couple of reasons for doing my own homework instead of using the already existing solution:

• The Suhosin patch doesn’t play nice with PHP extensions available as binary blob only as it breaks the ABI. Therefore a consistent bcrypt implementation lacked before PHP 5.3 since at most I could use just the Suhosin extension.
• The random seed generator was quite broken across different platforms. Unices have /dev/random and /dev/urandom. Even more, these devices are not consistently implemented across them. If you aren’t inspired (/dev/random under Linux is a blocking device), then your random seed generator is screwed till the system gains enough entropy. Under Windows, the used PRNG was something really crappy, if you could use the system implementation, otherwise people used to hack something together. phpass is still a mess, while being backwards compatible with a lot of PHP implementations.

openssl_random_pseudo_bytes() is still slow under Windows, although since PHP 5.3.4 it fixes the blocking behavior. But it’s still better than the previous mess. My method for generating the bcrypt salts is still about 5.8X faster than the method used by phpass under Ubuntu.

I am not saying that my implementation is the implementation, but I got the feeling that for once I am doing it better that I used to do it before, or doing it better than phpass does.

Grab/fork it from GitHub if you’re curious.

Update: I also implemented Ulrich Drepper’s SHA2-based crypt() scheme, but the v0.2 is incompatible with v0.1 due to the fact that it returns the hashes as base64 string. This is a side effect of using 16 bytes of random seed for the SHA2 salts. The PHP manual simply states 16 chars, but that’s a little bit vague. Except chr(0) and chr(36), any other single byte char is valid for the random seed. bcrypt takes a more cleaner approach for the 16 bytes random seed by using a 22 chars base64-like string that encodes those 16 bytes.

Few months ago I had an interesting task: place something in front of Amazon S3 in order to cut down the file distributions costs. While S3 is great for storage, for the actual distribution you’d see that the traffic counter is spinning fast and the USD counter even faster. Certain web hosting providers have good deals for dedicated servers, therefore something like the S3 traffic could be turned on into a fixed cost, given by the amount of machines and the bandwidth pool specified into the contract.

This is the part where the technology kicks in. As I said earlier, the hardware rarely is the limit. Remember the C10K problem? Some servers deal with with this stuff better than others. This means that I wouldn’t pick Apache, or any other web server that deals with the connections by the same model. The network has a high latency compared to the rest of the hardware, even the local disk. People should stop thinking in blocking mode.

For the above task, nginx did the load balancing, fail-over, and URL protection part really well. squid did the other part aka the proxy cache job. Now you may wonder why the good old squid since I think nginx doesn’t need introduction.

I tried Varnish as the proxy cache, but the people behind it thought it was funny not to have an unbuffered transfer mode. This might work well sometime, with the “most of the web traffic”, as they say, but when you try to push a large file over HTTP this point is somewhat the real world usage for our case:

HAI, I ARE TEH CLIENT. PLZ GIVE ME /large-file.zip
LOL, OK, BUT WAI TILL HAZ IT IN ME CACHE

I considered trying Varnish for all the hype surrounding it. I kept myself for going straight to squid because of so called StackOverflow type of specialist that doesn’t recommend it since Varnish is “way faster”. Got a little news flash for you brother: it doesn’t matter. The fact that a client needs to wait till the proxy cache buffers all the stuff from the origin server (S3) won’t count as “speed”. It may work great for CDN-like type of traffic where certain patterns emerge, but not here where we have to deal even with hundred of megabytes for a single object.

The next bus stop was Apache Traffic Server. While it looks good on paper, it required rocket scientists to configure a simple usage mode. While I like the pictures with the baseball bat explaining the concept of a cache hit or miss, it still lacks proper, understandable documentation for a new comer in order to get something up and running in a timely manner. I am not a sysadmin newb, but some things are harder to pick up from scratch than others. Work against a short deadline in order to get my point. My squid configuration file has exactly 18 lines. All human readable and easy to understand. The default Traffic Server installation had as much as configuration files. How am I supposed to figure out this mess?

Things I also considered: nginx – it is dreadful as proxy cache due to the lack of HTTP 1.1 for the proxy backend, therefore no If-Modify-Since or If-None-Match support for invalidating a cached object. Amazon S3 sends both Last-Modified and ETag. A half-baked product from this side of the story. We have HTTP 1.1 and the 304 status. Igor, are you there?

lighttpd – also annoyed me with some proxy cache limitations that didn’t play nice. Don’t get me wrong, excluding some security record, lighttpd is actually a great web server, performance wise. It pushed our file distribution traffic for almost 4 years from a couple of FreeBSD machines. We changed the hardware every 2 years though. I never saw those machines going over 200MiB of RAM even at “rush hour”, while the CPU used to sip power from the socket instead of going like crazy. I didn’t even install a monitoring system since there was nothing to alert. The hardware load balancer did the fail-over stuff back then. Believe me or not, I actually felt sorry for turning off a web server with almost 600 days of uptime, after moving to EC2. But I still hate to use lighttpd as proxy cache.

I thought about Apache + mod_proxy for 3.14 seconds, but I moved on. Even with Event MPM it falls behind the competition. While I have a great deal of respect for the Apache Foundation, I still don’t get how they manage to screw things up with the Apache HTTP Server. I mean, come on, I expect more from the so called market share leader.

I gave a thought to node.js which I think is a terrific piece of technology, if you can ignore some of the silliness of JavaScript. But I wanted a piece of software that’s maintain towards my need by a bunch of people that do this stuff for quite a while, not a home brewed proxy cache.

Which brought me down to squid. A lot of “expert opinions” kept me from picking it in the first place, sparing me the pain of installing, configuring, and testing a bunch of alternatives, even though I’ve being using it as forward proxy for years. It does everything that it needs to do for a proxy cache with more or less configuration. It is an asynchronous server. It isn’t as slow as advertised, even though it’s written into that “slow, modern language” called C++. The configuration is straight forward. In fact, since the cluster is in production, it didn’t went even near the hardware limits.

Something bad happened into the first couple of days due to lack of proper production traffic that didn’t match the theoretical testing: clients who send invalidation headers along with ranged requests, which squid doesn’t handle very well. This sent the whole cluster into a vicious circle. The solution was simple at the nginx layer: strip down all the headers, pass just the relevant stuff. nginx does the header management way much better than squid that tries to implement some sort of “standards compliant” stuff that makes it very stiff. But even then, even though the cluster was working like crazy, the CPU didn’t even come close to 20% while the RAM was mostly doing nothing. The network pipe on the other hand, was working at full capacity, therefore some high I/O waits started to build up. Same goes for nginx.

Therefore no, I won’t do the same mistake twice. I stopped doing benchmarks. I don’t care about faster, but just about fast enough and well enough engineered. Business wise, we don’t pay less for using less CPU or RAM. Or even for drawing less power from the socket. Even Amazon doesn’t do that with it’s pay-as-you-go-only-what-you-use model. We have a bunch of dedicated hardware that we can use at 1% or at 100% load. There’s a bandwidth pool that’s the actual interest in saving money. I encourage fellow sysadmins do the same: stop caring about irrelevant stuff like: this web server is 10% faster. Even if it’s 200% faster (made up number), if the network pipe is the limit, it doesn’t matter overall. That stuff matters into the application server market where you have to shuffle data around, instead of taking a bunch of bytes of the disk and push them over the network interface. Today’s hardware is stupid fast. When 10 Gbps is going to be a standard, then yes, some of the great debates may actually have some ground. Currently they don’t. People like to argue about anything. Even toiler paper positioning. I fail to see where’s the productive part.

Don’t get me wrong. I hate bloat-ware. I love the raw speed. I love the machines that return a fast answer. But sometimes good enough is enough. Don’t keep “the religion” standing in your way of properly doing your job. People should focus on finding solutions instead of fighting religious wars with colorful graphs.

For example, serving 55.000 requests per second for a 2 KiB object over the loopback interface requires higher bandwidth than a 1 Gbps network pipe can provide. 2 KiB is a very small file. Most JavaScript libs, and style-sheets go over that limit, therefore the bandwidth limit is easier to hit. I don’t have to mention the images or plain large objects. Yes, some tests may be great for the ego of a developer behind a certain product. But most of the time some sort of apple to oranges comparison is involved. Certain features cost CPU time. But most of the time, the lack of specific features costs us time. The time is money. The CPU time, in this case, isn’t.

PS: we’re caching hundreds of GiB. We’re serving at least one TiB of data per day. Not bad for an old, slow, proxy cache running on a couple of machines that nobody wants to use since we have this new cool technology. Do we?

The stuff that works:

• the plug-in core, end-to-end. This means that I can add / remove functionality without wonder how to fix stuff under the hood. I made my life easier for me or for any hacker that wants to add functionality. XHTML Video Embed uses complicated boilerplate in order to add a new service or edit some core functionality. XVE doesn’t. This means that everything, even the user interfaces, scale by iterating over configuration array. I can add a new service or media player in minutes. Basically I just need to figure out its syntax. I don’t have to figure out the plug-in’s syntax. I know, I shouldn’t release the old plug-in as such in the first place, but I didn’t have the current experience while it grew from 50 to over 1000 lines of code. XVE was designed from scratch.
• All the currently supported media players for the flv/mp4 content. It supports: Flowplayer, the old FLV Player, and JW Player. I resisted the temptation to use a switch(), therefore I can add any flv player that supports a couple of input options: path to flv/mp4, and optionally a path to a splash image. This feature was requested for quite a while. Now it works seamlessly for all the supported players.
• The permissions are more granular. The options are more granular as well. Anything can be controlled from the tag, besides the object size.
• new-style embedding tags. This means: video for all the supported video services such as YouTube, Google Video, or College Humor, etc, audio for Trilulilu, and 220, image for Trilulilu. While the legacy support (youtube, google-video, collegehumor, etc. tags) is implemented, it is disabled by default. You can enable it, but …

The stuff that doesn’t work, yet:

• … but the admin UI doesn’t do anything, yet. Therefore you can’t enable the legacy support. Since my jQuery FU is not strong, I still have to learn how to do edit in place and other stuff like this that allows you to use your custom settings. Please remember that usually I write backend code. Not so much an user interface guy.

The stuff that’s changed:

• the attributes w and h won’t work. The tags support just a single attribute: attrib. attrib=”width=680&height=440″ – longer, but more powerful aka for flv I also have img=http://example.com/path/to/splash/image.png, and player=jwplayer if you want to override the default media player (Flowplayer as of this version). The value of attrib is sent to PHP’s parse_str(), therefore you need to be careful about this.
• read the CHANGELOG.TXT for all the other geeky details.

Download: XVE 1.0 Beta 3.

Basically it’s just a bunch of bash scripts that automates the burden of moving a SVN server from an old machine to a new machine. It wasn’t designed for large dumps though, but if requested, I can add the functionality, or you can send a pull request .

You can grab it from GitHub: svn-migrate.

Given the fact that recently Firefox and Google Chrome got an upgrade under Ubuntu, I couldn’t resist the temptation to run a bunch of benchmarks. Since Opera also got an upgrade almost a couple of months ago, it was a fair fight. However, I didn’t run any CSS benchmarks since my interest is mainly into the JavaScript area. I used: V8 Benchmark Suite v6, Kraken JavaScript Benchmark v1.0, and SunSpider Javascript Benchmark v0.9.1.

V8 Benchmark Suite

The V8 Benchmark suite is Google’s tool for load testing the JavaScript performance. While it was expected that Google Chrome would take the crown of this test, I didn’t expect Opera falling behind by such noticeable margin. Below, the higher results are better.

Kraken JavaScript Benchmark

This is Mozilla’s answer to the SunSpider JavaScript Benchmark. While they do somehow look alike, their results are very different. This one really took me by surprise. It is a pretty heavy benchmark, taking way much more time than SunSpider. It was expected that Firefox would take this one since it plays onto the home field, but I didn’t expect such decisive victory. While Opera also falls behind for almost 50% of the Firefox score, this test looks more like an anti-Chrome benchmark. Ouch Google, that hurt. The results are in milliseconds, lower is better.

SunSpider JavaScript Benchmark

SunSpider is almost a classic in this field. While being created by the WebKit guys, Google Chrome also plays onto the home field since it is a WebKit based browser, but with a different JavaScript engine. While it was expected to see Firefox coming near Google Chrome, I didn’t expect it overtaking Chrome. Actually, at work (different hardware configuration, Core 2 Quad Q9400 + 8 GiB RAM, same OS) Chrome was slight faster. Well, I can say that this falls into the negligible area. However, I didn’t expect “the fastest browser on the planet” falling behind these two by a noticeable margin. The results are in milliseconds, lower is better.

Configuration: OS, Hardware, Software

I ran the tests under Ubuntu Lucid (10.04.02 LTS, amd64) with kernel 2.6.32-30-generic. The OS was upgraded before the tests. All the browsers ran their latest published versions into the (semi-)official repositores, without any add-ons, except Firefox that had Ubuntu Firefox Modifications. That means: Firefox 4.0, Google Chrome 10.0.648.151, and Opera 11.01. While Firefox 3.6.15 ships with Ubuntu Lucid, Mozilla provides a semi-official Firefox 4.0 PPA.

My hardware is a pretty modest Athlon II X2 240e @ 2.8GHz + 4GiB of RAM.

Conclusions

While I switched to Google Chrome since Google started to provide stable releases and public repositories for Debian & Friends, going with Firefox from 0.8 to 3.6 is not an easy to forget experience. I tested every Firefox release since 4.0 Beta 1 and all I can say I’m glad the browser is finally back into the top area of the charts. Given the fact that Chrome still lacks certain features that I still like to see, while Chrome doesn’t come anywhere near a decent memory management given the appropriate number of extensions and tabs, I might give Firefox another change. Nice come back Mozilla.

Here are the screenshots that I took after the benchmarks: download.