three stacks of high society

EC2 Ephemeral Disks vs EBS Volumes in RAID

Victor — Sat, 02 Jan 2010 06:55:33 +0000

EC2 Disk Overview

Amazon's EC2 service is really neat, but its disk subsystem has some peculiarities that are not initially obvious. Up until very recently, root directories ('/') at EC2 were limited to 10Gb, a limit defined by the maximum size of an Amazon Machine Image (AMI), essentially a template of an EC2 instance. In order to use more disk space, Amazon provides ephemeral disks that one can format and mount anywhere on the file system. However, in order to get persistent storage, one has to use network-attached EBS volumes, a sort of limitless in capacity but bound in I/O wonder of Amazon architecture. There are clear performance implications in choosing how to configure an EC2 instance's disk subsystem, so I recently benchmarked some various ephemeral and EBS RAID configurations.

Ephemeral disks

Pros:

Free (included in cost of EC2 instance)
Stable, predictable performance on par with a standard physical hard disk
Abundant storage (up to 1.7TB on a c1.xlarge)

Cons:

Ephemeral - if the instance shuts down, all data is lost
Average random seek performance (6-7ms seek times per spindle)

EBS Volumes

Pros:

"Highly available" - AWS claims to provide redundancy and a lower failure rate than physical disks
Portable - an EBS volume can be connected to any instance in a single availability zone
Backups - can easily create snapshots

Cons:

Extremely variable performance - seek times can range from .5ms to 10ms+
Maximum throughput of 1Gbit/s
Costs associated with storage and I/O

Testing

For this testing, c1.xlarge instances were used due to their high CPU performance, memory capacity, "I/O Performance: High" (according to Amazon), and 4 available 450GB ephemeral disks.

I created 5 c1.xlarge instances with 5 configurations: 4xEphemeral RAID0 local disk, single EBS, 2xEBS RAID0, 4xEBS RAID0, 8xEBS RAID0. All instances were created in the us-east-1b Availability Zone and all EBS volumes attached were newly created specifically for this test. Testing was done using bonnie++ on fast mode (-f flag, skips per-char tests).

mdraid was used to create RAID0 arrays with a chunk size of 256k, for example:

mdadm --create --verbose /dev/md0 --level=0 -c256 --raid-devices=2 /dev/sdi1 /dev/sdi2

blockdev is used to set the read-ahead buffer to 64k:

blockdev --setra 65536 /dev/md0

XFS is used as the filesystem:

mkfs.xfs -f /dev/md0

Finally the RAID array is mounted with noatime at /mnt/md0:

mkdir -p /mnt/md0 && mount -o noatime /dev/md0 /mnt/md0

I logged the results of Sequential Writes, Sequential Reads, and Random Seeks. bonnie++ was run 6 times on each instance.

bonnie++ results

bonnie++ averages

Instance	Write	Read
4xEphemeral RAID0	231680	176739
1xEBS	37827	48991
2xEBS RAID0	80578	75966
4xEBS RAID0	98836	89752
8xEBS RAID0	69921	93175

Sequential Throughput

Four ephemeral disks in a RAID0 configuration has extremely high throughput and an acceptable random seek performance. The ephemeral array results are almost a 4x scale of the same test of my desktop's 7200RPM desktop drive, which is what one would expect out of a RAID0 array of physical hard disks.

The EBS results are a little less predictable. A single EBS does not have the throughput of a single ephemeral drive. The 2xEBS RAID0 shows almost twice the throughput of the single EBS volume, while the 4xEBS RAID0 and 8xEBS RAID0 instances do not scale much higher than the 2xEBS RAID0 instance for throughput. Since EBS volumes are access via network, this indicates that EBS volume throughput is limited by the gigabit interface.

Random Seek Times

Instance	Total Ran. Seek/s	Per Volume
4xEphemeral RAID0	658	165
1xEBS	250	250
2xEBS RAID0	396	198
4xEBS RAID0	5138	1285
8xEBS RAID0	2554	319

Random Seeks

The ephemeral array does about 165 random seeks per second, which is comparable to a desktop hard disk.

EBS random seek performance, however, is not easily predictable. The volumes that make up the 4xEBS RAID0 instance clearly are higher performing than those of the other instances. Is EBS performance more of a property of the EBS volumes or the instance?

Another interesting result I noticed (but didn't include in these graphs) is the deviation of performance from one run to another. The standard deviation between the runs was much smaller for the ephemeral drives than for the EBS volumes.

Swapping EBS volumes to identify bottleneck

I attached the two EBS volumes from the poorly performing 2xEBS RAID0 instance to the fast 4xEBS RAID0 instance and re-ran the tests. If the performance of the two EBS volumes improves when attached to the 4xEBS RAID0 instance, then perhaps we can attribute the difference to the instances, but if the performance is the same, then we can blame the EBS volumes themselves.

Results:

Configuration	Seq W/s	Seq R/s	Ran. Seeks	Seeks/EBS
2xEBS Volumes on 4xEBS Instance	110146	91555	795.6	397.8

The I/O channel is more or less saturated, but we still see the same poor random seek performance that the 2xEBS RAID0 instance exhibited with these two same EBS volumes. This leads me to believe that the seek times are inherent to the individual EBS volumes themselves.

To confirm, I mounted the high performance volumes from the 4xEBS RAID0 instance and the poorly performing volumes from the 2xEBS RAID0 instance to the 8xEBS RAID0 instance. I wanted to test if we can "export" the high the performance from the 4xEBS RAID0 instance to the 8xEBS RAID0 instance. I then repeated the bonnie++ tests.

Results:

Configuration	Seq W/s	Seq R/s	Ran. Seeks	Seeks/EBS
8xEBS RAID0 (benchmark for instance)	39238	90403	1629	204
2xEBS Volumes on 8xEBS Instance	108108	94189	735.3	368
4xEBS Volumes on 8xEBS Instance	125459	93972	9285	2321

Once again, the 2xEBS volumes are still poorly performing and the previously fast 4xEBS volumes are still fast (they were even faster than before). At this point, the evidence is pretty clear that the performance of the EBS volumes are inherent to the volume itself, since they exhibit the same level of performance regardless of the EC2 instance that mounts them.

Instantaneous EBS performance over time

I wanted to test if EBS performance varies over time, so I created a new c1.xlarge instance, in another EC2 availability zone. This new c1.xlarge instance had 4 new EBS volumes, configured as a 2xEBS RAID0 array (two EBS volumes unused) and a 4xEBS RAID0 array (all four used). I ran bonnie++ over two weeks.

Results:

Configuration	Seq W/s	Seq R/s	Ran. Seeks	Seeks/EBS
2xEBS RAID0	107513	92681	2642	1321	(week 1)
4xEBS RAID0	112326	94844	7829	1957	(week 1)
2xEBS RAID0	35799	68619	215	108	(week 2)
4xEBS RAID0	88012	92863	623	156	(week 2)

The same instance using the 4 same EBS volumes show a huge discrepancy in performance from week to week. During the first week, the 4 EBS volumes perform admirably. During the second week however, performance drops dramatically and the 4xEBS RAID0 volumes don't even seem to saturate the gigabit channel. This doesn't bode well for EBS performance predictability.

During the second week, I ran 'iostat -x -m 240' while I ran bonnie++ to see if I could identify the poor performance.

2xEBS RAID0:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.02    0.00    0.47   11.99    0.02   87.51
 
Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
md0               0.00     0.00  0.00 615.20     0.00    25.42    84.63     0.00    0.00   0.00   0.00
sdi1              0.00     0.06  0.00 307.41     0.00    12.71    84.67   148.55  483.20   3.25 100.02
sdi2              0.00     0.04  0.00 307.60     0.00    12.71    84.61    10.49   34.10   1.19  36.67
sdi3              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdi4              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

Notice the particularly poor 'await' time of /dev/sdi1, one of the two members of the /dev/md0 RAID0 array. Because of how poorly /dev/sdi1 was performing, the entire /dev/md0 array exhibited poor performance. During a read request, the file file system sends a request to the /dev/md0 device, and mdadm determines that it needs to split the request into two separate requests, one that goes to /dev/sdi1 and /dev/sdi2. If one of the two members of the array is poorly performing, as in this situation, it becomes a bottleneck for the entire array. (As an aside, a few hours after seeing these poor numbers, I re-ran bonnie++ on these 4 EBS volumes, and they were once again fast.)

Conclusions

These tests show a tremendous variability in EBS performance, not just between one EBS volume and another but also within a single EBS, from one point in time to another. Like CPU time on a shared server, an EBS volume's performance is dependent on how busy the entire EC2 ecosystem is. However, AWS guarantees a certain number of CPU cycles and amount of RAM in an EC2 instance; it's not clear that AWS provides a similar guarantee for EBS performance.

There are a few key takeways:

EBS volumes, even several in RAID0, have a throughput limit of 1 gigabit/second on an EC2 instance. Two EBS volumes in RAID0 have the ability to almost max out an instance's EBS I/O channel
EBS volumes can have random seek times ranging from .5x to 10x+ that of an ephemeral disk
Ephemeral disks in RAID0 are very fast
It is advisable to use both a mixture of ephemeral and EBS RAID arrays, depending on the situation
4 or 8 EBS volumes in RAID0 seem to be enough to smooth out the peaks and valleys of individual EBS volumes' seek times
Increasing mdadm RAID0 chunk size to 256k and setting 'blockdev --setra 65536' seems to help sustain I/O throughput even when seek times are bad. Mounting EBS arrays with '-o noatime' should also theoretically help with poor seek performance.

You should also really read Joe Stump's good writeup and Heroku's Getting Good IO from Amazon's EBS.

From CET to CDT

Victor — Tue, 29 Jul 2008 05:47:00 +0000

Memory is really strange. On the one hand, I'm amazed at how fast the last three years of my life has gone by. I remember walking into my hotel room on the day I landed in Stuttgart, a full month before Rebecca would come, and putting down my bags and really wondering what I had gotten myself into. I had decided to take a job in a city I had visited only during my interviews, in a country I had only spent a few days in as a tourist, and here I am sitting down in my hotel room, in need of a shower, exhausted from jet-lag, only then grasping that I had committed myself and Rebecca to living in Stuttgart for at least three years. At that moment I could only hope that we had made the right choice to come. It turned out to be one of the best decisions of our lives.

Remembering specific events, though, and time doesn't seem to move so quickly. I think about my first weekend in Germany, when a colleague invited me to an Onion Festival in the medieval town of Esslingen, and it seems appropriately placed about three years ago. Then I remember when our friend Laurel visited, our first visitor, I think, and how we went out to a besenwirtschaft (a uniquely south-west Germany gem, in which vineyard-owning families sell their own wine out of their living rooms) and got extremely intoxicated with a super friendly German couple. We ended up getting invited to their home for a few more bottles of wine, and Rebecca got sick in their bathroom just as our taxi pulled up. I remember all of the festivals - the Hamburg Fischmarkt, Karnival, the Weindorf, and of course the Bierfests (Germans love to find a reason, any reason, to have a festival). I remember all our visitors - our families and lots of friends from home - who took advantage of us living in Stuttgart and allowed us to share our newly found love of Germany with them. I think fondly of all the trips we took - the Turin Winter Olympics, Sardinia, the Lake District, Poland, and so many more. The more memories I conjure up and place into a mental timeline, the more it seems like it really has been three, full, years since I stepped into my room at the Millennium Hotel, and I'm both at once happy for the experience and sad that I can no longer call Stuttgart home, even if it means I no longer have to walk up 6 flights of stairs to be home.

We hope to go 2 for 2 on picking cities randomly and moving without any prior connection, and so far Austin has really been a great place. Many great things about German culture are embraced in Austin - love for the outdoors and festivals being the two most obvious. There are even biergartens, and the town of Fredricksburg, located in the center of Texas wine country (another huge similarity to Stuttgart!), was founded by Germans, and I think the German influence on the local culture shows. There's even a local waterpark called the Schlitterbahn.

I think we're off to a good start.

FOSDEM over; Crisis averted

Victor — Sun, 02 Mar 2008 14:03:18 +0000

I went to Brussels last weekend for FOSDEM 2008, which was held at ULB Campus Solbosh. The free event was a good way to check in with the overall Open Source community and to see all of the interesting things people outside my normal circles are working on.

Friday Night Beer Event
Things got off to an memorable start on Friday night. I timed my arrival so that I could attend the Friday night "Pink Elephant" beer event held at the Delirium Cafe. I met up with a colleague, and we had a few good beers while chatting with other FOSDEM attendees. Lots of people had their gadgets out for others to play with. I got to play with a EeePC and a Nokia 810 while my iPhone was passed around. I even picked up the presence of a OLPC OX-1 over wifi, but was never actually able to find it.

After a few hours of drinking beer and talking about software, we met up with a few more friends to go to dinner at an underwhelming yet wistfully overpriced restaurant in the middle of the tourist trap. I had another beer or two over dinner, and so when we left the restaurant, I was a little toasted.

For some reason (playing with my phone?) I was straggling behind as we walked out when these two guys sidled up to me and started dancing, ~~singing~~ yelling, and doing some weird line dance kick between my legs. In my drunken state, I was a bit confused but thought they were just drunk too and danced along. After a few moments of this silliness, they walked off. I luckily had a moment of clarity and thought it best to check my pockets. Wait, my wallet is missing. Yup, it really is still missing. The two guys hadn't taken more than 20 steps down the street, so I ran up to the nearest one, forcefully grabbed his shoulder, and demanded, "Give me back my wallet." He looked a bit surprised and immediately pointed to his accomplice. I turned to him and without a word, he reached into his coat pocket and handed over my wallet. I took it from his hands, and strangely enough, we just parted ways. The entire episode lasted probably 30 seconds or so, and my friends, who were only a few steps ahead, missed it all.

Talks
The next morning I was a bit slow getting up and got to FOSDEM about an hour late, missing the opening keynote (it didn't help that I stayed up for a few more hours playing poker with the hotel staffer and his friends, but that's another blog post). I pretty much spent Saturday in the Janson auditorium listening to the big talks - "How a large scale opensource project works" with Robert Watson, "Perl 6" with Patrick Michaud, and "Unicoding with PHP 6" with Andrei Zmievski. I also squeezed in some quick 15-minute "lightning" talks about smaller open source projects like Alfresco, OpenAFS, and Squeak.

I was even slower getting up on Sunday morning* and missed the Drupal opening talks by Dries. I did catch Kris Buytaert's "Drupal and MySQL High Availability", which was quite good. In addition, I took the opportunity to see a talk on CakePHP and Mozilla's upcoming Prism.

Thoughts
My colleagues in attendance weren't too enthusiastic about this year's FOSDEM. Their main complaint was that it has become a little too commercialized with seemingly marketing-oriented talks, rather than more in-depth code talks. While I can understand this sentiment, I think the problem is mainly with their expectations of FOSDEM. FOSDEM should be a venue for projects to open up to people outside of their core community. A code-driven, detailed talk about the intricacies of the Form API in Drupal 6, for example, would only be digestible by experienced members of the Drupal community, most of whom would be familiar with the FAPI in the first place. Higher-level talks allow small projects, such as Squeak and CakePHP, to attract people like me who have a passing interest and may even be pulled in enough to try the stuff out.

Some of the speakers were certainly better than others. FOSDEM (and Open Source in general) is a pretty international affair, and because the conference was conducted in English, there were varying levels of English public speaking abilities. Overall, however, I thought the speakers were quite good and spoke to the subject matters well. My only complaint is that FOSDEM seems to be outgrowing its britches. There were lots in attendance, and at times, it was a little bit difficult walking through the masses to get to the talks in time. That probably speaks to the growing popularity of OSS, which is always a good thing.

More photos from FOSDEM 2008.

*I discovered the Grand Casino Brussels on Saturday night and was there until almost 4 in the morning waiting on a seat at the Hold 'em table. Generally casinos in Europe are quite stuck up about dress code and appearances (to the point of making you rent an evening jacket), but I found Brussels casino to be very welcoming. You still won't find flip-flops and t-shirts like you would at some places in Vegas, but at least you can walk in reasonably dressed. Anyway, at 11PM I was #3 in line for a seat and only got to #1 by 3:30am before I had had enough and just left. They had two tables of €5/€10 NL Texas Hold'em, but apparently they sometimes also have €10/€20 limit as well.

Local root exploit in Linux kernel 2.6.17 to 2.6.24.1

Victor — Mon, 11 Feb 2008 00:27:19 +0000

Pretty scary stuff, even if you trust all of your users:

victor@mercury ~ $ ./exploit
-----------------------------------
 Linux vmsplice Local Root Exploit
 By qaaz
-----------------------------------
[+] mmap: 0x100000000000 .. 0x100000001000
[+] page: 0x100000000000
[+] page: 0x100000000038
[+] mmap: 0x4000 .. 0x5000
[+] page: 0x4000
[+] page: 0x4038
[+] mmap: 0x1000 .. 0x2000
[+] page: 0x1000
[+] mmap: 0x2ac3dee3c000 .. 0x2ac3dee6e000
[+] root
mercury ~ # whoami
root

What's really amazing is that news of this vulnerability didn't really hit the mainstream web until today, but yet on Friday there was already a kernel patch. There's even an in-memory hotfix that you can use (I tried that too - it works) if you prefer to wait until an official kernel makes it downstream. Open source is amazing.

Had this been proprietary software, no one would have known about it except for the all the people exploiting it. Servers all over the world would get owned, and the software company wouldn't even discover it for a few more weeks. Or worse, they would know about it, but would hope to keep it hush-hush until the next Patch Tuesday.

Migrating to Google Apps (and getting everything working)

Victor — Sat, 12 Jan 2008 20:00:11 +0000

For the last few years I've been using Gmail exclusively and have been forwarding emails to @victortrac.com to my Gmail account. Google's spam filters are the best I've ever seen, and the interface is elegant and fast, and combined with loads of storage and IMAP access, Gmail is nearly the perfect email application. The XMPP integration is just icing on the cake.

Because of these features, I voluntarily gave up having a customized email address on my personal domain to take advantage of Google's infrastructure and technology. The decision was fairly easy - I was deluged in spam and GMail's web client was better than any other thin or thick client available. By forwarding my domain's email to my Gmail account, I was letting Google's wonderful anti-spam technology work its magic. This allowed me to retain some use of my previous email address, but as I started to use XMPP (aka Jabber or as Google calls it - Google Talk) I became more and more dependent on my Gmail identity. Sure, I had other Jabber IDs, but it was just too convenient having a unified email address and Jabber ID provided by Gmail.

However, let's say that in five years Google shuts down or, more likely, another company comes along and provides a better service or product. By this time your Gmail identity has evolved into a unified presence, communications, and identification address where anyone can reach you at any time and is also your OpenID login to the majority of sites on the internet. If you've spent 10 years building this identity around a Gmail address, you're not in a great position to easily transition. By using Google Apps on a domain that you own and control, you've at least separated the address from the services and would be able to move around as you want. It's like being able to live all over the world, moving to where the grass is always greener, yet still always having a constant mailing address.

Getting it all to work

So today I registered and migrated victortrac.com to Google Apps, allowing me to use all of Google's great software on my personalized address. The registration process is really quick and simple, and the actual migration part is just a handful of DNS changes depending on what services you want to switch over to Google. For me it is just email and chat, and Google's documentation made it clear which MX servers I need to point my domain to.

For XMPP, however, the documentation isn't very complete. According to this page, you need to add the following SRV records to your DNS server (replace gmail.com with your own domain):

_xmpp-server._tcp.gmail.com. IN SRV 5 0 5269 xmpp-server.l.google.com.
_xmpp-server._tcp.gmail.com. IN SRV 20 0 5269 xmpp-server1.l.google.com.
_xmpp-server._tcp.gmail.com. IN SRV 20 0 5269 xmpp-server2.l.google.com.
_xmpp-server._tcp.gmail.com. IN SRV 20 0 5269 xmpp-server3.l.google.com.
_xmpp-server._tcp.gmail.com. IN SRV 20 0 5269 xmpp-server4.l.google.com.
_jabber._tcp.gmail.com. IN SRV 5 0 5269 xmpp-server.l.google.com.
_jabber._tcp.gmail.com. IN SRV 20 0 5269 xmpp-server1.l.google.com.
_jabber._tcp.gmail.com. IN SRV 20 0 5269 xmpp-server2.l.google.com.
_jabber._tcp.gmail.com. IN SRV 20 0 5269 xmpp-server3.l.google.com.
_jabber._tcp.gmail.com. IN SRV 20 0 5269 xmpp-server4.l.google.com.

The _xmpp-server._tcp and _jabber._tcp SRV records tell the requesting server to look at Google's XMPP servers when there's an XMPP request. There are two minor problems here:

Both _xmpp-server._tcp and _jabber._tcp records serve the same purpose (_jabber._tcp was even deprecated earlier this year)
There's no _xmpp-client._tcp record

This means that Google's example only really adds s2s functionality to the thin client built into Gmail or Google's GTalk thick client, which contradicts this help page for configuring Pidgin to work with your Google Apps domain (there's a whole thread on Google groups about people following Google's directions exactly but not being able to connect properly with Pidgin).

In order to get a third party client to connect to Google's XMPP servers, you'll have to manually configure a "Connect to server" to go directly to talk.google.com. The better solution, however, is to add another set of SRV records (again, replace gmail.com with your own domain):

_xmpp-client._tcp.gmail.com. IN SRV 5 0 5222 xmpp-server.l.google.com.
_xmpp-client._tcp.gmail.com. IN SRV 20 0 5222 xmpp-server1.l.google.com.
_xmpp-client._tcp.gmail.com. IN SRV 20 0 5222 xmpp-server2.l.google.com.
_xmpp-client._tcp.gmail.com. IN SRV 20 0 5222 xmpp-server3.l.google.com.
_xmpp-client._tcp.gmail.com. IN SRV 20 0 5222 xmpp-server4.l.google.com.

With these additional records, when XMPP clients try to log into your domain.com, your DNS server responds down the list and tells it to check on port 5222 on one of Google's servers.

OpenID next?

I'm only a few hours into my migration over to Google Apps, but I think it'll be a good fit for me. Now if only Google would roll out OpenID.... :)

Where's the "Undo" on Google Reader?

Victor — Tue, 18 Dec 2007 23:12:52 +0000

There's been a bunch of press lately about Google Reader's new features, most notably the "Friends' shared items" section and Profiles. A handful of people were instantly complaining about the lack of privacy and control, but I don't really see Google's implementation as a problem. It seems pretty simple to go to "Manage friends" and hide the contacts you don't want looking at your shared items, but this really kind of defeats the purpose of using your shared items to begin with. If anything, I think Google's decision to automatically include your GTalk contacts just makes it that much likelier that I'll read and see my contacts' shared items. I wouldn't spend the time to go through my contacts to subscribe, but having them there automatically is great. It's giving me the benefits of filtered reading list through a social network I wouldn't have bothered to use otherwise.

However, I can see this causing some annoyance in the form of repeated posts. If a lot of my contacts are subscribed to the same stuff I'm subscribed to and decide to share it, I'll see it twice - once on my own feeds and again when I go through their shared items. This has been annoying me on my Techmeme feed enough to want to consider unsubscribing from techmeme, and so I can see this becoming a bigger problem as I get more contacts who read the same stuff I read.

Whenever Google gets around to fixing the duplicate feed problem (and I really hope they do soon), they should also add an "Undo" button in Google Reader. In GMail, anytime you archive, delete, flag as spam, or otherwise move an email from one view to another, GMail gives you the option to undo the operation. This is great because Archive, Report as Spam, and Delete are all right next to each other and easily mis-clicked. Even if Undo wasn't an option, it would still be possible to manually reverse the change.

What really annoys me with Google Reader is that there is no undo option when you click on "Mark all as read." "Refresh" is stupidly directly next to "Mark all as read," so I regularly end up clicking on the wrong button. The best you can do is switch over to the "All items" view and hope that you could skim through to see what you might have missed. Maybe I should take this as a blessing so that I can get through my feeds faster.

I'm a bit baffled why the smart guys at Google haven't fixed these problems. Surely I'm not the only Google Reader annoyed by duplicate posts and the inability to undo a "Mark all as read" mis-click.

Technology is Applied Magic

Victor — Sun, 09 Dec 2007 02:18:30 +0000

A few hours ago, Rebecca and I were walking through the Schwabstrasse S-bahn stop in Stuttgart, and as we reached the escalator to go up, we felt a cold wind coming down from the street level. I was wearing only a short-sleeve polo shirt and a pair of light pants, and so we stopped to put on warmer clothing. Only a few hours earlier, we were having paella on a warm Malvarrossa beach in Valencia, Spain.

As we were putting on our jackets and gloves, Rebecca made a comment that reminded me of one of Arthur C. Clarke's three laws:

Any sufficiently advanced technology is indistinguishable from magic.

She observed that we had not been outside since stepping out of our friends' car and into the Valencia airport, and had we instead taken the Valencia metro to get to the airport, we would have been able to step underground in downtown Valencia and then return above ground in downtown Stuttgart, having not been outside and exposed to any sort of weather or natural light the entire distance across three countries. We were completely comfortable in the clothing we wore in the warm Valencian weather up until the point of reaching the Stuttgart street level, and that to me is amazing.

So I was pleasantly surprised as I came across this image while catching up on some of my RSS feeds:

It's a great little drawing, based on the London Tube map, that shows all of the worlds metropolitan mass transit systems either currently in existence or in the works.

The culmination of our technology, ranging from efficient metro systems to air travel to client control systems, is indistinguishable from magic for nearly everyone who's lived before the 20th century (and even for certain people living in the 21st century, for that matter).

Image via strange maps.

de.php.net has an invalid DNS entry

Victor — Sun, 02 Dec 2007 20:02:53 +0000

For a few days now, I've been unable to reach http://de.php.net, not because the site has been down but because of incorrect DNS configuration by de.php.net's Germany host. When you request a PHP manual page, PHP.net does this trick of geo-locating your IP and redirects you to your closet PHP.net mirror.

If you take a look, de.php.net is actually a CNAME record for php3.globe.de, which is in turn authoritative at ns1.dns-service.net. The problem is ns1.dns-service.net doesn't have a record for php3.globe.de:

Searching for de.php.net A record at k.root-servers.net [193.0.14.129]: Got referral to d.gtld-servers.net. (zone: net.) [took 48 ms]
Searching for de.php.net A record at d.gtld-servers.net. [192.31.80.30]: Got referral to ns1.easydns.com. (zone: php.net.) [took 42 ms]
Searching for de.php.net A record at ns1.easydns.com. [216.220.40.243]: Got CNAME of php3.globe.de. and referral to m.root-servers.net [took 76 ms]
Searching for php3.globe.de A record at c.root-servers.net [192.33.4.12]: Got referral to C.DE.NET. (zone: de.) [took 36 ms]
Searching for php3.globe.de A record at C.DE.NET. [208.48.81.43]: Got referral to ns1.dns-service.net. (zone: globe.de.) [took 46 ms]
Searching for php3.globe.de A record at ns1.dns-service.net. [212.124.35.10]: Reports that no A records exist. [took 133 ms] Response: No A records exist for php3.globe.de, and php3.globe.de does not exist. [Neg TTL=86400 seconds] Details: ns1.dns-service.net. (an authoritative nameserver for globe.de.) says that there are no A records for php3.globe.de, and that the hostname php3.globe.de does not exist. The E-mail address in charge of the globe.de. zone is: guardian@globe.de. NOTE: One or more CNAMEs were encountered. de.php.net is really php3.globe.de.

Normally when the master nameserver is non-functional, DNS queries should fail-over to slave nameservers and pages are served normally. However, in this case the master nameserver is fully functional - it just doesn't have a record for php3.globe.de. What's funny is that the secondary nameservers for globe.de, ns2.dns-service.net and ns3.dns-service.net, have the correct A record for php3.globe.de, which means that the servers' zone serial numbers are off and master->slave propagation isn't happening correctly.

I've easily solved my problem by just adding the correct entry into my local machine's hosts file, but I shouldn't have to do this (nor does this fix the problem for everyone else in Germany). This just goes to show that DNS can be complicated and even the pros mess up every once and a while.

Update: It seems to be working today. It took three days after I posted before someone at Globe.de noticed the problem and added the correct DNS entry.

Upgrading a 1.0.2 iPhone to 1.1.1 Painlessly on Windows

Victor — Thu, 25 Oct 2007 21:59:01 +0000

I've been mulling over upgrading my iPhone's firmware to 1.1.1 for the last few weeks but have been put off by the complexity of it. All the tools and steps had been outlined so it was merely just running through the steps, but there were a ton of them, and it seemed like more of a pain in the ass than it was worth. Well, the hacker community has come through again with a easy solution, and now the 1.0.2->1.1.1 process is pretty painless.
If you have an unlocked 1.0.2 iPhone and used AnySIM 1.0x to do it, you must "virginize" your iPhone before you can upgrade to 1.1.1. Apparently AnySIM 1.0x had a bug that damaged the seczone of your baseband firmware, causing the 1.1.1 upgrade to brick your iPhone. With the new release of the iphone-elite RevirginizingTool, here's how you do it on a Windows machine:

Make sure iTunes is configured to sync your contacts to something (e.g. Windows Address Book) and your photos are backed up (I lost the photos on my iPhone).
Virginize your iPhone back to 1.0.2 OS and baseband using RevirginizingTool
Use iTunes to upgrade to 1.1.1
Jailbreak 1.1.1 using CARNAVAL
Use AppTapp (installed by CARNAVAL) to install BSD Subsystem and OpenSSH
SCP AnySIM 1.1 to your iPhone
Run AnySIM
GREAT SUCCESS!!!

Assumptions

You have an unlocked 1.0.2 phone that used AnySIM 1.0x
You have BSD Subsystem and OpenSSH packages installed (install with Installer.app aka AppTapp)
You have a strong WIFI signal
If you have a virgin phone, just use iTunes to upgrade to 1.1.1 and then skip to step 4

Detailed Procedure

Step 1

Run a full sync on iTunes to back up all of your settings. This will take care of your address book and most of your various settings. Copy off any photos you want to your computer.

Step 2

Download this. This is the latest release from the iphone elite team packaged with a script that will automatically virginize your iphone by backing up your seczone and flashing the baseband to its factory 1.0.2 state. Now SCP (try WinSCP) this file to your iPhone using login "root" and the default password of "alpine". SSH into your iPhone using root/alpine, and then make sure the file is in your iPhone's root directory, since the default home (really /private/var) is a partition mounted with noexec. Move the file from root user's home to the file system /:

mv ~/virginiser.tar.gz /

Then change dir to /, then run this command (all on one line, or you can run each command between the && separately):

tar -xzvf Virginiser.tar.gz && ./Virginiser/virginise.sh && mv Virginiser/seczone.backup .

This untars the archive, runs the automated script, and then copies a backup of your seczone to the filesystem /. SCP this seczone.backup file to your computer. Hold down the power button, slide to confirm, and then reboot your iphone. Now you should have a 1.0.2 iphone locked but still jailbroken.

At this point I recommend using iTunes to restore your iphone to 1.0.2 factory fresh. I had some problems with CARNAVAL installing its version of BSD Subsystem and OpenSSH over my previous installations using AppTapp. After the iTunes restore, it'll reboot to exactly how a 1.0.2 iphone comes out of the sealed box - needing activation and SIM-locked.

BTW, this step is basically an attempt to automate the manual steps shown here, so if you have trouble or need more detail, just read that page.

Step 3

Use iTunes to upgrade to 1.1.1. You should now have a factory fresh equivalent of a 1.1.1 iphone.

Step 4

Download CARNAVAL and unzip. Connect your iPhone to your computer using the USB cable. Make sure you kill iTunes.exe and iTunesHelper.exe in your Windows Task Manager. Run the file "(CLICK HERE) by brasuco.bat", which will walk you through the process. FOLLOW THE INSTRUCTIONS CAREFULLY. During one of the steps, you are asked to create a URL link to http://jailbreak.toc2rta.com - if that URL doesn't take you back to the activation screen as expected, no worries, just tell clear it and type into the Safari URL bar this address instead: http://jailbreak.kengz.com. Visiting this will download the exploited TIFF, essentially rooting your iPhone.
After you've followed all CARNAVAL instructions and learned about Brazil, you should be greeted with a fully jailbroken and activated iPhone.

Step 5

CARNAVAL installed AppTapp, which shows up as Installer on your iphone's home screen. Use that to install BSD subsystem and OpenSSH. Now go to your iphone's WIFI settings to find your iphone's IP address. You should now be able to SSH/SCP into your iphone using root/alpine.

Step 6

Download AnySIM 1.1. Unzip AnySIM-1.1.zip, which will give you a anySIM.app directory. SCP this directory (not just the files) into your iPhone's /Applications directory. Then chmod all the files inside your iphone's /Applications/anySIM.app/ directory to +x (0755 will work). Once this is done, reboot your iphone to find the AnySIM icon on your iphone home menu.

Step 7

Run AnySIM, slide to confirm, and go get a cookie.

Step 8

You now have a fully jailbroken, activated, and SIM-unlocked iphone. You also have Installer.app, a aptitude like application that lets you install a ton of really cool third party applications over the air, complete with notifications of application updates.

Good Luck

The entire process only takes about 30 minutes. I've been running 1.1.1 for a few days and haven't noticed any problems. I really dig the increased speaker volume and the space, space to add a period and a space when using the keyboard. Good luck, and leave a comment this works for you (or if you have any questions or suggestions).

iPhone owned

Victor — Tue, 02 Oct 2007 04:30:20 +0000

This is the first time I've been back in the States since the iPhone was released, and after getting a chance to play with one yesterday, I decided I had to go out and buy one today. The problem is that AT&T-locked JesusPhone doesn't work in Germany without a little bit of hackery, and Apple's just-released-last-week 1.1.1 firmware is reported all over the internet to brick hacked iPhones. Would an AT&T store in Mount Pleasant, SC turn over so many iPhones that they would already be selling boxed 1.1.1 models?

South Carolina isn't exactly a technology hotspot, so I took a bet that I could buy a boxed iPhone with an older, perhaps original 1.0, firmware (flip-flops and sundresses, on the other hand, sell like hotcakes here). I won my bet and within an hour of getting home, I had a completely unlocked iPhone with a myriad of cool unofficial apps, thanks to tools like iBrickr and PACAY. My new iPhone now has a youtube viewer, OpenSSH, BSD utilities, a wifi stumbler, flickr uploader, and a completely cool Over The Air application installer.

The device is incredibly well built and solid. It's smaller than I had imagined, but the screen is definitely adequate for browsing. The touch screen obviously lacks tactile feedback, but it's also the most accurate touch interface I've ever used. I'm able to type much faster than I ever could using normal keys and T9.

With it unlocked and freed from Apple's proprietary grip, the iPhone is very special. Free and open source software on such a beautiful piece of hardware is very exciting, but when Apple sets out to hamper such innovation and creativity, I can't help but think of their past mistakes with closed systems. Things could have turned out very differently had MacOS been a little more open early on, and things may very well turn out similarly if Apple doesn't change their ways. Here's to hoping Google's gPhone will be the IBM PC to Apple's II.

Highly useful link: Unlock your iPhone using the latest AnySIM