Today I left a meeting a few minutes after the beginning of the Back to my Mac event. I went to my car to drive home, and before turning on the engine, I grabbed my iPhone, launched Safari and tuned in to the live video feed. I placed the phone somewhere on my car where I could listen to it, and took off. During all the 20 minute drive home, I could listen to the event (I didn’t watch it, because, well, I was driving!), and only twice did the iPhone pushed it back a few seconds to recover (immediately) from dropped packets. I then got home, and switched to the Mac using my fiber connection.

Putting all this in perspective, Steve and his team were speaking in Cupertino, a bunch of equipment was capturing the event, encoding it and streaming it to half a world away, to a (fast) moving car on a freeway, where I was receiving it with a powerful, stunning device that has more CPU power than any computer I had when I was a kid and still fits in my pocket. And the quality was perfect.

Yes, this is why all of us, in this industry, in this passion, do what we do.

I’ve done this to learn about REST, and get the feeling of how should things be made, what problems would I find, and so on. I spent some time creating an easy-to-use methodology on the iPhone side, making the communication APIs simple but flexible. Also, I looked at problems like authentication and error reporting.

If you’re interested, the source is published on Google Code, together with some nice documentation on the project wiki, explaining the general architecture of the client and solutions to some of the problems I found. Hope you find it useful. :)

In this post I’ll explain how to recover from a catastrophic failure, like when a hard drive dies. Although much of the steps are identical, this post is focused on how to recover the entire file system and not a small set of files that an user accidentally deleted. To recover something like that, you can simply run amrecover on the server, recover the files you need and transfer them using SFTP or any other protocol to the client machine.

Before going any further, let me just point out an important detail: if you have a downtime on a machine that lasts for a few days, and your tape rotation cycle is not long enough, you may loose your backups at some point (assuming you backup frequently enough). To avoid this, you have several options. The easiest one is to simply disable whatever mechanism triggers your backups (launchd, crontab, etc). Of course, this only makes sense if you have only one machine being backed up on a given Amanda configuration. If stopping the backups is not possible, you can copy the tape directories or, better yet, the tar files that match the affected machine’s volumes to a safe place. Remember you have to copy the correct files, from level 0 to whatever level you are on. If Amanda overwrites the tapes after doing this, you can always recover directly from the tar files.

To recover from an Amanda backup, you should be prepared with a small hardware setup that will make your life easier. I recommend you prepare the following before a problem occurs, so you can avoid doing this under stress. What you’ll need is an external hard drive with Mac OS X installed and the Amanda client installed on it. This way, you can boot the Mac with the freshly installed empty drive and restore the backed-up contents directly to that drive. Note that if you handle PowerPC and Intel Macs, you’ll need to be ready for both situations. The easiest way is to have two drives, although I believe it’s possible to have two bootable partitions on the same drive using a few tricks. Also, I recommend you install the latest OS for both processor families (Leopard on PowerPC, Snow Leopard on Intel).

When recovering, the Amanda client needs to contact the server, which means (assuming you’re using ssh security) you need to generate a public/private key pair on the client and install the public key on your server. I recommend you remove the public key from the server as soon as your recovery operation ends, for security reasons.

So, let’s assume you have a client made out of a Mac with a new, formatted and empty internal hard drive, and running Mac OS X installed from an external drive, with Amanda client installed and a public/private ssh key pair whose public key is already on the server. Let’s also assume the volume list of this machine is the one I recommended on my previous post:

/
/Applications
/Library
/Users

Here are the steps to recover everything from the backups to the empty drive:

1. Launch amrecover on the client using the following command. This will start the recovering console.

/usr/local/sbin/amrecover -C configname -s my.server.com -t my.server.com -oauth="ssh"

Replace configname by the name of the Amanda configuration you want to use for recovering, and my.server.com for your backup server. Note that you must refer to your backup server using a canonical name, not an IP. If you don’t have a canonical name, you can edit /etc/hosts and create one.

2. Run the sethost command to set the host you want to recover. You can get a list of hosts managed by the current Amanda configuration using listhost. To recover from a host called andromeda, you would use:

sethost andromeda

3. Use the setdisk command to define what volume you want to recover. Let’s start with the root volume, so type the following:

setdisk /

Like in the hosts case, you can use the disklist command to list the backed-up volumes.

4. Use the lcd and lpwd commands to move to the root of the empty disk. This will tell Amanda where you want to recover stuff to. If your empty disk is called NewDisk, the command would be:

lcd /Volumes/NewDisk/

You can use the lpwd command to verify you defined the local directory to the correct path. It’s extremely important you don’t forget to define the local directory! Doing so results in Amanda recovering the data to whatever local directory is starts on, which is almost certainly not what you want to happen.

5. Add the list of items to recover.

We want to recover everything, so let’s just add the root directory:

add /

You can use the usual UNIX commands (cd, pwd) to navigate trough the backed-up hierarchy and select exactly what you want to recover. Note the paths are relative to the root of the current volume, not the root of the original client.

6. Initiate the recovering procedure. To do that, simply use the extract command:

extract

Amanda asks you to confirm the operation, and will ask you if you want to load the needed tapes during operation. For unattended operation, I recommend you to hit the return key several times to answer “Yes” to all the tape change questions (assuming you’re using virtual tapes, and not real tapes, of course). This will take a while.

Repeat the following 3 steps for the remaining volumes (I’ll use /Library as an example):

7. Use Finder or another terminal window to create the root folder of the volume:

cd /Volumes/NewDisk
mkdir Library

8. Back on Amanda, change disk and local directory to match the new volume:

setdisk /Library
lcd /Volumes/NewDisk/Library

9. Add all the files in the current volume, and initiate extraction:

add /
extract

Again, repeat the previous steps for all the volumes you want to recover.

10. Use Disk Utility to Repair Disk Permissions on the new disk. This will make sure the proper ownership and permission settings are assigned to the directories you created (and everything else).

You should now be able to boot from the recovered drive. You’re done. :)

As with any UNIX tool, Amanda can be compiled, installed and configured in a lot of different ways. How you should do it depends on your needs, so don’t feel pressured to do everything in the same way I did, as it’s not “the right way”, just one way. Also, everything I describe here should work on the Leopard or Snow Leopard versions of either Mac OS X or Mac OS X Server on Intel or PowerPC Macs. I’m not sure about previous versions of the OS, but you may find more information about those in the Mac OS X installation notes page of the Amanda wiki.

Also, I won’t write anything here about installing an Amanda server. I’m assuming either you already know how to do it and just want to check how to backup Mac OS X clients, or that you are new to Amanda and will follow the appropriate tutorials and do some experimentation before moving Amanda to production.

Before installing Amanda on a client, you need to decide what you want to backup, and how to split that into Amanda “disks”. I usually backup the entire machine (a single Mac OS X volume), so the obvious choice to make would be to consider the root of the file system as the only Amanda disk. But I recommend you to not do that. Why? Two reasons:

• If you consider the entire Mac OS X volume to be a single disk, a level 0 backup will use a lot of space, and take a long time to complete. Even worse, if the space needed for a level 0 backup is as large, or larger, than a single tape, you may not be able to perform the backup at all. By splitting the volume into smaller disks, Amanda will spread level 0 backups of those disks over time, which will also spread the space and time used for those necessary level 0 backups.
• Different parts of your Mac OS X volume have different usage patterns. The /System directory, for instance, contains mostly read-only files that change only when installing a system update, which is a relatively rare event. On the other hand, the /Users folder contains user home directories, whose contents may change on a daily basis. This different behaviors lead Amanda to optimize it’s planning, specifically the decision to maintain of increase the backup level of a given disk, as explained in the previous article. If you don’t split two main directories like these, with radically different usage patterns, in different disks, you are making it harder for Amanda to make the right decision.

The way I split an OS X volume into several disks depends on what machine is used for. However, I end up doing it always the same way for normal use desktop machines. I create the following disks:

/
/Applications
/Library
/Users

This usually works fine. Note that, if you are not careful, by considering / (the root) a disk, you’ll include all the other disks (those directories are inside the root) and any other devices (hard drives, CD-ROMs, USB pens, etc) that may be plugged in that computer (they will be mounted into the /Volumes directory). I was told Amanda won’t include mountpoints (which means it would not backup anything inside /Volumes anyway) but I haven’t tested. We’ll see how to solve this issue shortly.

If your users home dirs contain a lot of data, or data that don’t need to be backed up, you may consider splitting the /Users directory into smaller disks, and excluding easily restorable directories, like ~/Music. I never had to split the /Users directory, but I do exclude the ~/Music directories and any directories that contain virtual machine hard drive images (like Parallels or VMWare). Those files are huge, and change every time an user launches their virtualization software. If your users don’t have important data on those virtual hard drives (like in my case, we only use Windows to run IE), you can easily re-install Windows from scratch. If you do have important data on those drives, I recommend you install backup software on the virtualized OS to keep that data safe.

Ok, enough talking, let’s do it.

1. Install a recent version of Xcode. You’ll need it to compile the needed stuff. Xcode 3.2.3 is the most recent version at the time I’m writing this and will do just fine.

2. Install MacPorts. Amanda needs glib2 to compile. glib2 has a lot of dependencies, so the easiest way to install it is using MacPorts. To do that, simply open the MacPorts installation page, download and run the appropriate installer for your version of OS X.

3. Install glib2. After having installed MacPorts, this can be achieved using one simple command (run it as superuser):

/opt/local/bin/port install glib2

All glib2 dependencies (and glib2 itself) will be downloaded, compiled and installed. This will take a while, specially on PowerPC Macs, so if you need to have a snack or refill your coffee mug, this is a good time to do it.

4. Create a user for Amanda and add it to the admin group. The best way to achieve this is to run the following list of commands as superuser:

sudo dscl localhost -create /Local/Default/Users/amandabackup
sudo dscl localhost -create /Local/Default/Users/amandabackup RecordName amandabackup
sudo dscl localhost -create /Local/Default/Users/amandabackup UserShell /bin/bash
sudo dscl localhost -create /Local/Default/Users/amandabackup RealName "Backup User"
sudo dscl localhost -create /Local/Default/Users/amandabackup UniqueID 5000
sudo dscl localhost -create /Local/Default/Users/amandabackup PrimaryGroupID 0
sudo dscl localhost -append /Local/Default/Groups/admin GroupMembership amandabackup
sudo dscl localhost -create /Local/Default/Users/amandabackup NFSHomeDirectory /Users/amandabackup
sudo ditto -rsrcFork '/System/Library/User Template/English.lproj/' /Users/amandabackup
sudo sh -c "echo 'amandabackup_server.example.com amandabackup' > /Users/amandabackup/.amandahosts"
sudo chmod 600 /Users/amandabackup/.amandahosts
sudo chown -R amandabackup:wheel /Users/amandabackup
sudo passwd amandabackup

The last command defines a password for amandabackup user. You can later configure your ssh deamon to accept only public key authentication.

5. Download Amanda. You may obtain Amanda’s source code from the Amanda downloads page. Although the 3.* versions are already available, I’m using 2.6.1p2 for now. This instructions assume you’ll be using the same version. I’m not sure if version 3 compiles well on OS X. After downloading, uncompress the file.

6. Compile and install Amanda. Amanda is installed using the standard UNIX configure/make/make install commands. You have, however, to provide some information on the configure command. The one I use is:

./configure --with-user=amandabackup --with-group=admin --with-ssh-security --without-server

After that, just run the usual:

make
sudo make install

7. Create the /usr/local/var/amanda directory with the appropriate ownership. The install procedure somehow fails to create this directory, so you need to create it manually and give it the right ownership. So, as superuser, run the following commands:

mkdir -p /usr/local/var/amanda
chown amandabackup /usr/local/var/amanda

Amanda is now installed on the client. If you are going to use ssh as the tunnel for your connection, you should now copy the server’s ssh public key to the client and test the connection. You can then add the client’s volumes to the disklist and run amcheck to test the configuration.

Now you need to add exclude files to make sure the root volume won’t contain all the others, and to exclude large files whose backup is not needed (like music or movies). To do that, you have to add a configuration directive in your Amanda server configuration files. That directive tells Amanda the name of the file that includes the exclude list, which is the list of files to be excluded from backup. This is a sample configuration I use:

define dumptype comp-tar {
    program "GNUTAR"
    compress fast
    index yes
    record yes
    auth "ssh"
    exclude list optional ".amanda-exclude.list"
}

Now you need to create the exclude files where needed, named “.amanda-exclude.list”. I usually use two of those files. One in the root of the file system with the following content:

./Volumes
./Applications
./Library
./Users
./home

This excludes the other Mac volumes (that are mounted in the /Volumes directory) and the other defined Amanda volumes (the word “volume” is used on both contexts which may lead to confusion, but you want to exclude both kinds of volumes, so it’s ok). If you’re wondering about /home, it has no useful content (I suppose it’s there just for compatibility with some UNIX tools) and causes problems if not excluded.

The second file would go in /Users directory, and contain something like this:

./<user>/Music
./<user>/Documents/Virtual Machines.localized

Replace <user> with the username of the machine user, and duplicate the lines for additional users. This would exclude the user’s music directory and the VMWare disk images. Of course, your milage may vary. If you want to backup everything, just don’t add this second file.

You should now have a perfectly functional Amanda client.

I’ve been using Amanda for a while now to backup all the Macs in our workgroup (10 machines) and so far I’m nothing but happy. Amanda is an open source backup system for UNIX-based operating systems, Mac OS X included (I believe it can also backup Windows clients, but I couldn’t care less).

Amanda was originally designed to backup to tapes. Today, since hard drives became cheap and are a great media for backup, Amanda also supports virtual tapes. A virtual tape is a directory on disk that essentially acts as a tape, storing raw information. While configuring an Amanda server, you create a set of tapes of an arbitrary size (100 GB, for instance) and Amanda will use at least one tape per day, eventually rotating trough all of them.

I’ll now lay down some considerations about how Amanda works and how does that reflect on usage and backup planning. Also, I’ll do some comparisons with Time Machine.

Scheduling

In amanda, you define one or more configuration files. Each configuration can have a set of hosts (a backed-up machine), and for each host a set of disks (a disk is any directory you want to backup, it’s not needed to match Amanda disks with physical disks or logical HFS+ volumes). You generally run a backup operation that will backup all the disks of all the hosts on that configuration file.

Launching a backup operation is done by simply executing an UNIX command from whatever mechanism you prefer (cron, launchd, manually, etc). This means you can define how often does your backup run, and at what time. Usually, a daily backup is performed, but you may want to backup every 6 hours or only once per week, depending on how often does your data change and how bad is it to loose a few hours of work. This means that, in Amanda, the server is proactive and initiates the backup procedure whenever you set it to. In Time Machine, backup operations are initiated by the clients every hour. The server is simply a dumb file server where backups are stored.

Storage

Amanda uses the tar format to store data on a virtual tape. For each backed-up disk Amanda creates a tar file inside the tape with data. Amanda stores data in incremental fashion, using some redundancy as well. This is implemented trough the concept of backup levels. The base backup (where all the files on a disk are backed-up) is level 0. From there, backup levels are incremented, and each level essentially means it contains the incremental changes relative to an archive in the immediate lower level. This means a level 3 backup contains the changes relative to a level 2 backup. That level 2 backup contains the changes relative to a level 1 backup which contains, as expected, the changes relative to the level 0 backup.

Amanda decides on the level is should operate on based on a complex planner that considers several factors that result in a decision. You may add some configuration options to customize the weight of some of those factors in the final planner decision. Essentially, Amanda tries to obtain the right balance between reliability (which, in this context, means the probability of recovering a backup successfully) and used disk space. Reliability decreases as the backup level increases, due to the fact that, if you want to recover a level 5 backup, you need to have the consistent level 0, 1, 2, 3, 4 and obviously 5 backups, because each of those build upon the previous one. A level 0 backup is more reliable in the sense that you only need the level 0 backup itself to recover. Of course, a level 0 backup is an “everything” backup. If you do a level 0 backup every day, you’ll use a lot of disk space.

This is an example of a production Amanda installation for a host called bergman and volume / (the root):

date                host    disk lv tape or file file part status
2010-07-01 06:05:24 bergman /     2 DAILYS-6       15  1/1 OK
2010-07-02 04:19:11 bergman /     3 DAILYS-7       23  1/1 OK
2010-07-04 04:47:44 bergman /     3 DAILYS-9        9  1/1 OK
2010-07-05 02:51:01 bergman /     0 DAILYS-10      21  1/1 OK
2010-07-06 02:51:34 bergman /     1 DAILYS-11      11  1/1 OK
2010-07-07 04:59:40 bergman /     1 DAILYS-12      12  1/1 OK
2010-07-08 02:59:32 bergman /     1 DAILYS-13      15  1/1 PARTIAL
2010-07-10 02:46:41 bergman /     1 DAILYS-15      20  1/1 OK
2010-07-11 05:25:49 bergman /     2 DAILYS-16       8  1/1 OK
2010-07-12 03:43:04 bergman /     0 DAILYS-17      16  1/1 OK
2010-07-13 04:25:23 bergman /     1 DAILYS-18      11  1/1 OK
2010-07-14 02:56:48 bergman /     1 DAILYS-19      15  1/1 OK
2010-07-15 02:54:01 bergman /     2 DAILYS-20      11  1/1 OK
2010-07-16 02:46:30 bergman /     2 DAILYS-1       11  1/1 OK
2010-07-17 05:05:08 bergman /     3 DAILYS-2       12  1/1 OK
2010-07-18 03:00:43 bergman /     3 DAILYS-3        9  1/1 OK
2010-07-19 05:11:36 bergman /     3 DAILYS-4        7  1/1 OK
2010-07-20 02:49:03 bergman /     4 DAILYS-5       24  1/1 OK

The backup level is indicated by the “lv” column. As you can see, I have 20 virtual tapes, and the last one to be used was DAILYS-5. The next one will be DAILYS-6 (its content will be erased and the tape will be reused). You may be asking yourself why are there two level 0 backups, and a lot of consecutive repeated backups with the same level.

Consider two rules of thumb to understand that:

1) A level 0 backup is needed to recover, so Amanda must make sure at least one level 0 backup exists at any time, given the number of tapes and their rotation. Also, bad stuff can happen, like a machine being down or unavailable at the time the backup runs. If that happens during a few Amanda runs, it may happen than you hit a number of rotations where you effectively loose the level 0 backup without creating a new one. That’s why Amanda makes a few level 0 backups among the way, to make sure you still have a second level 0 backup if the first one is destroyed. You can configure the maximum number of runs that go by without a level 0 backup being created. It’s highly recommended that that number is lower than the half of the total amount of tapes you have, so that, in the worst case (like in my example, where I have 20 tapes), you have a level 0 backup in the “middle” of your tape recycling circuit. Now, why did Amanda created two level 0 backups, one in July 5, and another one in July 12, effectively less than the maximum time allowed? See below.

2) Despite what seems intuitive, Amanda does not increase the backup level every run until it gets back to zero again. I don’t know in detail all the data the planner uses to make a decision, but there are at least two interesting considerations the planner takes into account.

The first one, is a level N+1 backup a lot smaller than a level N? If the answer is no, Amanda decides to keep the same level. There’s no point in increasing the level (and lowering reliability) if no significative amount of disk space will be saved. This may happen if you change approximately the same set of files in each run. In that case, the incremental backup from level N to N+1 would be of almost the same size as the N-1 to N.

Second, Amanda sometimes promotes level 0 backups ahead of schedule to spread them trough time. Doing all the level 0 backups in the same day would be very slow and might not fit in the maximum number of tapes allowed for a single run (you can define that value on the config file). There’s also another factor: free space on the tape. Remember that, for now, Amanda cannot use the same tape in two consecutive runs, so there’s no point in leaving unused space on a tape. If there’s enough space to perform a level 0 backup instead of a higher level, Amanda may decide to do it.

As you can see by now, data storage is handled quite differently in Amanda and Time Machine. I’ll assume you know how Time Machine works, so I’ll get straight to the point: in Amanda you may have more than one copy of the entire volumes you are backing up (ie, several level 0 backups of the same data), so you need to take that in consideration while planning storage space. On the other hand, amanda allows gzip compression (and encryption) of backup data, so it’s not completely obvious how much more (or less) space you need for Amanda backups compared to Time Machine. In some situations, if your data is highly compressible, you may even end up with two level 0 backups taking less space than a single Time Machine backup, although the opposite will happen most of the time. What’s cool is that if you use compression, Amanda learns with time how compressible your data is, and adjusts it’s planning according to that.

Another interesting note is Amanda using standard file formats for storing backups (tar and optionally gzip). This allows recovering of data even in machines where Amanda is not present. If you navigate into a virtual tape directory on your file system and run the “head” command in one of the stored files, you’ll see something like this:

AMANDA: FILE 20100715010001 serpa /Library  lev 2 comp .gz
  program /usr/bin/gnutar
DLE=<<ENDDLE
<dle>
  <program>GNUTAR</program>
  <disk>/Library</disk>
  <level>2</level>
  <auth>ssh</auth>
  <compress>FAST</compress>
  <record>YES</record>
  <index>YES</index>
  <exclude>
    <list>.amanda-exclude.list</list>
    <optional>YES</optional>
  </exclude>
</dle>
ENDDLE
To restore, position tape at start of file and run:
	dd if=<tape> bs=32k skip=1 | /usr/bin/gzip -dc |
          /usr/bin/gnutar -xpGf - ...

The first part is metadata used by Amanda. After the metadata ends, there are two lines used to tell you how to restore the data stored on that file with standard UNIX tools. This means you don’t need to worry about being able to recover your backups if Amanda development happens to stop for some reason. As long as you have an UNIX machine, you’ll be able to restore your data.

Security

There are two points I want to mention about security: transport and storage encryption, and system architecture. I’ll assume we’re always talking about having a backup server and several clients. If the problem you’re trying to solve is so simple that it can be fixed with an USB disk and Time Machine, you’re just wasting your time reading this. :)

Amanda supports encryption both during the data transport and on data storage. Transport security is guaranteed by using ssh with public/private key pairs. It also supports data encryption on the data storage by using symmetric private-key based encryption of public/private key pairs. Encrypting your backups is important, specially if you store them in an offsite location (either via network transfer to a remote data center, or by physically storing hard drives or real tapes in a safe). In the event data ends up in the wrong hands, encryption will rend it useless for the bad guys. Things are substantially different in Time Machine. Data storage encryption is simply non-existing (unless you use a lower level encryption method, like PGP). Transport encryption may be provided by the AFP protocol, used by Time Machine to connect to the backup server, depending on your server configuration.

More interesting, in my opinion, is the implicit security resulting from the client/server architecture used by Amanda. A backup server should be one of the most safer and well guarded machines you have in your network. All your data will be there. If the backup server gets compromised, it means all the data from all the backed-up machines might now be in the wrong hands. You would want your backup server to run as few services as possible, and to allow access to the server (like ssh) only by trusted admins from controlled networks.

In Amanda, this is possible. As I described before, when a backup operation starts, the Amanda server will contact its clients using the method you chose in the configuration (ssh in my case). So the connection is made from the server to the client, and not the opposite. This means your user’s machines (which will naturally be less secure than your server because those pesky users will run all the crap they get from the internets!) will never access the backup server, and though compromise it’s security. Time Machine works in the opposite direction. There’s no concept of a “backup server”. The clients run the show, and the server is simply a file server where backups are stored. This exposes the user backups to whatever malware and trojan horses they may have running on their Macs. If the server is misconfigured, or if some hacker exploits an unknown vulnerability in the AFP protocol, other users backups may be compromised as well.

Conclusions

Amanda may be a great option to backup always-on Macs, like xServes or desktop machines (along with Linux or FreeBSD servers, of course). It offers a vey fast, reliable and secure infra-structure upon you may build your backup system. However, it lacks the user interface and simplicity of Time Machine (most configurations require sysadmin intervention for recovering data from a backup). Also, Time Machine may be more appropriate if you rely heavily on laptops that will not be available on the network on a predictable schedule.

Amanda pros:

• Works on any UNIX system (and Windows clients) which may help when planning a multi-OS backup scenario.
• Offers very good control of used disk space.
• Allows data encryption and compression.
• Secure client/server architecture.
• May be used on real tapes, not just hard drive-based backups.

Amanda cons:

• Not appropriate for laptops that may be out of network reach during backup operations.
• Step learning curve, a little hard for beginners to configure and get everything running smoothly.
• Unless your users are computer experts and have access to the backup server, it requires sysadmin intervention for recovering data.
• Recovering a single file may take some time, because the entire tar archive has to be read until the file is found.
• Backups are usually less frequent than Time Machine.

Time Machine pros:

• Very simple setup.
• Non-expert users may recover lost files and easily browse filesystem history using the stunning user interface, without requiring sysadmin intervention.
• Works fine with laptops with intermittent network connections.
• Very well integrated with Mac OS X, makes backing up and recovering very easy and part of normal usage and new machine installation workflow.

Time Machine cons:

• Works only on Mac OS X, and requires a Mac OS X Server as backup server (unless you go with unsupported devices and face possible consequences).
• Offers no data storage security.
• Very hard/impossible to control data storage strategy and used space.
• Requires clients to access server, which decreases security.
• Works only with disk-based backups.

There’s no clear winner here, it highly depends on your needs and restrictions. I hope this article gave you a general idea of what Amanda is, how does it work, and it’s advantages and weaknesses. In the next article, I’ll describe how to install a Mac OS X amanda client, and how to recover from a catastrophic drive failure.

What I find paradoxical is that most people who are buying iPads are the ones who need it less. It’s the people who love gadgets, and those people already have everything – iPhones, laptops, desktops, you name it. What will they be able to do with an iPad that can’t be done with every other piece of electronics they have? Nothing.

But, as many people said and wrote between the iPad first announcement and this weekend where it finally started to reach the buyers hands, the iPad may open the computing world to a whole new class of users. Those are the users who do not use computers. The older people, the computer illiterate people, or those who simply don’t have patience or will to learn about operating systems, files and folders, network configuration, installing software and updates, and all the stuff needed to use and maintain a computer. iPad opens a new range of possibilities for those folks, allowing them to consume and create content and sharing it. Or does it?

Unfortunately, this amazing possibility is set aside by one simple fact: iPad needs a computer to work. In fact, according to the early reports, the very first thing you are required to do as soon as you fire up your iPad for the first time is to hook it up to your computer. Which means you must have a computer.

But… why? The iPad is a computer by itself. Technically, there’s no reason at all for it to need any help from an external computer. It can boot. It can connect to any wireless network, download whatever it needs from the cloud, and install it. What the heck, my Mac does that on it’s own. Why won’t the iPad do it?

Of course, there’s another problem: the data. The main reason the iPad (and the iPhone) is required to be plugged to a computer during installation is to create the necessary bond between both machines so data can be synced and backed-up. If only there was some autonomous device that could talk to the iPad and backup it’s data… well, there is. The Apple’s very own Time Capsule, of course! And with 150 Mbps of wireless bandwidth between them, there’s more than enough horsepower to provide a seamless, continuous backup of the data stored in the iPad flash drive.

What made me fall in love with computers when I was a small kid was the fact that computers enabled. They enabled people to do so much more that could be done previously. They enabled me to write good-looking reports on my Mac Plus that stunned my school teachers. To edit images and sounds in a way that could only be done by artists with expensive equipment. To build simple applications and games with Hypercard that actually worked. To reach other people and talk to them, and to have passionate arguments about mostly everything. To download software so I could do even more stuff. To post my own websites with information I created to help other people, and ultimately, my software, that would enable others to do what they needed and wanted.

So… my suggestion: reach those who can’t or won’t use a computer, and tell them there’s a new way to enter the digital world. There’s a new way to create. There’s a new way to enable and be enabled. A way they can understand, a way they can feel comfortable with, a way they may even end up loving. Give them an iPad that flies solo. Give them a magic box that they place somewhere in their house that keeps their data safe. Give them what you gave me on that day my father brought that strange beige expensive box home. Do it!

Retrospect 6 was a very good backup software, but its age was showing up. It was still fundamentally a Mac OS 9 app running on top of OS X Carbon API. The worst part about this was the need to launch the application in Finder, forcing you to have automatic login configured and remote-acess your backup machine via Remote Desktop or VNC, assuming it was running on a data center. Despite that, it was very reliable (I had two situations where Retrospect complained about corrupted data, and both situations were caused by faulty hardware). Also, it was not the fastest software I had seen, but it was good enough. Keep in mind Retrospect was designed when file systems had hundreds or a few thousand files on them, not a million or more like it’s normal today (I have about 1.5 million files on my laptop drive).

Facing the unavoidable, EMC decided to re-write Retrospect from the ground up as a modern OS X product, using Cocoa APIs and changing it’s architecture to a proper UNIX daemon (with a remote graphic console). Besides theoretically solving all the version 6 downfalls, they added some nice goodies, like AES 256 encryption, grooming, and some details. A nice one is that the disk backups are now stored in 100 MB files instead of a unique, giant file. This solves a lot of problems related to NAS systems.

Assuming you need backup software, you would think this were great news, right? Well, so did I. However, reality seems to be a lot worse than the perfect scenery I described above. EMC worked a lot of time on this version, and you would think they had made it, made it good and made it fast, right? Well, they decided to stop somewhere in the “Make it good” part. The problem is, rules of thumbs are nice, but common sense helps. If you launch a product that is simply too slow to be useful, people won’t use it.

Our backup machine at the university is an old G4, dual CPU (1 Ghz, I guess). Yes, it’s not exactly a screamer, but what the hell, we are talking about copying and storing files. It’s not exactly rocket science, and if the machine was up to the task when it was new, it should be up to the task now. We upgraded the RAM to a decent amount, of course.

So, last week I needed to recover the /etc directory of a colleague’s laptop to recover some apache and PHP settings that Snow Leopard installer happily overwrote. We’re talking about 3 MB of data, something that should be as simple as pressing a few buttons and get the data back. The laptop had about 750 thousand files on it, which, in my opinion, is not that many for today’s standards. So, why the hell took me about 3 hours to recover those files? Loading the catalog into the UI took almost 2 hours. Deselecting the whole file tree took almost 1 hour. The rest was the recovery process itself.

Ok, EMC. I know this stuff is optimized for Intel, and you do a lot of byte-order swapping on PowerPC. I know I’m using an old machine. But for god sake. What the hell are you doing to my CPU that needs 3 god damn hours to load a 750 000 files catalog into memory? And what’s the story with deselecting all the files? Are you telling me that, when I press the checkbox on the file root, you REALLY go trough the entire file tree and deselect each individual file? (In a rather inefficient way, because changing 750 000 booleans would take about… what? 1 millisecond on a 1 Ghz CPU?)

Well, I have at home a Dual 2 Ghz PowerPC machine that acts as backup storage, among other things. I have an AES 256 encrypted disk image (note that, at the university, I’m not using encryption, or else it would render things unbearable), served by AFP, acting as the time machine target for my laptop. When I installed Snow Leopard, I also had to recover some stuff from the /etc directory. Do you know how long it took? About 5 freaking seconds! And please, don’t tell me it’s because the G5 is faster!

Enough is enough, and 3 hours to recover a few files is ridiculous. I’ll study the possibility of migrating backups to Time Machine. Yes, space management is worse. Yes, client machines have to access the server, making the whole setup less secure than Retrospect where the server accesses the clients. But people won’t have to stop working for hours waiting for some files to be recovered.

Lesson for you, EMC: I’m not asking for you to make it fast before making it good, but at least make performance acceptable before releasing the product. Specially after taking so many years to build it.

• Unlimited songs!
• No DRMs! Listen and record to whatever you want, legal or not! Just play it. Or record it!
• Social features: Share your music! Plug two headphone sets, enjoy your favorite tunes with a friend, in stereo! Or switch tapes with your mates. No complicated wireless setup, no waiting for files to be transfered, no internet access required. The best social experience ever!
• Built-in stereo microphone and line in. Records full quality sounds for as long as you want (with short breaks every 45 minutes)
• Advanced space duplication algorithm: when you think the media is full, just warp it around and you have 100% more space!
• No desktop, laptop, tablet, super-computer or any other damn thing with transistors in it required. No software. No platforms. No broadband internet connection. Nothing. It just works.
• Strong, hard, heavy-duty case. Won’t break, period.
• Very reliable, can work for decades, even under rough usage (trust me).
• Replaceable built-in batteries. Can also be powered by external batteries or power supply.

Some photos:

(Unfortunately, I forgot to remove the batteries about 15 years ago… there’s a huge mess inside the battery compartment)

Despite this, our database kept growing like crazy, with no apparent reason. I decided to investigate what was going on.

The vacuum verbose is a neat command you can use in the PostgreSQL console to check what’s going on with the built-in cleaning service. This will run a vacuum on every table, giving you extensive report on how many used and unused rows exist in each table, and how many of the unused were deleted and their space reclaimed by PostgsreSQL.

The most interesting output of this command are the lines (one per table) like: DETAIL: 2942 dead row versions cannot be removed yet.. This means you have, in this example, 2942 rows on that table that are not used any more, but for some reason, vacuum was not able to delete them.

Browsing the web, I found out the most probable cause of this issue is the existence of a transaction that started a long time ago, and never finished. PostgreSQL will only delete rows older than the start timestamp of the oldest transaction still running. Newer versions might be needed by those transactions, so they are kept in place, even if their death sentence is already declared.

So, I started looking at what was going on inside the database engine, searching for those pending transactions. A lot of useful information about the current status of a PostgreSQL database can be found in many tables that exist in memory during PostgreSQL execution, that can be queried as any regular table. The one that was important for me was pg_stat_activity, which contains a list of all the current transactions.

By running the select * from pg_stat_activity; SQL command, I saw that each instance of my application had create two database connections, and for each instance, one of the connections had a transaction that started at the time the application instance was launched, but never committed or rolled back.

After some time of poking around, reading code, and talking to the right people, the problem was revealed. By default, WebObjects applications will open two connections to each database they talk to. One for the actual data queries, and the other to ask the database, at startup time, for what’s called the jdcb2info. This is a bunch of information needed to map between PostgreSQL and Java data types. And, due to one of those very old bugs that strangely nobody fixed yet that make WO fun, the transaction where that information is requested is never committed. So, the transaction will remain open, and PostgreSQL won’t vacuum a thing until you quit your application.

Although there’s not yet a solution for the problem itself (hopefully there will be, as I filled a bug in Radar), you can avoid it, by using the jdcb2info bundled in the PostgreSQL plugin instead of requesting it from the DB. To do this, simply append ?useBundledJdbcInfo=true to your DB connection URL, like jdbc:postgresql://localhost/dbname?useBundledJdbcInfo=true. This way, the second connection is not created at all, and no pending transaction will occur.

And of course, after doing this, the DB happily vacuumed hundreds of thousands of dead rows and stopped growing like crazy. Also, a tip: if you really need to vacuum the database right now, but you can’t stop your apps and interrupt service, simply obtain the pids of the PostgreSQL processes that are maintaing the offending connections from the pg_stat_activity table, and kill them, using the unix command kill <pid> command. The connections will die, but your apps will not.

• The IT track (the one that is of most interest for me) was very poor on this WWDC edition. There were a few talks about OS X Server and some of it’s new technologies, and that was it.
• There was one (yes, one) talk about Java. This is ridiculous, specially if you consider that, a few years ago, Apple was selling the Mac as the best platform for developing and deploying Java applications. The first time I was at WWDC, 2006, there were a few Java sessions with interesting content. Don’t get me wrong, I love Cocoa, but some people need decent Java support to work.
• No WebObjects content at all. The only official note about WebObjects was an announcement of an announcement that should happen within weeks. I don’t have any more details (and if I had, they would be under NDA, of course). This is sort of understandable, as they focused their attention on WOWODC, but it makes WWDC kind of pointless for people working on WebObjects projects in full time.
• They somehow managed to serve even worse food than last year. Those guys must have some special talent, I thought it could not get worse than last years, buy they crossed that barrier at full speed. Also, there was no food on monday of tuesday evening, making it a little hard to eat and get back to watch the Apple Design Awards ceremony on time.
• There were attendees distributing flyers and merchandising material inside the Moscone, despite the fact that doing that is clearly against the conference rules. Many people that attend WWDC work for some company, but a large part of the attendees own their own business, and everyone would love to promote it. But this is not a consumer exhibition, it’s a developer conference, with well defined rules, and people must respect them. What pissed me even more was the fact that the same flyers were distributed every day, by the same people, and no one from security kicked those guys out. It kinda looked like Portugal, where people break the laws with no apparent consequences.
• Moscone center was at the limit of it’s capacity, and that showed. Although they could arrange the room sizes and session schedule way better than last year (I could attend all the sessions I wanted, which didn’t happen last year due to some of the rooms being full), there were not enough seats (either on the round tables, or the sofas) on the halls. On the first days it was really hard to find a place to rest for a while, or to write some lines of code.

I think Apple should seriously consider moving to a larger space, or breaking the WWDC in two separate conferences. Maybe one for newbies and another one for advanced users, or one for iPhone OS and another one for Mac and IT. I believe the WWDC is becoming less interesting every year, and the numbers back me up.

Apple claims that 60% of the attendees at WWDC 09 were first timers. Given that they sold out in 2008 and 09, and the Moscone capacity was the same on both events (around 5200 attendees), clearly a lot of people that came to 08 didn’t find it very interesting to come back this year. The fact that they sold out is based on the iPhone SDK hype, but people seem to be not coming back on the next year. I know I won’t, unless something deeply changes about WWDC.

I could not finish, of course, without congratulating João Pavão and the Sofa team for their well-deserved Apple Design Award on Versions, their SVN client. Really good job, guys! :)

The hardware upgrades on the MacBook Pros were cool. First, Apple seemed to acknowledge that the difference between the unibody MacBook and the MacBook Pro was not strong enough, so the MacBook is dead, long live MacBook Pro. This means the 13 inch laptop has now some of the Pro features, namely Firewire was brought back, in it’s 800 Mbps version. Other nice stuff, like the non-removable high capacity battery and the back-lighted keyboard made their way to the MacBook Pro 13”. The usual CPU speed bumps were there, of course. The only decision that left people a bit confused was the introduction of the SD-card reader. Not only it seems useless (because you may plug any SD-card enabled device to your Mac using USB) but because it kicked out some old features on the Macs that got it. The Express card slot went away on the MacBook 15”, and the Audio in port on the MacBook (now Pro) 13” was replaced by an Audio in/out jack found on the iPhones. Although you can buy an Apple headset to plug there, there’s an obvious question not yet answered, how to record audio from the line in?

The MacBook Air also got some CPU upgrades, and very interesting price cuts. Finally they are not ridiculously overcharging (at least, not as much as they did) the MacBook Air form factor, despite the fact that the machine specs were disappointing. I think they should ship them with 4 GB of RAM instead of 2 (specially because it’s not upgradable), but despite that it’s becoming a really interesting machine to buy if you travel a lot. The only feature I miss (and I was hoping to find it on this new machines) is a 3G modem, which would avoid carrying one more USB dongle. But they may be because of…

… the iPhone. Apple surprised me with the new iPhone 3GS and the 3.0 version of the OS. Despite the ugly name, the 3GS is a really nice improvement to the old model, and the differences are in the details. Among all, we have:

• (3GS only) Hardware encryption of all your data. That’s really cool on a device that can easily be lost or stolen. This helps keeping your data private, but it’s not all.
• Find my iPhone. Every mobile phone user went trough this at least once. You look for your iPhone, but it seems to be nowhere. If it’s somewhere around your house, you just grab the landline phone, call it, and follow the ringtone. But sometimes, no ringtone will sound, and that’s the time you start panicking. Did you left it in the car? Or maybe in a restaurant? Or, worse, someone stole it without you even noticing? Now, if you subscribe MobileMe, you can simply access the service, and ask for the current localization of the phone. If it’s turned on, the iPhone will report it’s current position (based on the usual localization facilites, like GPS or WiFi triangulation) that will be presented on a map. But there’s more: you may ask the phone to make some noise, and display a message with your landline phone number and address, and with luck, someone will pick it up and return it to you, or deliver it to a local police station. There’s the nice detail that the phone will make noise even if it’s set to silent. But, even better, if your phone is lost forever, you can still keep your data private. Just beam it an order to wipe all the data on it, and the phone will start erasing your stuff. If it’s encrypted, the wipe is immediate. Later, if you get back your iphone, just plug it to your Mac, and the content will be restored from your backups (which can also be encrypted, for extra protection).
• Tethering. This one is big, at least for me. iPhone will share it’s 3G connection either trough the USB cable or Bluetooth. This is awesome, allowing you to pay only one 3G subscription instead of two (one for the phone, and one for the laptop). Also, no more crappy 3G modems with questionable quality drivers.

There are also some nice details, like the faster CPU on the 3GS, MMS support, better camera, etc. The iPhone is becoming more and more interesting, and I may find myself finally getting one, if the portuguese operators manage to sell them for decent (or not very indecent) prices.

Some time ago, that same page started to be the target of some bug reports stating that users would occasionally get a blank page when accessing it, and the only way to get the page back was to hit reload. This was happening on Firefox 3.0.7 and newer.

As I wrote on the previous post, our page does an RPC call while loading the header scripts, like this:

<script src="/some/path/prototype.js"></script>
<script src="/some/path/effects.js"></script>
<script src="/some/path/wonder.js"></script>
<script src="/some/path/english.js"></script>
<script src="/some/path/combo.js"></script>
<script src="/some/path/feedback.js"></script>
<script src="/some/path/jsonrpc.js"></script>
<script type="text/javascript">
var jason = new JSONRpcClient("proxyUrl");
</script>
<script src="/some/path/comboboxes.js"></script>
...

After poking around with Firebug, I discovered that all the code below the call was being ignored by the Firefox parser. That would naturally result in an empty body, which lead to a blank page.

I started googling around, and I found two interesting Firefox bug reports, 444322 and 478277.

The first of those bugs mentions the original Javascript evaluating order issue I wrote about before (which was not, in fact, a JavaScript evaluating order, but an issue where the parser would not wait for the result of the RPC call to continue parsing). They also mention the Firefox team released a “fix” for it in 3.0.6, but people kept reporting the issue was not yet fixed (some claimed it in fact got worse), which lead to opening the second bug report. Also, an important fact about this bug is that it seems to happen only when all the page resources are already cached locally by the browser. This includes all the resources loaded in the page header (usually, JavaScript and CSS files).

A few interesting comments (#30 and #34) clarify what’s causing this, and mentions this is a piece of “fragile code”. I don’t know enough about the Firefox code base (which is a nice way to say I know nothing at all) to be sure about this, but I believe the “fix” caused an even bigger problem, where, in some conditions, the parser will enter a state where it will simply eat all the input without parsing it. This leads to the blank page problem.

Well, this is all very interesting, but I had a problem that needed to be solved. So, I hacked. On of the things that avoid this bug to be triggered is at least one of the page resources to not exist on the local cache when the page is loaded. So, I forced this situation to happen. I picked up a small JS file that is loaded before the Ajax call, and configured Apache to add all the necessary headers for browsers to not cache it. This is done with something like this:

<Files "somefile.js">
	Header set Cache-Control: "private, pre-check=0,
post-check=0, max-age=0"
	Header set Expires: 0
	Header set Pragma: no-cache
</Files>

This will prevent browsers from caching the file. Remember to to this on a small file, or you’ll be wasting bandwidth and causing the page load time to increase.

Now if only someone would really fix that bug… :)

I had the problem described in the Apache 2.2 adaptor: SSL fails on FreeBSD amd64 thread in the WebObjects-Deploy mailing list. As the thread suggests, the problem is not related to 64 bits at all, but with a problem related to the SSL client certificate becoming corrupted somewhere between an SSL request entering apache and leaving to the selected WebObjects application instance.

The thread suggests a little hack in the Wonder’s Adaptor code to ignore the SSL client certificate stuff, but I found that to be a little too dirty. So I dug this as much as time allowed.

I found the adaptor will forward all the SSL http headers it has to the application. This allows the application to analyse the SSL information (like the certificates themselves, the certificate information, etc) if, for some reason, that’s needed. However, the SSL client certificate information is becoming corrupted, and screws up the request, making it invalid. As the application will not reply to what it believes to be an invalid http request, the adaptor thinks the application is non-responsive, and considers it invalid for some minutes. That’s why the user starts to get the infamous “No instance available” message.

I still don’t know if this is an Apache, mod_ssl, WO Adaptor or whatever bug. But I know how to work around it, avoiding this error. As the problem is related to providing the SSL information to the adaptor, I chose to simply block it. This can be done in the Apache configuration. Just add this to the right context (usually the SSL virtual host configuration block):


SSLOptions -FakeBasicAuth -ExportCertData -StrictRequire -StdEnvVars


This will hide all the SSL information from the adaptor, avoiding the error. I’ll try to dig this a little more if I have some free time and update this post.

• Please fix the wireless driver that causes my Mac to crash about 10% of the times I turn Airport off.
• Please fix the trackpad driver, or whatever is causing the trackpad to behave strongly erratic during about 30 seconds after waking the Mac up.
• Please fix the damn copy/paste bug that makes the paste command paste the previously copied object and not the most recent one. This is specially irritating when you cut a piece of text, paste and you realize you are pasting something else, and that your supposedly cut piece of text is lost forever, unless you can undo and get it back.
• Please fix the irritating bug that causes an iChat window to keep being the active one even after I click Safari, making its window go in front of iChat’s. That’s specially annoying when I type apple-W to close the Safari window, and the ichat one goes away.
• Please provide replacement keyboards for people who has pre-unibody MacBook Pros that, you know, actually sense a keystroke every time the key goes all the way down, without the need to almost punch the key.
• Please fix whatever is causing my father’s MacBook Pro to keep waking up and going back to sleep when the lid is closed and the charger on, despite I had already turned off every god damn thing that could wake it up, including the lid open event.
• Speaking about the charger, please provide chargers where the charge light doesn’t go off for some unknown reason. It still works, but it doesn’t inspire a lot of confidence in it ans it’s safety.
• Please provide granular updates to Mac OS X Server. Please please please pretty please.
• Please care a little more about the entreprise and IT markets, namely your own web application technology (WebObjects, of course).

Thank you, guys! You must hate me but you’re nice people anyway. Sometimes.

Facing that, and having the possibility of easily keeping a backup media offsite, I started looking for a solution. My requirements were simple: it had to be fully automatic and give me zero work, besides the obviously needed physical transportation. As I already had a backup system in place (using Retrospect and a backup disk) it was easy to conclude that a nice solution would be a RAID system with swappable drives. This way I could have two mirrored drives online, and a third one offsite. Every time I switch drives, it’s just a matter of taking out one of the drives and later inserting the one that was kept away. The RAID system will rebuild the mirror and, most important, I would have to do nothing!

After searching for a while, I was recommended the Startdom SOHORAID line of products. Don’t be fooled by the crappy site, because, as we are about to see, the product actually surprised me for it’s quality.

There are two models, the SR3610 and the SR3620. The SR3610 is actually the best one, including an LCD display for easy configuration and system monitoring. The SR3620 only has one option, USB2 + external SATA interface, while the SR3610 comes in two options, USB2 + external SATA or USB2 + Firewire 800

As the machine that is driving the system is a PowerMac G5, my first idea was to buy the firewire 800 model. But I ended up getting the USB + eSATA SR3610 option due to a mix of reasons. The first one, the price. The Firewire 800 model was much more expensive (about 70 euros). Also, facing the fact that Apple is moving away from Firewire and that the reliability of firewire controllers is not that great, I ended up opting for the USB + eSATA model. This model has an additional fan to help keeping the system cooler.

I decided that, as the RAID would be only used for backups, USB would be enough. But the Stardom came with an eSATA to SATA bracket that allowed me to turn an internal SATA port in an external one. As I would not need the internal backup disk any more, I got a free SATA port on the G5 and used this bracket to plug the RAID directly to the SATA interface. It was a nice way of adding an eSATA port to the Mac without having to buy a PCI eSATA card.

Stardom sells their RAIDs under OEM to many manufactures, including WiebeTech. As there’s already an extensive and incisive review of a WiebeTech SilverSATA II Dual Bay SATA Enclosure online (which is exactly the same product as Stardom’s) I won’t reinvent the wheel. Instead, I’ll add some points to Arthur’s great work:

Power supply

One thing that surprised me in the Stardom system was the quality of the power supply. As you can see in this Arthur’s photo, the power supply is not an external made-in-china low quality brick, but it’s built in the case itself. And those big capacitors are actually making great work there. Remember on the old Macs where you would turn off their switch, and they would take about half a second to actually power down? Well, here is the same thing. After cutting off the power, the system will still work for about the same time. This is great to avoid quick power cuts, and shows the power supply appears to be relatively immune to bad quality mains power.

Interface

The LCD panel + control buttons were actually a deception (the only one). First, the menu system is complicated and non-intuitive to use. A good example is the RAID mode, that can be switched between 0 and 1. When you get to the RAID mode menu, it will display one of the modes… that is actually the mode the system is not in. The idea is that, if you want the RAID mode to be that one, you press the ENT button. Well…

But there’s worse. The LCD display, during normal operation, displays the status of both drives. If you try to do anything else, including monitoring the system temperature of fan speed, the RAID operation will stop. The LCD will display a warning about this, and if you confirm, no disk activity will occur while you are on the menu system. This is pretty bad, because you won’t be able to constantly monitor the system health during operation.

Also, I have noticed that, sometimes during the internal controller initialization, after powering on or resetting, the LCD panel will display random characters. Despite being scary, it has no apparent side effects related to the data consistency and disk operation.

Finally, I also don’t get what the drive leds are supposed to mean. Each drive has an operation led on it. When writing data, the leds on both drives will light up. When reading, sometimes only the upper drive led lights up, and sometimes no led lights up at all. At first I thought the system was only reading from one drive, but according to the speed tests I made, both drives must be in usage, as a single disk cannot sustain the +70 MB/s I got (you can see more speed tests in Arthur’s review). So I suppose this is just a controller bug, again, without any visible harmfull effect on the data reliability or normal RAID operation.

Bottom line, this is something that Stardom needs to improve.

RAID reliability

As I intend to keep removing and inserting drives in this thing, I wanted to be sure how reliable the RAID system was. To test that, I created a RAID mirror using two disks, and copied some big video files to it. To test the consistency during the tests I describe next, I used the command line md5 tool to create a hash of the original file (on the G5′s internal drive) and the files in the RAID. As long as both hashes are the same, the files are not corrupt.

First, I did the obvious: removed a drive. A very loud “beep” started sounding from inside the box, and the status led turned yellow. I pressed the mute button to bring back the peace, and tested consistency. Everything OK. Then I inserted the drive I had just removed. The system started rebuilding the RAID. I did consistency checks during and after the RAID rebuilding process, and everything was OK.

But this is lame. The system did what is supposed to do. Big deal. Next test was a little more agressive. I removed a drive again, and reinserted it. The rebuilding operation started as expected. Then, at about 2% of the rebuilding process, I turned the system off, to simulate a power failure. As this is not exactly a high-end expensive RAID system, I expected a disaster. Well, much to my surprise, not only the system started rebuilding the RAID mirror immediately after being powered on again, as it actually knew where it was before being turned off, and resumed the process from there! Again, consistency checks were done during and after the process, and everything was fine. Thumbs up for the Stardom people!

Noise

This thing has two hard drives, a RAID controller and a power supply inside it, so it really needs to cool itself down. Cooling down means generally fans, and fans mean noise. On of the great things I felt when I upgraded from a PowerMac G4 to a G5 was how silent my office become. The PowerMac G5 has very high-quality fans that turn only as fast as needed, which means they turn very slowly and quietly most of the time. I was afraid the Stardom enclosure would bring the noise back.

To my surprise, it’s much more silent than I expected. Arthur told that in his review, and I do the same: you’ll hear mostly the air flow through the enclosure and not the fans themselves. Of course, it makes noise, and it’s clearly louder than the G5, but it’s not that bad. To get an idea of it, if you play some game on a MacBook Pro and bring its fans up to full speed, the MacBook Pro becomes much more noisier than the Stardom RAID.

Conclusion

The Stardom SOHORAID seems to be a great system, specially being cheaper than most concurrent products. It’s very reliable, and seems to be made out of quality components. It’s also fast (if used through the eSATA interface) and quiet. The only bad thing about it is the user interface, specially the fact that using it will halt the normal RAID operation.

Versions is finally out! :) João Pavão, together with the Sofa team, released their new Subversion client. It’s a really powerful application made by people who deeply understand how a Mac application should be. In a world full of dubious software, it’s good to see that some people still care a lot about their code quality and the detail. I feel honored for having been one of the few who saw this application being born and getting mature enough to be released in the wild. Congratulations, João! :)

This is a pretty old product, and it was never tested with drives this big, but the fact is that it has been working flawlessly for months now (since we upgraded the original 120 GB drives). Until last week. As I had some empty bays and unused small drives, I thought about doing a second array to store some archive stuff, as the RAID 5 array is getting pretty full.

I remembered that the case controller reboots itself when the user creates a new array, so I opted for turning off the Mac and the Promise itself, and install the disks with everything shut down. You know, this thing has hot swap, but it’s getting old, and we don’t want to push it too far. So let’s play it safe. Big mistake. Big big mistake.

I pop in the drives, press the power switch. Controller boots itself up and, as Steve Jobs would say, boom! The first two drives had the red led of death glowing. I spent a quite dramatic few minutes looking at the damn thing, and thinking that it happened. The least likely, the most feared of all things that could happen on a RAID 5 system had just happened. Two drives failed. At the same time. All the backups, some of them more than one year old, lost. Forever.

I got rational again and thought, no, this can’t happen, this thing did not boot correctly, there is something with the drives I’ve just inserted that is screwing this up. I powered down the case, removed the drives I had just inserted, and powered on again, this time carefully watching all the lights and bells. The Promise RAID, when booting, scans all the bays to see what’s going on there. You can see that happening by watching the drive lights, all of them blink quickly in a slow sequence. Well, the first two weren’t blinking.

I thought, hum, bigger drives then expected, too much time to spin up, the controller is testing them too soon. I powered off the damn thing, removed the first two drives, and reinserted them on lower bays. Another big mistake. The array disappeared. It was lost forever. Looks like the drive position is crucial for the arrays to be recognized. I had just killed what was left of it.

After some moments of desolation, I went to recreate the array again, assuming that the backups were lost I had to start from the ground up. I started the process of creating the array, and then I saw the light. On the little LCD display, the controller had the best of the words I could see on it’s first line: INITIALIZE. It allowed me to choose Yes and No. I stopped for a while, and though, if the hard drives are OK, and if I can create an array without any kind of initialization… all my data will be there! Right?

Power off, insert the drives on the original positions, power on, create array, RAID 5, default block size, initialization OFF, gigabyte boundary on, and GO. The array was created. No activity on the drives whatsoever. Perfect. Reboot. I fired up the G4 and run to the KVM console. OS took ages to boot (actually it took as long as every other time, but the adrenaline was all around). The desktop appeared and… YES! There was the RAID volume, as if nothing had ever happened. I did some quick tests, but that was it. The RAID was back in all it’s glory. Months of backups, saved.

Knowing this, I decided to push my luck a little further and turn the RAID off again. After powering up, the history repeated, first two drives were “dead”. I simply destroyed the array, created a new one without initialization, and I was back in business. Then I turned it off and quickly on again, not allowing the drives to spin down to a full stop. That time, the controller booted up correctly and the array was online.

I went to Promise site to check on this issue, and I see they had released a new firmware that announced to support some newer drives. I installed it (and it was a terrible experience, it started by having to download an older firmware to get the updater software, as Promise forgot to pack the software together with the new firmware on the ZIP archive, and ended up with an old PC with a serial cable plugged to the RAID, two floppy disks – yes, two floppy disks, and yes, we are in late 2008 -, one with DOS, another with the software, and about 30 tries – power cycles on the Promise RAID and software reloads on the PC – to get the serial communication working). After that, I powered off the RAID, waited for complete spin down, powered up again, and everything worked fine. Although I’m not trusting it fully, it looks like the problem might be solved.

So, lessons to learn: if this happens to you, 1) Do not panic (yet); 2) Do not change the order of the drives; 3) Use the LCD display and the buttons to obtain all the settings of the array (block size, gigabyte boundary status, etc); 4) Delete the array; 5) Create a new array with the same settings and initialization off. You should be off the hook by now, unless the problem WAS in fact two drives dying at the same time. Which, you know, doesn’t happen. It just can’t. Really.

One technique I use often is to create a DIV with no content or background that covers the entire page, place it on the page with z-index above the page content and below the displayed object, and bind it’s onClick event handler to a method that hides the object and the DIV itself. Something like this:

As you can see, the transparent layer (represented here in blue) gets between the object to float above the page content (here, in red) and the page itself.

So, how hard can it be, right? Just do some quick Javascript code or use CSS to set the DIV properly, handle onClick, and you are done… or not. Yes, IE. IE will screw this all up.

When I implemented this recently, I noticed that, on IE, the mouse click would go through the overlay to the page itself, clicking on links or buttons, or simply not hiding the floating object because the layer onClick was never triggered. Even weird, on some areas of the page it worked, and on other areas it didn’t. It depended on what was below it.

After gooling and trial-and-error for a while, I found the solution. The problem is that IE doesn’t like handling clicks on transparent objects, like DIVs with no content or background. So, the solution is… add a background. But wait, if you add a background, it won’t be transparent any more, right? Wrong. There’s a neat trick you can use: create a transparent GIF file with about 200*200 and use it as the background of the DIV (the size is irrelevant for this, but if it’s too small, it will make browsers on old computers slow when reproducing it to fill all the background). IE will work, because from it’s point of view, there is SOMETHING there belonging to the DIV (even if it’s a transparent GIF) and the onClick will be triggered as expected.