the git model requires about 37 times the space that CVS does

and:

that’s not counting the overhead of 962 inodes and the related directory bookkeeping

He finishes off with an acknowledgement that git has data packing features:

I know that git offers some more efficient storage methods via “pack” files, but investigating those is left as an exercise to the reader.

I generally enjoy Hubert’s posts but as a daily user of git this one didn’t sit right with me. I thought I’d take up the aforementioned exercise.
I retrieved the GENERIC,v rcs file¹ and created a git repository².

I then ran a script³, which committed each revision of the file along with a single line commit message extracted from the rcs log.

The repository then weighed in at 22,352kb⁴ with 3,174 files and directories⁵. This is where git-gc comes in. From the man page, “git-gc — Cleanup unnecessary files and optimize the local repository”. After running git gc⁶ the size of the repository was down to 1,068kb, 1.24 times the rcs file. The file and directory count also vastly smaller at 64.

So all in all git fares pretty well. Sure the repository is bigger than CVS and there’s a few more files but its not in order Hubert suggests and its a small price to pay for all the benefits git provides.

________________________

1. From ftp.netbsd.org.

2.

mkdir git
cd git
git init
Initialized empty Git repository in /Users/wmoore/Source/NetBSD i386 GENERIC/git/.git/

3.

4. Repository sizes detemined via:

du -sk .

5. File and directory counts determined via:

find . | wc -l

6.

git gc
Counting objects: 2871, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (1914/1914), done.
Writing objects: 100% (2871/2871), done.
Total 2871 (delta 951), reused 0 (delta 0)
Removing duplicate objects: 100% (256/256), done.

Testing performed on Mac OS X 10.6.2 with git 1.6.4.2

The iPad is the console of computers
— mmustapic

The iPad announcement last week created a cacophony of commentary ranging from, “Meh, its just a big iPod touch”, to, “Wow, we’re witnessing the future of computing”. On announcement day I was firmly in the former camp. There appeared to be nothing new aside from the larger screen and a custom Apple SoC inside.

Since then I’ve read a lot of smart commentary on the iPad, which has swayed my opinion quite a bit.

Computing For Everyone Else

Rob Foster suggests its not the geeks that are excited about the iPad:

The darndest thing happened in the last five days and I was fortunate to be privy to it. Apple has gotten people excited about computing.

But this time, it’s not nerds or geeks and certainly not IT industry analysts. It’s everyone else.

I’m a programmer, I write software both as a profession and a hobby. On my computer I frequently use compilers, interpreters, an IDE, advanced text editors, web servers, database servers and command line tools in addition to the more common web browsers, email client, music player, video player and chat client. The iPad feels weak to me because I clearly can’t do a lot of those things on it. I am of course in a minority. For everyone else a computer that just works is all they ask. Your typical computer user shouldn’t have to know about drivers, viruses, file permissions, software installers.

The Console of Computing

Think of it as a video game console where you download the games. Now extend that thinking to all the applications.
— John Debs

I realized today that we think of the iPad as a computer, when it’s really a console. No outcries about the Wii being closed
— Cory Foy

There’s a class of computers that has been operating like Apple’s mobile platform for decades: video game consoles. The model of computing that Apple is using for the iPad, iPhone and iPod touch is very similar to that of say Nintendo. Closed systems, software approval, single main task at a time, limited expandability, ease of use, hardware abstraction are all features of both consoles and Apple’s mobile devices.

Steven Frank and Fraser Speirs suggest we’re entering the future of computing with the iPad and I now think they may be right. The iPad and its successors will make computing a bit less general but a lot more approachable for the typical consumer.

All is not lost for us tinkerers though. Just like game consoles the iPad needs software too. And whilst the iPad is not self-hosting (yet) there’s still a need for “real” computers and programmers to write the software.

A Dark Future?

Alex Payne is wary of the future that iPad style computing may bring:

For now, though, I remain disturbed. The future of personal computing that the iPad shows us is both seductive and dystopian. It’s not a future I want to bring into my home.

He makes a good point. If every computer out there were an iPad style device, totally locked down and under the control of its parent company there would be little room for exploration. However that’s never stopped people before, just look at the efforts of the iPhone jail-breaking teams or the ongoing efforts to conquer the Xbox 360. Not only that, there’s the open source movement fuelled by a desire to make everything from hardware to software totally visible and free to modify.

The products of open source projects don’t have a great reputation for having a fine attention to detail or remarkable ease of use but they exist. And just by existing they ensure that there will always be things for the budding tinkerers out there to explore, modify, extend and learn from.

Single Focus Computing

Much attention has been made of the fact that its only possible to run one app at a time. Many people seem to attributing this to a limitation of the processor used in the iPad or the iPhone OS itself. Both of these are of course totally wrong. The iPad, along with the iPhone and iPod touch are quite capable of running more that one process at a time. In fact that’s how email gets sent and received in the background and how your music keeps playing when you’re in the Facebook app. Milind Alvares attempts to clear up some of these misconceptions in, Understanding Multi-tasking on the iPad: What is it really?.

The iPad is promoting single focus computing. It does away with the memory requirements, power use, and task management that comes along with running more than one major app at a time. Just like hiding the underlying file layout simplifies things, so does this.

As the iPhone has shown us, there’s very little actual need for more than one app running at a time. As long as the experience of switching between apps is fast and state preserving it often doesn’t really matter if an app is stopped and started or just switched to. There are of some widely cited exceptions to this though: Chat clients and third party music players.

iPad doubters — remember how iPhone didn’t have COPY-PASTE for 2 years? Now compare THAT to multitasking!
— Jevgeni Kabanov

The inability to run a chat client or third party music player in the background right now does not mean that it won’t be added in the future. I’m sure Apple is well aware of these use cases, just like it proved to be well aware of the need for copy-and-paste on the iPhone.

There are solutions to both these problems that don’t require full graphical apps to be run in parallel. The music player problem would be solved if the iPad SDK permitted a non-graphical background process to be left running. The OS could even impose strict limitations on the system resources that the process may use. In fact this is already possible in Mac OS X¹, so given iPhone OS’s pedigree may already be present.

The chat client problem could be addressed in a similar fashion: A non-graphical background process with the added ability to send notifications. This is already possible with push notifications, however imagine they could be generated by the background app and perhaps also include standard user interface elements, much like third party apps in the Settings app. This way a new chat message could be displayed, with a field to allow a reply. Obviously these notifications would need to be less intrusive than they are now but it shows its possible.

Flash Must Die

RT @mashable: Apple’s iPad: Will it Improve the Web? (Cashmore/CNN)- http://bit.ly/d1UgWF flash, like IE6, must die
— Jevgeni Kabanov

From my geeky point of view Flash is a rash on the web and the quicker it just goes away the better. However I know there’s a lot of people out there that are writing the iPad off due its lack of support for Flash. The thing is, if Apple sticks to its guns and doesn’t include Flash on the iPad it can only help push the adoption of alternatives like the HTML5 canvas and video tags. Sure there’s going to be some pain until that time but we’ll all be better off at the end.

E.g. git clone hostname:path/to/repo. The solution is to be a little more specific:

git clone ssh://localhost:2222/~username/path/to/repo

After that it was time to check-in at Linden Gardens. Google Maps gave us a dud route that sent us down a few interesting roads but we made it in the end. Our lodging for the night was “Sanctuary Tower”, a three level mini-house with great views. After an in room massage we had a nice dinner at Wild Oak Restaurant just around the corner. The food was excellent and the servings very generous.

The accommodation was fantastic, right down to the graphic design of the web site and stationary. There were lots of inclusions and thoughtful extras. The one complaint was that the bedroom was quite light when trying to sleep given not all the windows had blinds.

In the morning we wandered around the gardens. The rainforest plans seem to have been planted some time ago and now it has that real rainforest feel. The paths in particular stood out as they were totally green with moss. After that it was time to go, it was a short but most enjoyable stay.

Photos of Cloudhill and Linden Gardens are on Flickr. Given the lovely weather we had I also took the opportunity to take bracketed shots to deal with the extreme range of light. I then merged them with the newly updated Bracketeer.

A selection of photos from the night are on Flickr along with some movies on Vimeo.

• Free for all 1.x customers!
• Completely rewritten from the ground up for Snow Leopard
• Faster, smarter, easier to use zapping
• New ‘Hit List’ feature lets you browse and filter all of your apps
• New ‘My Apps’ feature lets you store license info for purchased applications
• Many overall improvements

This is great news. Although it would have been nice to see some evidence the application was still alive in the 3 years between updates.

Also worth noting is another app that I mentioned in the original post, Xtorrent has also seen some recent activity with the release of Xtorrent 2 in beta. I wonder if we’ll ever see any of those amazing new features promised for Disco.

The problem with this change is that is appears to default to the light option. As previously identified in, “Consolas on Mac Update”, my preferred fixed width font Consolas doesn’t look nice with the light smoothing and I prefer medium. Fortunately you can still choose the medium option but setting the preference directly via the defaults command in the Terminal:

defaults -currentHost write -globalDomain AppleFontSmoothing -int 2

You will need to relaunch any running applications for the change to be picked up. You can do this either by quitting and relaunching or logging out and then back in again.

Before:

After:

I’ve been meaning to write a post about backup for some time now. This morning after posting a comment on Stephen’s blog about it I decided it was time to finish it off.

My backup strategy has been evolving since around 2006 when I decided that my photos were not amply secure in case of disaster. My backup solution at the time was to periodically burn a copy to CD, which I kept in my house. The obvious limitation here is that if the house is destroyed, by fire for example, all my photos are lost. What got me thinking about this originally was there actually was a major fire in one of the apartments in the block that I lived in. I was away at the time so was fortunate it didn’t spread to the other apartments.

Contents

• The iPod Solution
• ExaVault
• Amazon S3
• Backblaze
• Conclusion

The iPod Solution

I looked in to off-site backup solutions such a burning CDs and sending them to my parents and syncing to an external drive and storing it at work. Online, “cloud” solutions were too expensive at the time for the amount of data I wanted to store. My solution in the end was to buy the biggest iPod available at the time, a fifth generation, 60Gb. This had enough room to store all my music and photos as well as some other important data. It was also always on me when I went out. I figured that if a fire or something happened when I was at home I would grab it on the way out.

This worked reasonably well as iTunes synced the music and photos automatically. There was a manual step to sync the data though, which mean it was never as up to date. This was all fine until the iPod was stolen. Whilst I was more upset at the circumstances that led to it being taken I also didn’t like the idea of someone having all of my important data.

ExaVault

With the iPod gone I started using ExaVault, an online service that gives you access to a certain amount of storage via rsync for a monthly fee. This worked well for smaller datasets (ie. not photos and music) but it was too expensive to consider pushing tens of gigabytes to. ExaVault was great because I could schedule the backup to run via cron and it didn’t require any thought after that.

To mitigate the risk of losing photos I setup a Time Machine drive when Mac OS X 10.5 was released and also stored a copy of my Aperture library in a vault on an external drive. I still didn’t have a remote solution for them though.

Amazon S3

When I started a new job a coleegue mentioned that he was using Amazon S3 for backup and it was really cheap. I looked into it and found that S3 offered pricing low enough that it was financially feasible to push a copy of all my photos to the service. I considered Jungle Disk but in the end didn’t go with it because I wanted a single solution that I could use on my Linux VPS as well as my Mac. I ended up using s3sync, which is basically rsync for S3. This had the added benefit of being a drop in replacement for my ExaVault setup.

The initial upload of over 40Gb of photos to S3 took a couple of weeks I think but after that it happily ran without any drama. I also moved VPS hosts and was able to pull down configs from S3 after the old server had become inaccessible.

Whilst I had the S3 solution in place I upgraded my Mac. I bought a second drive for the new Mac, which left me with the factory supplied drive. I use this drive as a Time Machine drive and it is big enough to keep a copy of almost everything on the main drive. The few things that aren’t backed up are my Downloads folder and my Movies folder (although it does copy movies I take on my camera).

S3 Problems

After a while I noticed that s3sync was coping a few files every time it ran, even though they hadn’t changed. These files had Unicode filenames. After some research I found out that HFS+ uses decompesed character sequences in filenames. However these did not play nice with s3sync. Presumably they were stored in pre-composed format on S3 and were then seen as different when the sync was run.

s3sync also uses a questionable method to store the attributes of the folders that it syncs. Since S3 doesn’t have a concept of folders s3sync creates a file with the name of the folder and stores its attributes (permissions, owner, etc.) in it. What this ends up meaning is that s3sync is the only tool that can perform a restore. Other tools create a file with the name of a folder then try to create files in that folder but can’t because the name is already taken.

Moving to Backblaze

To save money I moved all the services I had running on my Linux VPS to my Mac at home. Since I now only had one machine that needed backing up and I didn’t need Linux compatibility I had some more options. With the S3 issues I was seeing I decided to move to Backblaze^$. For US$50 per year I get unlimited, versioned, encrypted storage and decent Mac support. They also have a web interface that can be handy for grabbing files from my Mac at home when I’m at work.

The move to Backblaze took a lot longer than I had planned. First off uploading nearly 200Gb of data was always going to take a long time but it took far longer than it needed to due to a questionable default upload limit in the Backblaze application. It wasn’t until after the two month or so initial upload finsihed that I discoved the setting. So advice to anyone performing an initial upload, remove the upload limit!

Deleting All Data in an S3 Bucket

An additional unexpected complexity in the move away from S3 was deleting the data in S3 after the Backblaze upload was complete. It isn’t possible to just delete an S3 bucket and have all its data go with it. You have to delete all the data individually. This is where S3 as a backup service started to show its limitations. Its great if you’re storing just photos or music but when you start syncing arbitrary files on a computer system you end up putting lots of little files up, some of which are quite weird. The primary example being Mac OS X Icon files, which are actually named ‘Icon\r’, yes that’s a carriage return character in the filename. I have no idea why they are named like this but attempting to delete these files gave me all sorts of grief.

I started deleting eveything using s3cmd and its ‘deleteall’ action.

S3CONF=/path/to/conf.yml s3cmd -s -v deleteall bucket-name

-v : verbose - show files
-s : SSL

This ran for a couple of days before I noticed it had got stuck in an infinite loop. It was deleting the same files over and over again. It was here that I had to step in and start deleting stuff interactively. This is when I found the awkward Icon files. I tried several S3 clients for the Mac. Cyberduck ended up being the only one capable of removing them.

• S3 Hub could see but not delete
• S3 Browser couldn’t delete
• s3cmd looked like it deleted but didn’t

Eventually I did get everything deleted but it seemed a lot harder than it should have been.

Conclusion

So that’s my backup story to date. Pretty much everything is stored on two or more hard disks plus remotely on Backblaze.

S3 is a great service but I don’t think its well suited to backup, particularly if you’re storing lots of small files. Using something like Jungle Disk would probably make it a lot more suitable though.

Backup is important and can be difficult to get right as well as keep it affordable. I believe my current setup keeps my data secure barring any worldwide catastrophes, which I think is good enough. The one one remaining thing that this strategy doesn’t cover is silent data corruption. Hopefully ZFS will become fully supported in Mac OS X soon and that will no longer be a problem.

• radiopaedia
• radiopaedia-stable
• radiant-sites

When I type rad [Tab] in my shell (zsh) it kindly completes this to radi and gives a list of the other completions since there are several to choose from. I though it would be nice if the shell could learn from the choices I make. So if I generally choose radiopaedia then rad [Tab] would complete to radiopaedia but still show the list of option.

I’m sure there are several other areas where an adaptive shell that learnt from usage patterns could be applied.

In situations like these where you end up building the package yourself I used to employ a system where I would install it into a subdirectory within my home directory. I did this because it doesn’t overwrite a pre-existing system installed version, doesn’t require root privileges and permits quick removal. To achieve this I just passed the --prefix option to the configure script before building. For example if I was building vala I might run the configure script as follows:

./configure --prefix=$HOME/Local/vala

Following that with the typical make, make install sees the package built and installed into that directory. The problem with this approach is that for every package you build you have to add the target directory to search paths such as: PATH, LD_LIBRARY_PATH, MANPATH, PKG_CONFIG_PATH. After a while this gets a bit unwieldy. This is where GNU Stow comes in.

Stow manages these separate target directories by creating symlinks into the parent directories. For example to build vala I would pass a prefix of $HOME/Local/stow/vala to the configure script. After vala is built and installed, change to the $HOME/Local/stow directory and run stow vala. This results in symlinks to the files in $HOME/Local/stow/vala being created in $HOME/Local, with directory structure preserved. So if vala installs files to bin and lib these files will end up under $HOME/Local/bin and $HOME/Local/lib respectively.

Stow can also delete the symlinks to support uninstalling. The end result is that you only need to add one extra directory to each of the search paths and custom software can be built, installed and uninstalled all without messing with the standard system and root privileges.