infrageeks: Search Results

ZFS & Leopard - not quite yet

Wed, 08 Jul 2009 10:11:08 GMT

Well it looks like the Apple implementation of ZFS will not be included in the upcoming Snow Leopard release. I admit to being somewhat disappointed that it didn't make the cut, but as I've been following the beta development, it's really not ready for prime-time wide scale implementation. They're still working hard on it internally, I and fully expect to see it out in the real world, but not before 10.7.

There are a number of rather complicated issues that they have to work through and I'm glad to see them pull it, rather than put out some kind of half-assed implementation. Notable issues that are still in the works are dealing with ZFS's not so great handling of USB volumes. Given that the vast majority of external disks used by Mac users are USB, this is a deal breaker. I've managed to kill a few mirrored zpools in my tests with unexpected disconnects and the like. Once they get a clear handle on this (which will require some more maturity of this code in the core ZFS implementation) I'll be more comfortable with this option.

The second issue that I've noted is that there's still no good consensus on how to map zpools and zfs filesystems into the current HFS oriented volume management user space. Creating a zpool creates a volume and is presented on the desktop as such, which is not exactly the intent of ZFS. The pool is the root, but not necessarily meant to be visible in the user space - it's the individual filesystems in the pool that need to be mounted. But this begs the question - where to put the pools in the meantime?

There are also some HFS and fsevent semantics that don't map over cleanly, which can result in applications not working properly (Spotlight being the number one culprit). Also some details surrounding how some of the default features would be implemented like NFS and iSCSI sharing since the necessary OS infrastructure is missing (at least for iSCSI) and the fact that these settings are part of the filesystem definitions so by moving a pool to another machine it should automatically be able to publish the filesystems according to the built-in settings. NFS should be doable, but there's still the attachment problem, on Solaris (and variants, a pool defaults to a directory at the root of the filesystem environment, while the OS X implementation maps it to /Volumes which would break NFS paths since it's not in the same place when you move a pool between environments. The lack of an iSCSI target daemon means that the shareiscsi=on settings will be ignored until Apple ports over the target code which given the reliability requirements is a non trivial task.

So I'm disappointed that it didn't make the cut but I'm also glad that they're waiting on getting it right before release.

In the meantime, my ZFS server (OpenSolaris) is quite happily serving up NFS and iSCSI volumes for Time Machine, online media, archives and overflow from my MacBook Pro's internal drive. I highly recommend it as the best possible solution for data consolidation. I even have some volumes served up to OS X Server via iSCSI, which then publishes them as Time Machine volumes for some portables on which I didn't want to install the iSCSI client. Once I get some more space (and an off-site backup solution in place) I'll be migrating all of the user home directories over to a volume mounted from the ZFS server and the OS X Server environment will be only the boot environment and essential core services.

PS - I'm using the GlobalSAN iSCSI Initiator

More on the iPhone keyboard

Wed, 08 Jul 2009 10:08:03 GMT

I just finished reading John Gruber's response to Tim Bray's commentary on the iPhone's lack of a physical keyboard. Which has inspired me to haul out my reasoning for why I think that Apple made the right decision (for Apple).

First off, keyboards are more of a habit than anything else. I used to be a Palm user and found the handwriting recognition a useful and reasonably effective means of entering information. Then I had a Newton, and despite the complaints I found that it actually worked pretty well. Then off to a series of Treos, followed by Blackberries and now the iPhone. Overall, each tool had a learning curve and I got pretty fast using each one.

But being a bit of a gadget geek I was prepared for change at each iteration so I adjusted each time. Currently the iPhone let's me type as fast as I need to and I'm easily at the same speed as my Blackberry using colleagues.

As Gruber has mentioned in another article, habits count for a lot and new users who start their smartphone life on an iPhone aren't going to miss the physical keyboard since they never built up those habits.

Something to remember here is that there is nothing natural about using any kind of keyboard. They've only existed for just over a hundred years and up until the last quarter century the QWERTY keyboard was a specialist's tool. It's a learned habit. The people I see who have the hardest time adjusting are those heavy duty Blackberry and Treo users that really haven't used anything else. Light users make the switch fairly easily.

But I digress. Apple has two other really good reasons to stick with the software keyboard:

Globalization
Logistics

We live in a polyglot world where more and more people use more than one language on a regular basis. I live in France and correspond via the iPhone in both French and English every single day. 90% of my colleagues are functionally bilingual for dealing with technical support engineers working in California (or India) and use their iPhone similarly.

Apple's ace in the hole here is that OS X (unlike all of it's competitors, except maybe the Pre) is a fully localized OS. This means that your choice of OS language is not definitive. I can decide that I want my iPhone in French one day and English the next. More important, I can switch between keyboards and the associated autocorrection dictionaries on the fly between words. On my other smartphones, my initial choice of language defined the dictionary I used from then on. I can't count how many times I sent messages with "thé" instead of "the" on my French formatted Blackberry.

The ability to switch keyboards and languages dynamically is a feature unmatched on any other smartphone OS that I know of. When you start looking further afield than the monolingual American market, you can see that this is of immense value. And I suspect at home in the US as well with the growing bilingual Hispanic population. The Chinese business market is multilingual as is most of Asia.

Obviously, this is not a feature for everyone, but I would opine that this falls right into Apple's definition of the best part of the market who are generally successful and affluent and ready to pay the so called Apple premium for something that has value to them.

Logistics
The second part may be coming from Tim Cook's side of the house in managing the supply chain. Think about the current situation. Apple has exactly 5 SKUs to manage worldwide:

iPhone 3G 8Gb (black only)
iPhone 3GS 16Gb (black & white)
iPhone 3GS 32Gb (black & white)

It's only really 3 with two different plastic backs. From a logistics and supply chain management perspective this is a huge deal. Every other physical keyboard based system has to maintain specific models by locale. The North American market is fairly homogeneous (with the exception of Quebec, but even they have adjusted to a QWERTY variant rather than the euro-French AZERTY layout). But when you go abroad you discover a plethora of keyboard variants optimized (or not, depending on your understanding of the history of keyboards) for the local language.

So you end up with models that sell only into individual countries and possibly even smaller segments (do Spanish and Catalan speakers use the same layout?). This means a lot more work on managing the production and distribution chain which translates to higher overhead costs.

Apple can manage distribution much more efficiently since high demand in one region can be met from the same production line serving the other hemisphere. That's just brilliant. Although I suspect that the major problem these days is simply producing enough iPhones period.

So Apple has both internal and user oriented reasons for wanting to keep the keyboard in software. Software gives them internal management flexibility and offers a wider feature set for their users.

And for the die hards, now that the SDK allows you to talk to hardware devices, we'll be seeing folding or clip-on external keyboard kits coming along any time now. How long will it take someone to design a battery pack with an integrated keyboard that slides out from under the iPhone like the Pre?

PS - this note written entirely on the iPhone.

iPhone 3.0 iPod podcast tidbits

Thu, 18 Jun 2009 13:52:39 GMT

There are some nice little touches tucked away in the iPod app that I'm really liking. Specifically, a few new features for podcasts and audiobooks. You can boost or slow the speed by tapping on the new speed button on the right. It rotates through 0.5x, 1x and 2x.

That's going to go a long way to helping me get through my backlog of podcasts. There's just too much interesting content out there to consume in the limited amount of time that I have available. My only quibble is that 2X can be just a tiny bit too fast for some speakers and an option to do 1.5x would be appreciated, although not critical.

The quick rollback is useful for when something interesting comes up in a podcast and you want to make a note. Tap the 30s rewind button, pop out to the home screen and open up a note or an email message and you can note the information as it goes by - another example where the 0.5 speed is really useful. Trying to note long URLs being dictated in a podcast can be a pain, and often, this happens while I'm in the train without a physical notepad easily available. Yup, I'm looking at you MacBreak Weekly Picks of the Week...

My other happy find was actually sent to me directly as Apple closed out my bug (radar://6409984) with smart playlists. Before, if you played back your podcasts from inside a smart playlist rather than the Podcast section of the iPod app, the play count wasn't incremented and they didn't autoupdate the smart playlist.

This is a great way to keep stuff in order and up to date. Start at the top of this playlist and you'll be going through all of your podcasts from all sources in the order that they were posted. As you run through them, they drop off of the list so you don't need to go checking in each individual podcast list for the latest episodes.

I am however, puzzled by the little envelope icon, as I tap on it and nothing happens. Logically it would let me send a link to the podcast to someone, but the code didn't get finished on time. Expect a 3.0.1 shortly :-)

Update - the other subtle thing is the new dynamic scrubbing speed. By scrubbig with your finger on or near the bar you are in high speed mode, but with little possibility of fine tune control. By moving your finger down into the almbum art area while moving left and right you can slow the scrub speed, allowing you to precisely navigate to a particular spot in the track very accurately. It's not a virtual click-wheel, but it works just as well.

Update 2 - the envelope does work for some podcasts, but I haven't yet been able to determine the criteria for activating the option. The other issue is that the difference between the disabled envelope and the active one is awfully hard to see. Try and spot the difference between the active link below and the disabled link in the first screen shot.

Snow Leopard Beta: Exchange integration

Tue, 16 Jun 2009 12:18:04 GMT

A quick summary of the Exchange integration in Snow Leopard since this seems to be a hot topic for lots of people. I can safely say that if you're a Mac user in an Exchange environment, you can now (finally) get rid of Entourage and replace it with the standard tools of Mail, iCal and Address Book. If you've used the iCal and Address Book integration with OS X Server you won't see anything new here other than the choice of the server. All of the UI elements and workflow are exactly the same as when you use OS X Server shared calendars and address books.

My take: nailed in one. It's exactly what I've been waiting for since seeing the ActiveSync integration on the iPhone.

This set of tests were done using a non-admin account which seems to have slowed the initial auto configuration down a little bit compared to when I first tried with a standard account.

The basic configuration steps begin in the Mail application with a new account. If possible, it will automatically select the Exchange server, or if it can't you'll have the option to select Exchange 2007 as the account type and enter the appropriate information.

Once selected it will automatically synchronize the contents of your mail, iCal and Personal address book from the Exchange server. The Global Address List is not downloaded, but is searchable as long as you have a connection to the server.

Exchange in blue points to your Exchange Global Address List address book, Exchange in brown is your personal address book, On My Mac is the local address book (and potentially MobileMe synchronized) and of course you can drag and drop between the various lists.

You'll see the new calendars in the iCal interface in a subsection with the name of the account, very similar to the way that iCal handles shared iCal calendars with OS X Server or other online calendars. Nice touches are that you can drag and drop meetings between accounts, which is nice when you forget to select the right account when adding things on the iPhone (where you cannot change the calendar after creation). iCal also lets you do things that you can't do in Outlook, like option-drag a meeting to create a copy.

There are a number of iCal specific options that you manage in the iCal preferences, like the external OWA server if it uses a different name from the internal address. This is also where you add in users from your Exchange organization that have given you read or read/write permissions so that you can view them directly in your iCal window, or automatically select available meeting times.

So with a user selected they appear in a new Delegates section in iCal, separate from your Exchange calendars. Like almost all Exchange integrated solutions, it's only the primary calendar that can been accessed, so (as far as I can tell) you only get the one calendar per delegate so the the two level view is a little redundant, but that may change in the future, but I don't have a multi calendar Exchange account to test against.

As far as I'm concerned this is a home run in terms of utility for Exchange users using OS X. I didn't try this with an Active Directory integrated computer to see if it was able to auto configure the Exchange account, but I expect that would probably work even better since you would inherit the Certificate Authority of the domain and perhaps even the auto identification of the server the way that Outlook does.

For many people this is the perfect mode of cohabitation between personal and work information - the data sources and storage are clearly separated, but you can consolidate the views. This means not storing personal data on the Exchange server with personal/private tags, but keeping it in your environment and (hopefully) helping you not book that meeting the day you're supposed to leave early for a birthday.

All of this requires exactly no special configuration or settings on the server side.

For this feature alone, I'd have paid the regular $129 OS upgrade fee without blinking. At $29 plus the overall performance increases and stability Snow Leopard (10.6) is an unbeatable upgrade.

Missing in action: I couldn't find any interface for managing server-side rules. Given that this is also a new feature of OS X Server, I suspect that it simply hasn't migrated down to the current beta release since they're still hammering out the UI.

Snow Leopard Beta first impressions

Tue, 16 Jun 2009 08:26:27 GMT

Very very nice. Here's a quick list of random observations

I did a clean install of the 10A380 beta build on a 16Gb USB key, testing on a 15" MacBook Pro so the overall performance is pretty sucky, but that's an issue with the key's performance rather than the actual OS or the machine.

First impressions:

The install process is pretty much unchanged from Leopard as far as I could tell, with some nice little touches like identifying immediately that the key did not use GUID style partitions so I needed to fix that before installing.

Once into the OS, I noticed a little alert icon on the battery and it was advising me that my battery probably need to be replaced which is very true and has been on my to do list for a couple of months as the retained charge is way down.

The usual post-install grind where mdworker/Spotlight index the entire hard drive (which is kind of painful on a USB key).

Mail/Exchange

The Exchange integration is flawless. I plugged in my name, address and password and it automatically found the proper Exchange server, complained about our unsigned certificates (normal) and offered the choice of integrating the Address Book and iCal. Some nice updated integrated shortcuts: Command-Shift-Backspace for purging deleted messages in all accounts for example. I did have some issues with the fact that the internal and external servers names for my office account aren't the same and I had to fall back onto a VPN connection to access Exchange, but that's not a huge problem.

IMAP Servers are just as easy, and less knowledgeable intervention is required, with SSL being the preferred connection, falling back to unsecured if it's not available.

Screen saver security

Another nice little touch - you can now specify the amount of time after the screen saver starts before it requires you to enter your password. Just a useful little feature so that you can quickly stop the screen saver within a reasonable amount of time before requiring the password.

Firewall

Still off by default - bad Apple.

Dock

The Exposé integrated dock functions don't seem to be working for me and I haven't found an option to activate/disable the feature. Click and hold gives the usual pop up menu with a list of windows.

MobileMe

Subtle little encouragement to sign up for MobileMe: There is an iDisk icon in the dock and clicking on it asks you if you want to configure MobileMe, taking you to the Control Panel.

Terminal

New default font! Alas, monaco, we knew ye well.

That's going to take some getting used to :-) Us old timers are pretty used to Monaco.

64bit (almost) all the time

Pretty complete coverage with Intel 64-bit being the standard for just about everything that comes with the OS. Still mulling over the kerneltask entry that is just plain old Intel though.

General notes

Previews and stuff are definitely much snappier. The new Stacks navigation is pretty and effective.

It looks like the current Preferences Application has changed a bit and (currently) refuses to load third party (or perhaps it's because they're 32bit) PrefPanes which currently means no iSCSI for me. You definitely notice that some applications launch at astounding speeds, Activity Monitor being one that stood out, especially since I'm running off a USB key. No bounces and the screen full of info appears practically immediately.

Some little logical design changes. The Directory Utility is no more. The user directory authentication component used for linking to Active Directory and LDAP servers is now a part of the Accounts Control Panel. The NFS mounts have migrated to a menu item in the Disk Utility which is better than the Directory Utility, but still not ideal since we're talking about network shares rather than disks. Why don't we have a simple NFS utility?

Overall, the whole environment feels faster (which was part of the point), but it's extremely IO bound. Running on the USB key with it's very slow IO highlights a lot of Just In Time disk fetch operations. I can really see this screaming on an SSD (anyone feel like lending me one to test?)

Not a trace of ZFS or iSCSI natively anywhere that I could find.

NFS performance is significantly better - when I browse big directory listings via NFS over Wifi it's almost like a local hard drive. Really really nice. Streaming 1080p is still a little hard on it, but most of that is an issue with the lack of local graphics horsepower and the fact that I have to scale down the raw source which is bigger than my physical screen resolution.

Perfect integration with External/Portable accounts let me log into my existing accounts, although this has confused MobileMe synchronization since it appears that the interchange formats for iCal and Address Book have changed.

Overall a solid A. If I had native iSCSI and ZFS, that would boost it to an A+, but a very stable beta release. Looking forward to september!

Useful Ubuntu commands

Thu, 11 Jun 2009 09:23:56 GMT

Need to change the keyboard layout on your console shell? Very handy when moving virtual machines around between Macs and PCs where the international keyboard layouts aren't exactly the same. Especially useful when your passwords have extended characters (and they do have extended characters, right?)

sudo dpkg-reconfigure console-setup

Restart your network after screwing around with the settings:

sudo /etc/init.d/networking restart

If you're setting up DNS servers (using bind9) with the latest Ubuntu Server (9.0.4) there's a small conflict with the apparmor security daemon that will block the reception of the transfer file on your DNS slave. You'll need to add the following line to /etc/apparmor.d/usr.sbin.named.

/etc/bind/zones/** rw,

otherwise you'll get named permission denied while writing the temp file to this directory (assuming that's where you put your zone files). Just about this line is the entry /etc/bind/** r which limits named to read only access to everything in the bind directory.

You'll know if this is the problem if you see this kind of message in syslog :

Jun 11 08:48:46 dns02 named[3099]: dumping master file: /etc/bind/zones/tmp-0u4cI0G2Mo: open: permission denied

Jun 11 08:48:46 dns02 named[3099]: transfer of 'mydomain.com/IN' from 10.1.3.22#53: failed while receiving responses: permission denied

Jun 11 08:48:46 dns02 named[3099]: transfer of 'mydomain.com/IN' from 10.1.3.22#53: Transfer completed: 0 messages, 46 records, 0 bytes, 0.001 secs (0 bytes/sec)

Jun 11 08:48:46 dns02 kernel: [ 2987.563871] type=1503 audit(1244702926.974:8): operation="inode_create" requested_mask="a::" denied_mask="a::" fsuid=104 name="/etc/bind/zones/tmp-0u4cI0G2Mo" pid=3100 profile="/usr/sbin/named"

Economics is all about value

Mon, 08 Jun 2009 13:21:09 GMT

People have to understand that basic infrastructure is not where you're going to be able to add value. An excellent Techdirt article (as they almost always are) highlights how the gas companies share a common distribution infrastructure and inject "value" at the end of the chain since the good (gasoline) is a commodity.

Are Cellphone Carriers Like Gas Stations?

I've been trying to make this point for a while now here in France where the carriers insist on using the infrastructure coverage and quality as a selling point. I use two major carriers for my personal phone (Orange) and business (SFR). Bottom line is that they are hit and miss depending on where you are. At the office, SFR has excellent reception and Orange sucks. On a client site Orange rocked out and SFR sucked. I don't particularly care that one is better than the other on any given day but is strikes me as silly that they are competing for coverage and doubling infrastructure investments. It would make a lot more sense to sign a sharing agreement and differentiate on services - especially since they'd be able to invest more in their service offerings if they didn't each have to sink so much into their independent infrastructures.

I don't care who installs the antenna in a given village. If it's Orange, then they arrange a bill-back for usage from SFR's clients who use their antenna and vice versa as applicable. I suspect that it will all mostly even out in the end and in all cases the customer gets a better experience which is all that we're asking for.

Bing fail

Wed, 03 Jun 2009 18:50:02 GMT

Well - just took Bing for a spin with the classic first search - vanity check ! And it find the usual suspects high up on the list like user profiles on various sites where I show up like VMware. Pretty good. But the first non profile type entry on the first page is for a forum entry where I was asking for some clarification about an iSCSI boot/migration tool from 2006. Not exactly current and it's a pretty exotic product so fairly low on most importance scales compared to a lot of my other forum and mailing list discussions.

And the fail? Well I clicked on the link to review what I'd written three years ago and got the following:

Uhhh - guys. If you're going to run around the internet indexing stuff, you might want to make sure that they're still online. Especially if it's front page on a search result when I know there's a packet of more current stuff out there. Sorry Microsoft, I think I'll be sticking to Google for the moment.

RE: Google fires back at VMware about virtualization for cloud computing

Wed, 29 Apr 2009 14:35:47 GMT

Google fires back at VMware about virtualization for cloud computing: "

It’s not a secret that Google is not crazy about hardware virtualization. They made it clear in June 2007, when the engineer André Barroso said:

'I think it will be very sad if we need to use virtualization,' he said. 'It is hard to claim we will never use it, but we don't really use it today.'...

(Via virtualization.info.)

Ummm - at the risk of stating the obvious, this whole discussion is only a result of the current fuzzy state of affairs around what constitutes "cloud computing". Google's business is built around building an infrastructure to run a relatively small number of homogeneous applications that can be dispersed across what is basically a supercomputing cluster in such a manner that redundancy is a function of the application.

VMware's cloud architecture is based on enabling people to run a massive variety of applications of a wide variety of heterogeneous operating system platforms.

They are both perfectly viable technical solutions to different problems. Until software vendors start writing software that is designed to run on a Google style architecture and they take the responsibility for building the necessary smarts for distributed processing and redundancy people are going to continue to virtualize their existing software stacks using a VMware style solution.

Currently I can't just take an existing application stack running a local server (or servers) and convert it to a Google cloud application the way I can do a P2V operation to a VMware virtual machine. While it would be nice to have the wherewithal to rewrite the application such that it is a portable and bulletproof that's not always (and in fact relatively rarely) a viable option. Often times there are applications running on obsolete hardware and operating systems that have no source code, no author available, but just need to keep on running inside the current architecture. In these cases, the VMware approach is ideal.

On the other hand, if you are developing a new application it's worth asking the question: do I develop this as a traditional standalone application on a commodity OS or do I develop for a cloud application platform? Can the application stand alone or does it need access to internal only data stores? Google's offering is designed for a global service platform on which to develop internet scale applications which is often overkill for many applications.

Neither of these two is doing anything wrong - they are simply addressing different markets, each with different needs. I vote that we put a moratorium on the use of the phrase "cloud computing" and everyone can just step up to the plate and explain what they can and can't do, how they intend to do it and who they're target market is.

Tempest, meet teacup and have fun exploring.

Tweet of the day

Tue, 07 Apr 2009 21:26:11 GMT

Merlin Mann in form today with the following comment:

'Organizing' your email is like alphabetizing your recycling.

In our current state of ubiquitous search, he's right on the money. Now I guess I should get around to going through the 1200 messages I have in the "To File" mailbox...

ZFS Disaster Recovery POC

Thu, 02 Apr 2009 20:35:36 GMT

I've had a couple of ideas floating around in my head regarding ways to build a cheap disaster recovery storage solution based on ZFS publishing NFS volumes. I had an excuse to build a Proof of Concept setup using virtual machines and in less than an day was able to put together the following setup:

2 OpenSolaris machines with two network cards each, one on a dedicated private network used for replication and the second for publishing NFS volumes to clients. The private network isn't absolutely essential, but if you're going to be making these systems work it's a good practice.

This setup assumes that you can put your two storage systems relatively close to each other with a gigabit network between them for the replication.

The idea is to leverage ZFS on multiple layers, using the built-in iSCSI and NFS publishing protocols managed as part of the ZFS filesystem.

The environment:

Public network: 192.168.254.0/24

Private network: 192.168.253.0/24

Node 1 Public/Private IP 192.168.254.31/192.168.253.31

Node 2 Public/Private IP 192.168.254.32/192.168.253.32

Install the iSCSI packages on both servers:

pkg install -v SUNWiscsi

pkg install -v SUNWiscsitgt

Reboot the servers. I'm sure there's a better way to get the service running properly without rebooting, but after tinkering for 30 minutes, I couldn't get it working so I just restarted and they came up perfectly. Can anyone help out with that?

iscsiadm modify discovery --sendtargets enable

This enables the iSCSI Initiator to autoscan servers for available iSCSI volumes/LUNs.

The storage zpools:

Node 1

zpool create n01p01 raidz c4t1d0 c4t2d0 c4t3d0 c4t4d0 c4t5d0

zfs create -V 2g n01p01/iscsi01

zfs set shareiscsi=on n01p01/iscsi01

Node 2

zpool create n02p01 raidz c4t1d0 c4t2d0 c4t3d0 c4t4d0 c4t5d0

zfs create -V 2g n02p01/iscsi01

zfs set shareiscsi=on n02p01/iscsi01

So now we have two storage pools composed of the 5 virtual disks I had attached to the VMs. Then I created a 2Gb volume on each, published via iSCSI.

Here's where we start layering ZFS. On the primary node, I'm going to mount the iSCSI volume published from the secondary (disaster recovery node).

iscsiadm add discovery-address 192.168.253.32

Running format gives me back a list of all visible disks.

format

Searching for disks...done

AVAILABLE DISK SELECTIONS:

       0. c4t0d0 <DEFAULT cyl 1563 alt 2 hd 255 sec 63>

          /pci@0,0/pci1000,30@10/sd@0,0

       1. c4t1d0 <VMware-Virtual disk-1.0-2.00GB>

          /pci@0,0/pci1000,30@10/sd@1,0

       2. c4t2d0 <VMware-Virtual disk-1.0-2.00GB>

          /pci@0,0/pci1000,30@10/sd@2,0

       3. c4t3d0 <VMware-Virtual disk-1.0-2.00GB>

          /pci@0,0/pci1000,30@10/sd@3,0

       4. c4t4d0 <VMware-Virtual disk-1.0-2.00GB>

          /pci@0,0/pci1000,30@10/sd@4,0

       5. c4t5d0 <VMware-Virtual disk-1.0-2.00GB>

          /pci@0,0/pci1000,30@10/sd@5,0

       6. c5t600144F049D4A3D30000505688403300d0 <DEFAULT cyl 1022 alt 2 hd 128 sec 32>

          /scsi_vhci/disk@g600144f049d4a3d30000505688403300

So I can now combine the iSCSI mounted volume and the local ZFS volume into a new mirrored zpool. A neat feature of ZFS is that each zvol is available under /dev/zvol/dsk/<pool>/<zvol>. I'm not actually sure if zfs is addressing this as a block device or a file, but the next step works, so I don't really care that much.

zpool create n01m01 mirror c5t600144F049D4A3D30000505688403300d0 /dev/zvol/dsk/n01p01/iscsi01

Now that I have a new zpool, I'm going to create the filesystem for NFS publishing, share it (use appropriate settings for sharenfs and ownership)

zfs create n01m01/nfs01

zfs set sharenfs=rw n01m01/nfs01

chmod -R a+rwx /n01m01/

I mounted the new volume on my workstation just fine, copied over some files and modified them.

Now if I need to failover gracefully to the secondary node in order to do some maintance on the main node, I need to stop NFS traffic first. If you're using this as a datastore for virtual machines, shut them down or suspend them before the next steps. Obviously, if you have a catastrophic failure on the primary, the next commands won't be needed.

Node 1

ifconfig pcn0 down

zpool export n01m01

Now moving over to the secondary node, I'm going to cheat a little bit so that I don't have to change any of the configuration on my client machines by adding the public IP address for node 1 to node 2.

ifconfig pcn0 addif 192.168.254.31/24 up

Now I need to be able to mount the zpool on node 2 so that I can start proposing it to clients. For this I'm going to use iSCSI in loopback mode.

iscsiadm add discovery-address 127.0.0.1

Using zpool import I can see the newly available pools and note that they are degraded since once half of the mirror is unavailable.

zpool import

  pool: n01m01

    id: 8068135109505309365

 state: ONLINE

status: One or more devices contains corrupted data.

action: The pool can be imported using its name or numeric identifier.

   see: http://www.sun.com/msg/ZFS-8000-4J

config:

        n01m01                                     ONLINE

          mirror                                   ONLINE

            c5t600144F049D4A3D30000505688403300d0  ONLINE

            8224377907512275624                    UNAVAIL  corrupted data

A regular zpool import with the name imports the pool, including its NFS publishing configuration so that the moment the pool is imported, it's available to clients.

zpool import n01m01

So I have successfully failed over to my mirrored storage system and work can now continue. The total downtime is minimal for the operation.

Now where many of the traditional SAN mirroring solutions fall down is when it comes to failing back. You usually have to break the mirror agreement and promote the secondary to an independent instance if you wnat to use it. To fail back you need to set up the mirror agreement in the other direction, resynchronize everything and then do it again to promote the original primary back to it's original state.

WIth this setup, it couldn't be easier. It's not a true cluster, so you'll need to stop user access again before doing the switch with the following sequence of commands:

Node 2

ifconfig pcn0 removeif 192.168.254.31 #stop the network

zpool export n01m01

iscsiadm remove discovery-address 127.0.0.1

This stops the network connections using node 1's IP address, exports the pool cleanly, and unmounts the locally mounted ISCSI connections. Then on node 1 you need to import the pool and bring up the network after running a scrub.

zpool import n01m01

zpool scrub n01m01

ifconfig pcn0 up

The scrub isn't absolutely necessary as I managed to do this on a small dataset without running the scrub before accessing the NFS volume, but it's probably a good idea.

It's not a true high availability cluster, but after poking around the AVS documentation and shaking my head at the complexity involved, I think that in many cases I can live with a reasonably fast failover solution that could even be scripted with the addition of some basic heartbeat monitoring. Best of all it can be done with free tools and required only a few very basic commands to manage.

This is one of those really cool ideas that went from "I wonder if that would work" to a fully tested setup that I've failed over multiple times this afternoon to demonstrate to colleagues that this could work.

Now the POC is currently running inside a set of virtual machine so there's not much point in performance testing, but that's now on the to do list to see just how much of a performance hit I'm going to take with the multiple ZFS/iSCSI layers. The other issues I can see coming are if I need to reboot node 1 as part of a maintenance operation, I need a way to prevent it from automatically importing the pools currently in use by node 2. I suspect that shutting down the replication network should do the trick since I couldn't get zfs to mount the zpool based on the zvol in /dev/zvol unless there was a device in /dev/dsk.

Other associated thoughts: I can reverse the setup and do some load balancing by using node 2 as the primary node for another volume, but I think that I'm not likely to gain much since zfs will already by load balancing IO across the mirrors. However, it does offer a reduction of risk in the case of a catastrophic failure since only half of the volumes would go offline and need to be remounted on the other system.

Anyone else trying to put together a high available storage solution with these tools? Ideas, comments and suggestions are welcome.

Understanding ZFS workloads

Wed, 25 Mar 2009 15:54:35 GMT

General notes on tools and tips for understanding and analyzing ZFS backed workloads.

Problem: When will my system benefit from the addition of a dedicated ZIL (ZFS Intent Log) SSD or high speed disk?

Even if you can't afford one of the new SUN storage bays with the fancy analytics engines, dtrace is there and people are building useful scripts to help analyze the workloads. Richard Elling's zilstat script is just what you need to understand what your system is actually doing.

Snow Leopard Server on Fusion 2.0.2

Mon, 16 Mar 2009 23:50:56 GMT

A useful tidbit from VMware in order to get Snow Leopard Server running under Fusion.

Here are the files you'll need in case their ftp server auto expires the files...

and

Building a better NAS for use with Macs, part 3 Getting started

Mon, 16 Mar 2009 23:25:24 GMT

Well there's nothing much more to say regarding the hardware which was pretty well detailed in the first article of the series. Setting up the box presented a few different choices, like which operating system/distribution was the most appropriate to my needs.

Basically the choices boiled down to:

Solaris 10
OpenSolaris
Nexenta

Nexenta was taken off the list rapidly since the free version is limited in the amount of capacity you can use, and I didn't really see much value add for my use since I'll be doing most of the management from the command line and the ZFS commands are incredibly easy to learn and use that the web interface seems almost silly. Although for a serious production environment, the interface developed by Sun for it's Unified Storage bays is nothing short of astounding with the detail available in its real time reporting.

Between Solaris 10 and OpenSolaris it was a bit of a harder decision, but OpenSolaris won out with the more open approach, mostly to do with managing package installations and updates.

Installing OpenSolaris is a breeze and other than the issue with the network card noted in the first article, pretty much next next next. There are a couple of little points that are worth noting though.

Creating user accounts

If you choose to create user accounts during the installation process this will activate the Solaris RBAC (Role Based Account Control) mode which is massive overkill for my needs and introduces a number of small but potentially annoying issues. RBAC goes beyond the sudo model for accessing root privileges and you use pfexec for executing commands with superuser privileges. Not a big deal, but given that it's primarily a storage server on a private network, I went the simple route, using only the root account for all management. I did open the huge security hole of permitting root access via ssh (edit /etc/ssh/sshd.conf) which generally I frown upon, but for this use it's reasonable.

I chose to use ZFS as the filesystem for the boot volume in order to profit from the snapshots, and eventually setup a mirror of the boot drive.

To avoid running into the problem I had with the network card, I strongly suggest that you burn a copy of the OpenSolaris CD and run the driver check application on your target hardware before making any final decisions. It's a LiveCD type of install so you can boot off the CD, run the check and make sure that all of the necessary drivers are included in the base install package.

I did run into some issues with the motherboard and determining the boot volume. For reasons that remain mysterious, the P5K will not take the latest BIOS update and as a result it ignores my requests to set the boot drive to the PATA disk so I have to hammer on F8 at reboot time in order to tell it which drive to use at boot. But given that the server rarely requires a reboot (and ever rarer unattended) and I've got it on a UPS this is not a big deal for me.

So the basic install has 4 1Tb SATA disks, all connected to the onboard SATA ports which I discovered using the format command to list all available disk devices. Setting up the basic storage pool is simplicity itself using the zpool command.

zpool create siovale raidz c8d0 c9d0 c10d0 c11d0

Now I have my basic pool from which I will start creating zfs filesystems. I've done some testing on the environment and I could easily host user home directories on the server directly for any machine on a wired connection, but since the media server is too far away and I can't run wires, I'm limited to Wifi which is (for me) not sufficiently performant or reliable for this task.

So I created a few different filesystems shared over NFS (something that OS X supports very nicely). Creating filesystems is another painfully simple operation and does not hard allocate any of your space so that each filesystem has access to the total space in the pool. This means that you don't have to make early decisions with long term impact just at the moment where you have the least amount of reliable data.

I started with the following filesystems:

zfs create siovale/archive

zfs create siovale/media

zfs create siovale/portable

archive will be used for all of the stuff I have cluttering up various drives that I really don't need on an immediate basis but that I can't bring myself to throw away.

media will be used for backing up the media drive (more in the next article).

portable is my generic workspace for current data that I'm working on.

Sharing the filesystems can be set as a value on the pool, and inhereited by subsidiary file systems but for the moment I'm setting them manually on each share. There are a number of little things to take care of at the beginning that aren't immediately obvious. The first is that you want to set the rights on the filesystem so that when you're connected across the network you can write to the volume. This would be the old fashioned chmod command on the directory representing the filesystem. Example:

chmod a+rwx /siovale/archive

The next step is setting the NFS sharing to be on. The simplest (and least secure) approach is to set the sharenfs propery to rw with the command:

zfs set property sharenfs=rw siovale/archive

Now using NFS requires a bit of a shift of perception if you're used to sharing files via AFP or SMB. These are protocols that demand a user authentication in order to be given access to a share. NFS really is "Network File System" and as such it has a different way of looking at things. Basically the rule is that if you're on the network and the rw flag is set, anyone who can get to the share over the IP network can use it (subject the security settings or ACLs on a given file or directory). If you want to lock it down a little more (say if you let other people use your network) you can restrict access a number of different ways, but always based on the identity of the connecting computer. It's not perfect or bulletproof security, but a reasonable compromise, especially since I would normally put this on a completely private storage network. I'm not going to go through the bother of setting up private VLANs on my home network just yet, but I might later on just to see how well it works (and if the Airport Extreme switches support this or not).

If you have multiple machines on your network you also need to review how you want to handle security and sharing. When you connect from your Mac, what the NFS server looks at for determining if you have the rights to something is your UID. Now the problem you may run into is that if you take a Mac straight out of the box, the first user will always be assigned UID 501 so as a result, the server will see everyone as the same person when they connect. I'm running OS X Server with portable accounts so I manage individual UID's on all of my machines from one context but you might need to look into manually changing UIDs on user accounts if the default 501 is used on multiple machines for different people. The side issue that goes along with that is that the Solaris knows nothing about any local groups you may have assigned to simplify sharing so you'll need to recreate user accouts and groups manually on the server if you need those kind of advanced sharing options. I haven't yet looked into getting OpenSolaris to work as an LDAP client to Open Directory on the OS X Server, but theoretically that should be possible.

If you manage your own internal DNS, and use static IP addresses or DHCP reservations, you can set those names as the only computers have rights to access a share with the syntax:

zfs set property sharenfs=rw=alphaport.infrageeks.com siovale/archive

Yes the double equals sign is ugly, but that's the way it is. The basic configuration does a reverse lookup on your address and tries to match the fully qualified name of the computer to determine if you can connect or not. So your DNS had better be set up correctly.

Connecting from a Mac can be done a number of different ways. The first obvious method is simply to use the Command-K, Connect to Server... in the Finder and enter the NFS URL like so:

nfs://shemhazai/siovale/portable

which will mount the share as a new volume. However, Leopard's volume management is a little odd when it comes to NFS shares and while you'll see the volume with the correct name in the sidebar if you go poking around in the volumes directory, the mounted volume will be the name of the server with a dash number for additional mounted shares from the same server. For day to day use you might not care about this but if you intend to do any kind of scripting or automation, you want to have fixed paths that you can rely on.

But there are other ways to manage this. You can map any NFS share to a directory on the existing HFS filesystem at the command line but that's not necessarily the easiest aproach. Apple has kindly included a simple interface for managing NFS shares in the Directory Utility (found of course, in the Utilities folder)

In the course of playing around with various different setting I've run into a few conflicts which led me to create a folder in the Volumes folder with the name of the server under which I map the various shares. In the example given I was testing out the ability to map a user home directory to an NFS share under /Users which works, but this caused SuperDuper! to die when cloning the disk. However, SuperDuper ignores the contents of the /Volume directory so putting mounts in here seemed to be a better solution. For day to day access I've dragged out the manually created folder into the sidebar for direct access.

The other nice thing about these mountpoints is that Leopard will automatically mount them if the server is available. This means that you can define these mountpoints on a portable as when you're away from your network, selecting the share will just give you back an error about an alias not resolving correctly, but the moment that you're back on your network, the links are live without any intervention on your part.

Note: Snow Leopard has moved a number of things around and I haven't yet sorted out just where the NFS mounts are managed yet.

Next up: Snapshots and other backup goodies

Building a better NAS for use with Macs, part 2 - The current environment

Wed, 04 Mar 2009 17:35:46 GMT

Building a better NAS for use with Macs, part 2 - The current environment

The current setup (snapshot january 2009) consists of:

1 machine running OS X Server (MacBook Pro)
- booting off an external 500Gb USB drive
- three disks in a concatenated RAID for holding Time Machine backups (2x750Gb, 1x500Gb)

1 media server, connected to the TV in the living room (Intel Mac Mini)
- booting off an external 500Gb firewire disk
- iTunes library, including films stored on an external 1Tb firewire disk (mostly my region 1 DVD collection bought before moving to France)

1 MacBook Pro for daily use
- internal 250Gb drive
- various external portable drives
- 1 USB 250Gb drive for SuperDuper! backups (variable schedule)

The two fixed machines are linked via two Airport Extreme Base Stations, connecting over the 5Ghz channel. Both machines are connected to the base stations via ethernet.

On the office Airport Extreme, there is a 1Tb USB drive connected which is used to hold a disk image of the media drive, copied via SuperDuper. Not an ideal solution, but I didn't want to push all of that data into Time Machine and extend the existing volume since one failure on any of the disks in the concatenated RAID set renders the entire thing useless (something that keeps me up at night). So a one to one backup seemed the most appropriate solution. If the main media disk dies, I can swap out the backup, mount the disk image and be back in business quickly.

The Time Machine volumes on the server hold the backups for all machines (media drive excluded). Time Machine to a network share is a viable solution but it does present some issues over a slow wifi network, notably that you incur the additional overhead of managing the disk images. A disconnection will result in hdiutil having to rescan the entire volume at the next backup and this can take hours. So it works, but there are times I find myself swearing at it if I leave the house quickly and forget to check if there's a backup in progress. For that reason I sometimes connect directly to an ethernet connection in the office.

My MacBook Pro and the Media Mini are using OS X Server's Portable Home Directories to keep a local copy of the home directory that is synchronized with a copy on the server so I have a quick means of recovering an account on any machine should the need arise. It's a little flaky from time to time, but since 10.5.6 I've noticed a net improvement in the reliability. That's also part of the reason that the iTunes library is on a separate disk - I didn't want the PHD making a copy of everything since there's not enough disk space on the server and PHD syncing had been causing me some grief pre-10.5.6.

Additional (irrelevant) bits and pieces:
- 3 Airport Express base stations, used for managing the 802.11g network accessed by my wife's PC and our iPhones as well as distributing music around the house from the Media Mini.

So things look like this:

Unfortunately, the layout and age of the house currently preclude easily running physical ethernet cables between the base stations, but I'm working on a solution since that would enable a few other possibilities like the ability to move the user home directories to the OpenSolaris server and get rid of local disks entirely on the Media Mini. The AEBS in the living room is the older 100Mb ethernet version that will have to be replaced if I manage to figure out how to run the cables discreetly (if it's ugly it won't get through the spousal approval process).*

It's worth noting just how big a jump there is between 100Mbps and 1000Mbps ethernet. It's obvious that it's 10 times faster, but most people don't stop to evaluate exactly what that means in real life. With 100Mbps ethernet, I can saturate the connection from a portable 5400RPM USB drive since the network transfer rate maxes out around 8MB/s. Now in a lot of cases, this isn't a big deal, but it means that there's not a lot of headroom for competing access. This means that streaming an HD video at the same time as a Time Machine backup is running is likely to cause problems for both. At gigabit speeds, transfer rates max out around 88MB/s which takes several disks to saturate a link. The net result is that I can be copying my entire media library to the NAS from the current disk at about 30-35MB/s (capped by the speed of the source disk) and still have a ton of bandwidth left over for everything else going on.

On the Wifi front, the theoretical top end of 802.11n is 300Mbs which is definitely a step in the right direction, but that's under ideal conditions and there's a heck of a lot more latency in the connection that there is over a wired one. Currently the connection speed between the two bases stations fluctuates between 45Mbps and 120Mbps. In practical terms this means that the initial backup of 700+Gb of data across this link is a multi-day process.

If you're wondering just how fast your Airport connection really is, check out the neat, almost hidden feature in the Airport Utility to watch connection speeds from various clients.

Open the Airport Utility
Double click a base station from the list and wait for the screen to load completely
Click on the title for Clients

Next up: Getting started...

* This was drafted before the new Airport Base Stations were released and it looks like the new multi-band option is exactly what I've been looking for.

Building a better NAS (for use with Macs, part 1 - Introduction

Tue, 03 Mar 2009 08:41:57 GMT

I've been playing around with various flavors of Solaris, OpenSolaris and the Solaris Community Express Edition for a while now in a professional capacity, trying to see exactly how to position the new ZFS and storage management options for use in large virtualization environments.

From Enterprise to Home

The other aspect I've been looking at is how this kind of technology can be effectively used in today's digital home where the volume of digital media is growing constantly. More details on my current environment in the next article, but suffice it to say that I have about a terabyte of media that I don't want to lose, plus all of the ancillary data arround it. Scalability is a two way street - you should be able to scale up to the multi-petabyte environments as well as scale down to the small home office.

Build it myself?
I've been looking at the various NAS options out there for a while but have a really really hard time justifying the cost for the limited functionality and expandability. The closest thing to what I wanted is the latest LaCie 5 disk NAS/RAID box which is really nice but horrendously overpriced IMHO. The Drobo was my other option, but again overpriced and while I have immense respect for the work that Drobo has put into automating RAID management it still lacks a lot of the other intelligence of the ZFS stack like snapshots, iSCSI and data healing. (BTW, there's nothing that exotic about the Drobo's capabilities, it's just some RAID operations that they've automated very nicely in their controller).

Building it myself seemed to be the appropriate solution. Which leads me to...

The state of the DIY PC market

Good god, this is insane.

I have been spending an inordinate amount of time browsing the mass of options concerning all of these components. I have yet to find a site with a filtering engine to let me find a motherboard adapted to my requirements by specifying my base requirements like form factor, gigabit ethernet, at least 4 onboard SATA ports, whether or not the SATA ports are port multiplier compatible (in fact, that tidbit of information is pretty much impossible to come by on any of the specifications pages), etc. I spent hours browsing the various stores trying to understand the variations, often with missing information (LAN port is not enough detail - I need to know if it's 100Mbps or 1000Mbps, and jumbo frame support would be nice to know as well). So from a board that I found on a store site, it's off to the mfg's site for more detail, then onto the various review sites to better understand the idiosyncracies of the model, repeat ad naseaum ...

Then there's the state of the processor market. Do you have any idea just how many processors models that Intel and AMD make? Again, insane. A huge learning phase is required in order to understand the different performance profiles, socket connections, power consumption so that I can make some kind of correspondence between the prices and what I'm buying and why.

Ditto for the cases, and then the massive confusion surrounding the different clock speeds of the various components, front side bus, CPU, memory and which combinations are compatible and what the performance side effects are of combining things badly.

On the pre-built side of the house from major manufacturers like DELL and HP, you either buy a POS entry level PC which would have been passable since I could transplant it into another case or a workstation class machine (way overpriced compared to a build your own). In both cases you can't find any real detail regarding the capabilities of the motherboards. How many internal SATA ports? Power supply wattage? (sometimes), case configuration, available disk bays, what kind of PCI slots?

I finally ended up buying a PC on eBay for 350€ with the following configuration:
Case: Antec P182. A good fit for my uses as it's sufficiently large to hold a pile of disks, fairly well rated in terms of quietness.
Motherboard: ASUS P5K. A nice board with the essentials for my project, 5 onboard SATA ports (I would have preferred 6, but hey), 4 slots for memory to allow for future expansion, enough PCI slots to be able to add in more SATA cards later, and onboard gigabit ethernet.*
Memory: 2Gb on two slots, so I can upgrade without throwing away the initial investment.
Processor: Intel Core 2 Duo 2.33Ghz. A decent enough multicore processor - I don't need the fastest thing on the planet but it's definitely useful to have the multicore design as OpenSolaris and ZFS are fully capable of taking advantage of its multithreading abilities.
Graphics card: nVidia GeForce 8800GT... Massive overkill for a server, but I can always hope that Sun will implement OpenCL down the line in response to Apple implementing ZFS so it will end up being useful for something.

I re-used one 1Tb drive I had in the house and bought three new ones at 89€ each for a total investment of 617€.

I count myself lucky to have stumbled on the PC on eBay that is pretty much perfectly aligned with what I wanted to do with it. This machine is probably overkill for its immediate intended use, but I'll look into using it for other stuff eventually, but the upshot is that you can get a basic PC for less than you'd pay for a prepackaged NAS box and have considerably more power and flexibility using OpenSolaris. Granted it's bigger and uglier, but it's quiet and can be tucked into a corner.

I really wanted to use my old Shuttle, but for some reason it core dumps on OpenSolaris.

Why not wait for Snow Leopard?
Apple currently offers a read-only port of ZFS on Leopard, and a more complete port should be available in Snow Leopard, but the actual state of development shows that it will not contain all of the features that I'm looking for (notably the hybrid storage pools). The other issue is that in terms of form factor, I wanted something that I could continue to add disks into easily. My current approach of stackable external disks has been OK, but doesn't scale up when you look under the desk and see how many power cables this requires.

Next up - the current environment and how it fits in...

* Except that the onboard Attansic network card is not yet natively supported in OpenSolaris. I did find source code for the drivers, but the basic install doesn't include GCC, and since I have no network... you see where this is going. I finally did get drivers compiled in a virtual machine and then installed via USB key.

eBook find

Mon, 02 Mar 2009 08:19:47 GMT

This was an unexpected and very welcome find. Webscriptions, associated with the Speculative Fiction publisher Baen Books offers eBooks the way it should be done. Reasonable prices, a wide variety of formats for different platforms, and DRM free! Other nice touches are that your purchase history is maintained on the site and you can redownload already purchased books in any format so if you change devices or technologies, you have a way to maintain your collection.

Webscriptions is integrated with the Bookshelf iPhone application which permits direct downloads, but this is not a requirement as you can download books in any format such as .lit and open the file in Stanza and transfer it across a Wifi network to Stanza running on an iPhone.

My only complaint for the moment is that the selection is relatively limited and even though many authors are present in the catalog, it's often just a very limited subset of their complete works. Now if only there was a simple way to make authors aware of this service and let them know that this is the kind of service that many people prefer...

We've already seen this played out on the iTunes Music Store - I just hoping that book publishers come to the realization that DRM isn't helping their situation faster than the music industry.

Re: Ben Rockwood on photo archiving

Mon, 09 Feb 2009 13:19:15 GMT

After reading his article asking about photo archiving, I think the subject could easily be expanded to cover all digital data in the modern digital household. It's an issue that I've mulled over many times both in personal and professional contexts and the solutions have evolved over time.

Currently, I think that the issue of archiving in the classical sense of offloading active storage to a separate architecture is no longer truly viable. Case in point, I have some archives in a drawer on AIT-2 tape. But today, I no longer have access to an AIT drive, I have no idea if any of the current versions of Retrospect will be able to interpret the old version that I used at the time (circa 2000). My CD and DVD archives are far from stable long term storage solutions and I've recently gone back to a few of them to pull out some older scripting solutions I'd developed for LDAP data manipulations and synchronization, only to find that some of the disks have damaged files.

True archiving means a living system that includes bringing along older data up to current formats (both physical and data types) in order to ensure their usability when you need them. Which is why I no longer think that archiving in the classic sense is a viable solution going forward (caveat - this is specifically addressed at the home user space - business and industrial environments have other archival requirements). The bottom line is that taking data offline no longer seems to make a whole lot of sense to me since there is no external media source that I have sufficient confidence in to use in this manner.

However, the biggest question comes back to (as usual) "why?". The original reasons for archiving were two fold:

to reduce bulky physical media like paper to a smaller more robust and manageable media like microfilm
to provide a form of disaster recovery plan in case the original media was destroyed

Addressing the first point it would seem that all of my current options for digital archiving are in fact counterproductive. I currently store data on highly reliable hard drives that offer a staggering level of data density. At the current state of the art a 2Tb 3.5" drive stores about 5,3 Gb/cm3, and a 500Gb 2.5" drives offers an astounding 7,57 Gb/cm3. Contrasted against a 50 Gb BluRay disc in a standard jewel box, we see just over half the density at 2.7 Gb/cm3. Of course 50Gb BluRay is still not widely available yet, and my current usual media suppliers only offer 25Gb discs, lowering the available density by another factor of two. Not to mention that at current prices, the 80 25Gb discs required to archive or backup my 2Tb disk would cost me about 480 euros today. And I can't re-use or repurpose them like I can a disk.

From the robustness perspective, data stored on hard drives vs optical media has a much lower bit level loss rate in my experience. This can be further mitigated by using RAID to protect against complete drive failure (exceedingly rare for non spinning drives). Going the optical or tape media route is also going towards increasing complexity since you now have to find efficient means of indexing this offline data so that you know where you put what and hopefully not end up making multiple copies of the same data.

Previous issues surrounding digital archiving (a different beast from classical archiving) was that disk capacity was expensive relative to the other choices. Today, the problem is that IOps are expensive and capacity is cheap. Backups and archives don't need IOps - they just need lots of capacity.

The second point is probably the real driving force behind the archival question today. The issue is the potential loss of data now that all of your family memories are in digital format, losing this data has a much wider impact that losing the last few year tax returns and copies of the Christmas letter. Taking a step back, it looks to me more like a properly organised backup strategy is required than anything else.

My solution (article coming) is to ensure that my primary storage environment is a living solution that will evolve with the technology to ensure that whatever I'm doing is going to be usable if/when I need it. Currently, the absolute best solution that I've found for this is ZFS. Because ZFS is the technology responsible for actually writing to the disks, whatever I write is portable even if the disks are swapped into another type of chassis or use a different connection protocol. I've even tested creating zpools on USB connected disks, pulling the disk from the enclosure and directly connecting it to an internal SATA port and had it recognized.

For many people something like the LaCie Big Little Disk would be the perfect portable data backup/archive for active current important data. For others that also include streaming media and video in their important data, a version based on 3.5" disks should do the job nicely. rsync is the simple solution, but even better is ZFS incremental snapshot transfers with zfs send and receive. The bonus is that this one also self corrects for user error so until you fill up your destination, you keep your old snapshots around. How many people have deleted something and not noticed until after it fell out of the backup retention cycle? Longer term, it would be nice to work out automatic snapshot transfers to another trusted site (sister, brother, uncle, service provider) for a hands-off solution, but that should be complemented with the safety deposit box and manual transfers from time to time.

Given that we are all in the process of becoming digital pack rats because we don't yet have software intelligent enough to tell us when to throw things out (and we're a long ways away if the current enterprise solutions are any indication of the state of the art). We need to work from the theory that will continue to accumulate ever growing amounts of digital information in the course of our lifetimes. Which means we need an easily extensible central data store (ZFS!) that can be easily backed up (ZFS!) to multiple destinations both network and direct connected (ZFS!).

I've been having this discussion with just about everyone I know recently. Everyone's trying to figure out how to handle digital media in the home now that it's gone beyond their ability to just dump a copy to a DVD from time to time.

What I really want to see SUN do is bring their Fishworks appliances into the general public/SOHO NAS space, based on 2.5" SATA disks on a SAS backplane for practically unlimited expansion and hot swap of backup modules. Build in the smarts to punch through a standard UPnP NAT connection to a partner box to exchange ssh keys and exchange zpools. Take a shot at the Windows home server space but do it better. Currently, all of the technology exists, but you have to put all the pieces together manually.

Something like the SATAVault box, filled with 4disk RAIDZ bricks would be perfect. I'd be using the SATAVault pictured for my own storage but it's no longer being produced :-(

OpenSolaris: Notes to self

Fri, 16 Jan 2009 10:27:25 GMT

During the installation, don't create a user account or it will dump you into RBAC-land and you'll have a hell of a time getting to root if you need it.

Use pfexec instead of su to execute commands with root/administrative privileges. sudoers seems to have disappeared somewhere into the ether.

Service administration

Self-healing SMF

svcs: list all registered services

svcadm restart network/ssh:default (not sshd!)

Fusion & VMWare Tools

Stupid automounter not working under Fusion. To get the VMware Tools disk to mount:

mount -o ro -f hsfs /dev/dsk/c3t0d0p0 /media

iSCSI

root@osdev:~# zfs create -s -o shareiscsi=on -V 5G test/disk5g

cannot share 'test/disk5g': iscsitgtd failed request to share

The standard OpenSolaris CD installation does not include the iSCSI Target package (SUNWiscsitgt) by default. Have to install it manually post-installation. If you're lucky enough to have network connectivity, just use the package manager.

root@osdev:~# iscsitadm list target

-bash: iscsitadm: command not found

Adding packages from a Solaris or Solaris Express CD

pkgadd -d /media/Solaris_10/Product packagename

The trick is that you don't fill in the whole path as you'd expect. It's the path to the Product directory, space then the name of the package (case sensitive, of course).

eBook Pricing Incoherency

Tue, 23 Dec 2008 08:52:50 GMT

I think that I've made it clear that I absolutely love using the iPhone as an eBook reader. Stanza remains my reader of choice, especially given that I can load it up via the desktop application from just about any source containing textual content.

However, the range of freely available books is limited to books that are out of copyright (via Feedbooks, Project Gutenberg and similar initiatives) and I'd like to be able to buy current or in copyright books to add to the list. Recently Stanza announced support for the DRM format used by Fictionwise (also available on the site eReader.com). They also offer their own eBook reader with direct download support (a little kludgy, but passable).

The problem (perhaps only in my mind) is the fact that despite the fact we are now dealing with a purely electronic good, the pricing for eBooks is insane. A quick example is in order. I wanted to get a better idea of the thoughts and attitudes of the president-elect of the United States as I've heard that he is eloquent in both spoken and written forms (useful for a politician). I can either go the traditional route and pop over to Amazon as I've done for years or try eReader for an electronic version. But...

	Amazon	eReader.com
The Audacity of Hope	$7.99	$14.95 (sale: $14.20)
Dreams from my father	$8.97	$14.95 (sale: $14.20)

How is it possible that the price of a product that began life as an electronic document costs more to acquire in electronic form? Is it possible that Amazon's finely managed logistics are so efficient that the process of acquiring, stocking, order management, and shipping a paper book cost less than the transfer of a file that's arguably smaller than some web pages?

This makes no sense whatsoever. And I would argue is one of the direct reasons that eBook sales are tiny. I'm paying a premium of close to 100% for convenience, plus I don't have a reasonable right of resale once I've read the book. If I buy the physical book, I have at least the possibility of sending it on to someone else at the end of the day. DRM'd eBooks don't offer this option.

The price discrepancy is somewhat smaller on general release fiction books, but it's still there. Why is this?

It looks like the publishing industry is preparing to go through the same pain that the music publishing/distribution industry is going through. The delay between the two shifts is the availability of devices that make the consumption of the media approachable and useful. The MP3 player was (and remains) a simpler device to create where an eBook requires much a higher quality screen and we're still trying to find that sweet spot between screen size, portability, battery life and usability.

My other complaint is a bit of a sideline and that's the dearth of non-English eBooks. Most of the major initiatives for scanning and republishing out of copyright works seem to be US based and as a result, the number of non-English works remains pretty slim. And on the current release front, there's practically nothing. I'm especially interested in french books, and I've only found a few sites that suffer from the same price discrepancy problem and on top of that, the commercially oriented sites are still mostly living in PDF-land. Not exactly ideal for reading on an iPhone or Kindle type device.

I would posit that with devices like the Kindle and the iPhone we've reached the level of "good enough" for many people. Which means that people will start trying to find more content for their devices. If it's easier to fire up a P2P client and type in a title than to acquire a book otherwise, that's what people will do.

It's clear that the process of transforming a paper book into Book format and putting up a torrent is considerably more labour intensive that the process of converting a CD or DVD. But there are tools that can automate much of this process, and while the resulting quality is highly variable it's generally good enough for a lot of people. But I'd rather buy a reasonably priced electronic version direct from the source without OCR errors and the like.

What's it going to take to drag the publishing industry into the 21st century?