In many ways the Linear Tape File System (LTFS) is my baby. I was the architect of the development work and a primary author of the LTFS Format Specification. I have spent a good deal of time advocating for LTFS including my posts about LTFS on this blog.

Last year I was offered an opportunity to bring more of the LTFS story to light. Jay Livens (Director, Product & Solutions Marketing, Iron Mountain) asked me if I would record some podcasts with him about the development of LTFS. Over the course of a few months we recorded 3 interviews with each session focused on a different aspect of the LTFS development; the Creation, the Technology, and the Future.

The podcasts that came out of these interviews were published today on the Iron Mountain website.

The post LTFS Podcasts appeared first on Michael Richmond.

I’ve always felt that this was the right move for me and so far it’s been great.

Today we are announcing the company to the world…

StackStorm is an emerging leader of the third wave of operations automation. Built from the ground up with DevOps in mind, StackStorm’s vision is a world of self-driving data centers that learn over time how to better operate themselves. StackStorm solutions leverage existing configuration management and monitoring solutions to deliver automation that safely ties together today’s loosely coupled, heterogeneous, ever changing cloud infrastructures.

The post stealth startup ⥤ StackStorm appeared first on Michael Richmond.

The kit consisted of  a maxed out Powerbook 180 (Greyscale 9.8″ screen, 14MB RAM, 33MHz M68030 CPU, 56.7kbps modem) and a Nikon LS-1000 film scanner with associated cables and power supplies. The kit allowed a photographer to shoot film, process the film at a local mini-lab in the field, scan the images, and transfer them to the newspaper office. In practice this allowed comparatively rather rapid delivery of photos into the editorial process.

During my work on LTFS I messed around with a bunch of side projects on the periphery of LTFS. In one of these projects tried to devise a portable system that would support LTFS. My thinking was largely modeled on my earlier experience with the Powerbook-based field kit. I wanted to be able to give someone a portable box that could be used with a laptop and a digital camera that could be used as a videographer field unit. My idea was something that could be carried to any location, would be easy to set up, and allowed the user to dump content onto LTFS tapes and FedEx the tapes back to home base.

As a demonstration unit I assembled an enclosure that held a LTO5 drive, a SAS HBA, and a PCIe to ExpressCard bridge. This enclosure ended up having the footprint of a 15″ Powerbook and was a little taller than a LTO half-high drive. I showed the demo unit at several trade shows and other public events we did for LTFS.

LTO/LTFS portable drive at NAB 2010

While refining this portable LTFS box I played with a few different ExpressCard->PCIe bridges and some early thunderbolt->PCIe bridges. A recent question on the Mac Enterprise mailing list asked:

“One route is to go with a TB to PCI chassis then SCSI card to LTO but what little I have found says this is not reliable. Ditto TB to SAS then to SAS LTO. Anyone have any experience with setups like this and with what software?

Tape is pretty much a given. This is a situation where people need to be able to go back in time and pick up a file or folder state from 3 months ago.”

My work building this LTFS portable drive provided me with somewhat relevant experience. My answer to the mailing list may be useful for a broader audience so I’ve included it below:

My experience using both thunderbolt->PCIe and ExpressCard->PCIe bridges has largely been successful. I used the bridges with both SAS and Fibre Channel HBAs to connect LTO4 and LTO5 drives to a few different OS X machines (and some Dell and Lenovo systems). In all cases I found the ExpressCard/Thunderbolt->PCIe bridges reliable.

The few problems I encountered were always traced back to the HBAs and/or LTO drive not supporting or recognizing various sleep state events. (Or OS X not reinitializing the bus after waking up from sleep which amounts to the same thing.)

Fortunately working around this sleep issue is trivial. Just disable system sleep. Allow the monitor to go to sleep if you like, but ensure that the OS X system does not sleep.

If you are going with LTO tape in the scenario you briefly describe, you will need to work out how you want to handle the file meta-data. (Filename, timestamps, folders, permissions, etc.) Traditional tape systems maintain separate parallel databases to hold the metadata. This works, but if the database gets corrupted then all of your tapes suddenly hold garbage.

A more recent approach that is possible with LTO5 and LTO6 is to use LTFS. LTFS is a POSIX-compliant filesystem that uses LTO tape as the primary filesystem datastore. This means the file meta-data is stored on the tape right alongside the file content. As a result, the tape is self-contained and can be mounted exactly like a USB drive can be mounted. Data can be added to the LTFS volume by dragging and dropping files. Restoring data is also a drag and drop operation.

Alternatively, since LTFS is POSIX compliant applications can seek to any byte offset within a file in LTFS. In comparison, traditional systems require a parallel database, or a packaging format like “tar”. Packaging formats typically require an extensive tape traverse during restore. Whereas LTFS supports a direct seek to the data being accessed. LTFS also provides file-system versioning baked into the filesystem. The file-system versioning might be of use in your scenario.

The implementation of LTFS for OS X, Linux, and Windows is free for single tape-drive operation.

The post Bridging ExpressCard to PCIe for LTO appeared first on Michael Richmond.

After 20+ years in a closet our well loved Atari 2600 was a little worse for wear. The console has accumulated grime and the reset switch is no-longer springy. Besides cleaning off the dust and replacing a faulty reset switch I wanted to improve the video output.

The stock Atari 2600 produces an RF modulated video/audio signal as output. On old TVs the RF signal connects to the antenna-in jack. On modern TVs it is often easier to hook up a composite video input. Eliminating the RF modulation should also improve the picture quality.

A few different hardware hacks that produce composite out from an Atari are described on various websites. The output board I selected is one of the better composite output circuit designs.

Parts list

• S-video and Composite video board from electronicsentimentalities.com
• Replacement original Atari switches from best-electronics-ca.com
• Rainbow ribbon cable with header jacks from an ebay seller
• Header pins

Original console:

While dismantling I found the remains of a previous home-made repair – an elastic band to add springiness to the reset switch. The original spring seems to have gone missing over the years. While the console was dismantled I replaced the reset switch with a new old-stock part.

After installation:

The video mod board is in the lower right with the rainbow ribbon cable going from the motherboard (inside the thick aluminium RF shield box) to the video mod then a second ribbon cable running up the right side to the composite and s-video output jacks. A couple of pieces of kapton tape hold the video mod board securely in place.

New ports:

Results

Photograph of Pole Position game on Atari 2600 heavy-sixer console before hardware modification. (Photograph of LCD TV screen.)

Photograph of Pole Position game on Atari 2600 heavy-sixer console after hardware modification.

Overall the mod is a resounding success. RF noise is eliminated, image colors have better saturation, and the console can be connected to standard composite input jacks thereby eliminating the need to re-tune the TV. Installing the video mod involved bypassing the RF modulator and removing the original coax cable.

Basic testing with Pole Position and Pac-Mac during reassembly showed normal operation. I’ll update this post when I’ve tested with other games.

The post Atari 2600 hardware mod appeared first on Michael Richmond.

Requirements:

• Download FreeBSD 9.0 install disk.
• Download virtio drivers for FreeBSD 9.0 from http://people.freebsd.org/~kuriyama/virtio/

Notes:

Commands and to be performed under Linux are shown in green.
Commands and file edits to be performed under the FreeBSD guest OS are shown in purple.

Create empty disk image (10GB):

•  On your Linux host, run the command:

dd if=/dev/null of=freebsd-9_0.raw bs=1024 seek=10485760

Boot FreeBSD installer and install:

• On your Linux host, run the command:

sudo kvm -vnc 0.0.0.0:54 -drive file=freebsd-9_0.raw,if=ide -net nic -net user -cdrom FreeBSD-9.0-RELEASE-amd64-disc1.iso

• Connect to the guest OS using a VNC client pointed to port 5954. (5900 + 54 from command above.)
• Step through the FreeBSD installer:
  • allocate the whole virtual drive to FreeBSD.
  • Configure networking to use IPv4 and DHCP.
• Reboot the FreeBSD guest.

Install Virt IO drivers:

• Download the binary package from http://people.freebsd.org/~kuriyama/virtio/
• Install the package by running the command “pkg_add virtio-kmod-9-0.239473.tbz” in the guest OS.
• Edit /boot/loader.conf to add the following lines:

virtio_load="YES"
virtio_pci_load="YES"
virtio_blk_load="YES"
if_vtnet_load="YES"

• Edit /etc/fstab in the guest OS to change the device identifiers to “/dev/vtbd*“. (The following example shows the original device entries commented out followed by the virtio entries.)

# Device Mountpoint FStype Options Dump Pass#
#/dev/ada0p2 / ufs rw 1 1
#/dev/ada0p3 none swap sw 0 0
 /dev/vtbd0p2 / ufs rw 1 1
 /dev/vtbd0p3 none swap sw 0 0

• Edit /etc/rc.conf in the guest OS to add the line shown in bold. (This line configures FreeBSD to use DHCP when bringing up the “vtnet0” network interface.):

hostname="freebsd-90"
ifconfig_re0="DHCP"
 ifconfig_vtnet0="DHCP"
sshd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="NO"

• Power off the guest.
• Relaunch using virtio for the virtual hard drive and virtual network adapter using the following KVM command:

sudo kvm -vnc 0.0.0.0:54 -drive file=freebsd-9_0.raw,if=virtio -net nic,model=virtio -net user

• Verify correct hard drive and network operation in the guest OS.
• If the hard drive and network appear to be working correctly then power off the FreeBSD guest.

VirtIO in FreeBSD setup resources:

• http://people.freebsd.org/~kuriyama/virtio/
• http://www.area536.com/projects/freebsd-as-a-kvm-guest-using-virtio/
• http://kdl.nobugware.com/post/2011/10/14/freebsd-90-guest-virtio-support-in-KVM/

The post Create a FreeBSD 9.0 machine image under Linux appeared first on Michael Richmond.

The image essence is what is usually thought of as the image, i.e. it is colours and the position of those colors in the picture. When an image file is viewed, the image essence is unpacked into a two-dimensional array of color values in memory. Typically each pixel is represented as a four-value tuple: (red, green, blue, alpha). Each element in the tuple can have a value from 0 to 255 and the “alpha” value indicates the level of transparency for the pixel.

The metadata can hold information such as a description of the photo, copyright information, camera details, and other fields. Additionally, the metadata contains information about the dimensions of the image. The dimensions are represented in two different formats. These formats are referred to as the “Pixel Dimensions” and the “Document Size” in Photoshop.

The “Pixel Dimensions” is the size of the image measured in pixels in the X and Y directions. These pixel-based dimensions allow the image essence to be mapped from a linear array (such as a file) to a two-dimensional array in memory. The Pixel Dimensions size determines the size of the image when viewed in the majority of software because software is typically written to show one image pixel for every screen pixel.

The “Document Size” dimensions format consists of three values: (width, height, resolution). The width and height are be represented in Photoshop as linear units (mm, inches, etc) or as relative units (percentage of original). The resolution value is measured in pixels/inch (or pixels/cm). The Document Size dimensions are typically most important when printing an image. If the pixels/inch value matches the resolution of your printer (dots/inch) then the image on the printout will be width by height in size. However, if the resolution of your printer is twice that of your image you are likely to get a printed image that is half the height and half the width of the “Document Size” values. The reason is that that without any specific request for scaling, the printer will map one image pixel to one dot of output.

To resize an image in photoshop, open the image and select the “Image Size” tool from the “Image menu. In Photoshop CS6 this produces the dialog shown below:

Photoshop CS6 Image Size Dialog

Changing the Pixel Dimensions will change the size of the image as measured in pixels. The checkboxes at the bottom of this dialog control which numeric values are locked to the same relationship as in the original image.

Resolution values specific to the output or viewing device. Typical values for various devices are:

• old monitor: 72 pixels/inch (common old publishing assumption)
• average monitor: ~120 pixels/inch
• low-end printer: 300 dots/inch
• typical printer: 600 dots/inch
• Retina laptop: 220 pixels/inch

As with most Photoshop dialogs, holding down the option-key will change the “Cancel” button to a “Reset” button. The reset button will revert the settings in the dialog back to the original values from the image.

The Image Size dialog is best used to change the size of an image without adding or removing any image content[1. Other posts describe the related “Canvas Tool” (for adding to images without scaling) and the “Cropping Tool” (to remove parts of the image.)]. That is, the Image Size tool is used to scale an image up or down.

The post Changing the size of an image in Photoshop CS6 appeared first on Michael Richmond.

IBM LTO5 data tape media.

The Academy of Television Arts & Sciences has announced the recipients of the 63rd Primetime Emmy® Engineering Awards. This year, one of four Engineering Emmys will be awarded one of which goes to IBM for the LTFS technology.

I was responsible for the architecture, format specification, and development lead for the Linear Tape File System (LTFS) from the start of the product development. During my work on LTFS I was instrumental in producing LTFS Single Drive Edition for Linux, OS X and Windows in 2010 (all platforms released within 12 months of starting the project); producing LTFS Library Edition in 2011; and laying plans for future LTFS products up to my last day at IBM.

omgbbqwtf!! Over.

“I’d like to thank the Academy, …”

The post And the Emmy goes to…    LTFS appeared first on Michael Richmond.

I will need to come up to speed before I can talk intelligently about what Nimbula is doing and my role in the company. Right now, after a week I can only comment on the simplicity of the on-boarding process.

Mon 9:30am good morning…

Arrive and shown to my desk.
Monitor, Apple Keyboard, Magic Mouse, and 13″ MacBook Pro already hooked up.
Laptop is wired for ethernet and also already connected to company WiFi.

9:35am passwords changed, email working

Logged in.
OS X password changed.
Logged in to email.
Changed email password.
A bunch of applications are already installed… Chrome set as default browser, SIP softphone, Skype, Adium, DropBox, Evernote, VirtualBox, OpenOffice, TunnelBlick and others. Company-specific settings such as VPN and SIP are already configured.

10:10am

I have DropBox syncing with my personal account and 1Password installed waiting for DropBox to finish syncing my password vault.

10:30am more passwords changed

IT guy comes over and points me to a wiki page with a list of new hire setup instructions and starts walking me through the handful of password changes I need to make. He realizes that I have already changed my OS X and Google Apps passwords so we run through the remaining company-specific passwords.

10:45am hr paperwork

Our office admin comes by to point me at the HR web portal. Two screens later and I have completed my I-9 form by verifying the pre-populated data, entering my Alien number, typing name and the date for an esignature. Our admin hands me the 401k plan documents and points me to the benefits signup pages and some other online HR documents I should read.

11:10am making lists

I start putting together a list of account passwords to save into 1Password when DropBox finishes syncing. The list expands to other cascading setup tasks: install Office2011, extract old contacts from previous Outlook identity, investigate iCal or Outlook with Google Calendars, install XCode, setup Fink, etc, etc.

11:18am continue through the list

I paste a copy of the New Hire wiki page into TextEdit and strike off the completed tasks. Roughly 50% of tasks are already done.

12:05pm access bug tracking

I skim the current bug list after my new hire task list directs me to login to bug tracking site and verify my account is working.

12:20 lunch

Recruiter reminds me that lunch is brought in four days a week and there is a picnic table outside with a cafe umbrella in the sun.

By lunch I was largely done with the on-boarding process and had access to all the systems in use other than the development cluster. It was predominantly self-directed without the baggage of other “self-directed” training I have experienced. Throughout the morning a number of people came by to say hello and ask if I need anything.

Early afternoon my mentor, who I had met during my interviews, walked me around the office to introduce me to the people in the office. Without fail, everyone dropped what they were doing for a few minutes to say hello to the new guy.

By the end of day Friday I had finished the HR paperwork, read the Employee Handbook, largely finished configuring my laptop, arranged my desk, and have started getting up to speed on the product.

The post joining Nimbula appeared first on Michael Richmond.

In my time here, I have shipped 4 different LTFS releases, numerous LTFS fixpacks, 3 LTFS Format Specifications, the first Information Archive appliance release, several internal projects, along with multiple papers and patents. I’ve worked with some extremely talented people and have helped an industry change the way they store and manage their data.

I will be taking some personal time before starting in my new role. I will continue posting my brain dumps about LTFS to capture as much information as I can share publicly. Over time I expect to post more about the new space in which I will be working.

The post leaving IBM appeared first on Michael Richmond.

With LTFS we took a very different approach. The LTFS filesystem-based approach to accessing data stored on tape brings significant benefits to the tape market. For example, application developers no longer need to learn specialized APIs, and users get a fully self-describing storage medium that largely behaves the same as more common storage media such as hard-disk drives and memory sticks.

The benefits afforded by LTFS are arguably sufficient to support releasing LTFS as proprietary software that uses a closed data format on the tape media. Rather than treating LTFS as yet another proprietary on-tape format, the LTFS team at IBM elected to release the LTFS Single Drive Edition (LTFS-SDE) software as open-source[1. Open-source release of LTFS-SDE is available for Linux and Mac OS X. The LTFS-SDE implementation for MS Windows is freely available as a binary release. The MS Windows implementation could not be released as open-source due to licensing terms on necessary third-party components]. The LTFS open-source software release is licensed under the LGPL license to protect the LTFS format while providing application developers the ability to dynamically link with LTFS binary code without risking their own intellectual property. Each fix-pack released for LTFS-SDE has also been released under the LGPL.

In addition to the open-source software release, the LTFS team wrote the LTFS Format Specification document and released this specification to the public. This Format Specification describes the on-media data structures in sufficient detail for independent developers to read and write LTFS tapes that are compatible with other LTFS implementations.

During development of the LTFS software and the LTFS Format Specification the IBM team built a working relationship with developers at both HP and Quantum through the LTO Consortium. These relationships afforded an opportunity to inform HP and Quantum about the up-coming LTFS technology and allowed for some pre-release co-ordination between these three LTO 5 vendors. These on-going relationships have provided the opportunity to conduct inter-operablity testing between all three vendors. We are fortunate to enjoy continuing discussions between vendors on the direction and development of the LTFS format.

This communication, the open-specification, and the open-source approach to software distribution is motivated by a desire for LTFS to become an industry-wide standard. During LTFS development we recognized that if LTFS was a proprietary release then we would be adding to the fragmentation and incompatibility of data tape use. Instead, a release of LTFS as an open implementation with an open specification would create an environment that may lead to broad industry adoption. Broad adoption along with the ease of use benefits may result in expanded use of tape media for data storage. Expansion of the tape market would be good for the industry and all of us involved in the data tape business.

The post LTFS Format Specification and Open-Source appeared first on Michael Richmond.