amXor

Automate The Itch - Part 1

2011-02-11T12:59:00.000-08:00

Small, single-purpose utility applications have a long lineage in computing history. They are the cornerstone of the UNIX philosophy, and are rampant on mobile platforms. They are the small building blocks in a larger task. These are apps that scratch an itch. They do things that could be done in another way, but with less time and effort.

Every Mac comes with a tool for building your own utility apps quickly and painlessly: Automator. These apps don't need to clutter your dock, but can sit in the contextual menu (right-click) whenever you need them. This series will show you how to build a few useful apps, and hopefully give you the tools to dream up your own.

The first part will deal with files and folders, then I'll dive into network tasks, and finally text. I may also throw in a bonus section on porting some command-line wizardry to Automator.

Batch Renamer

One of the most common, time consuming tasks is emailing files. You may want to rename them, compress them or convert them somehow. Automator can deal with almost any of these tasks. Say you have 1000 files named [DSC1020.jpg-DSC2020.jpg], and want them to be called [Vacation0001.jpg - Vacation1001.jpg]. Here's how Automator can help.

Open Automator and make a new "Service"
Set the service to receive files or folders in any application.
Add a "Rename Finder Items" action. (Don't make new copies of the files)
Change "Add Date & Time" to "Make Sequential".
In the actions "Options" pane, click "Show this action when workflow runs", so you can tweak the settings each time you run it.

Save it and it will show up whenever you right click any files or folders.

Resize Images

If you email photos a lot, your workflow may also include copying them and resizing them. Here's how that might look:

Create a "Service" that receives "image files", or just continue on from the previous workflow.
Add a "Copy Finder Items" action. Set it to copy to the Desktop, your Dropbox or wherever you want the new files.
Add a "Scale Images" action and set the scaling. 600-pixels is probably good for general usage. If you want to adjust this each time it's run, check the box "Options > Show this action when run"
4a. Optionally: add a "Reveal Finder Items" action to show you the files in Finder when it's complete. This one throws an error for me, even though it does the job perfectly, so it's up to you…
4b. Or: add a "New Mail Message" action to automatically add those photos to a new message in Mail. If you're feeling lucky, you can also add a "Send Outgoing Messages" action to automatically send the prepared email.

Other Handy File Actions

Use "Sort Finder Items" prior to a sequential rename, to organize them by date, kind or size.
Include "Label Finder Items" or "Set Spotlight Comments" to organize the files you run through a particular workflow.
"Create Archive" is useful if you want to zip the files before sending.
"New Alias" is a great one-action service that just takes the selected files or folders and makes a shortcut on the Desktop.
Use "Choose From List" whenever you need to confirm an action. You then have the option to "Select None" or pick and choose items before the workflow continues.

Bonus Points - Email Screen Recording

Here's a much more esoteric, but potentially very useful workflow. This one captures a screen recording, and emails it to someone.

Make a new "Application".
Add a "New Screen Capture" action.
Add a "Start Capture" action and check the box "Wait for capture to complete"
Add a "Ask For Finder Items" action, and set it to start at "Movies". This is where screen recordings are saved by default.
Add a "New Mail Message" action like before. The selected movie will get added to the body of the mail message.

This starts the recording right away, if you haven't used screen recording in QuickTime you probably want to run it once manually to set up your levels. Instant tutorials!

Hashtag Search

2011-02-09T14:11:00.000-08:00

As a sideline to one of my bigger projects, I decided to build a little webapp that would search Twitter for a hashtag. Fairly basic, but it's got a few interesting little features:

Name & profile pics link to user's profile
Inline URL's are converted to links
'More' button loads more tweets via AJAX

Source on Github

Online Version

Automator Services

2011-02-07T11:55:00.000-08:00

I keep discovering new uses for Automator.

I'm not sure when the "Services" option became available, but it's worth checking out. I had previously wrote a text-to-markdown action, which converted the clipboard contents from markdown to HTML. This was convenient, but it involved copying, running the app and then pasting. The script did this:

pbpaste | Markdown.pl | pbcopy

With services this becomes much simpler. It's now in the contextual(right-click) menu for any application that deals with text, and I also set up the keyboard shortcut CMD-ALT-SHIFT-M. It replaces the selected markdown text with the HTML. The action is even simpler; just a single command:

/usr/local/bin/Markdown.pl

If you've installed the markdown script somewhere else, you'll have to set the appropriate path. And here's a screenshot of how the input is configured.

Search and Replace

Almost any shell command can be wired into OSX this way. If the command needs special arguments for each run, just check the box: Options > Show this action when the workflow runs. I did this for Sed search and replace action. Here's the workflow script:

sed s/old/new/g

On each run you want to change old and new with what you're searching and replacing, but it doesn't have to end there. You can change the command on the fly to anything that deals with text transformation. sed /^$/d deletes any empty lines. grep andyvanee filters out any lines not containing 'andyvanee'.

View Command Output

For text commands where you don't want to replace the text, just see the results of some command, uncheck the 'Replaces Selected Text' box and add a "Choose From List" action, or maybe "New TextEdit Document" to view the results. For good measure, add a "Copy to Clipboard" action at the end, just in case you want to do something with the output of the command.

This is probably more useful for grep type commands, so I saved this one as "Grep Text" and the previous as "Sed Text", although either can be used for a wide range of commands.

Democamp 13

2011-01-19T23:00:00.000-08:00

I went to my first Democamp this evening. I thought I'd share the experience while it's still fresh in my mind.

First off, I wasn't too sure what to expect, but it was fairly close to what I thought. The community is pretty tight, and there is a good energy. There was only 5 demos, and a couple were somewhat underwhelming, but I'll start with the good:

pluggin.it presented their new app for selling stuff on fb/twitter and sharing the profits with the friends who helped you get the word out. Seems like a great idea, and it seems like they've thought it through quite well.

A guy (I forget his name!) from 3Di who works with their pureLIGHT lighting engine presented his concept game that was pretty cool. It was based on the idea of mapping the 3D engine onto curved space... Kind of hard to describe, and unfortunately his demo only covered the basic "training" level, which wasn't very curved at all, but I could definitely see the potential of a game that ran with that idea. The graphics and lighting were very top-notch for indie developer.

There was guys from deckle there. Solid looking application, but I had this creepy feeling when they demoed the feature of printing bank quality cheques. I suppose I'm just not all that starstruck with the wonders of print media.

And the rest don't really deserve links, in my humble opinion.

Afterwards, we headed to the pub and I got a chance to meet some cool people in the Edmonton developer community. Good stuff!

Rapid Game Prototyping in Quartz Composer

2011-01-07T12:32:00.000-08:00

I am a Quartz Composer junky. I love tinkering with ideas and hacking together logic gates within this kind of visual programming environment.

I think this it's the best way to teach core CS techniques that are often obscured in text-based code. Things like logic gates, the value of modularity, and application structure start to really make sense when you lay them out visually. It's like a UML diagram that functions and evaluates in real-time.

As an example, I will use a game-idea I had, and how I quickly mocked it up in Quartz Composer. It's called "Stellar Debris", and the point of the prototype is to demonstrate vertical parallax motion in a 2D game (like the background scrolling in Canabalt).

The storyline is that your character is a flying robot(spaceship?) that takes off from the surface and must make it through the layer of orbiting debris in a post-apocalyptic earth. The controls are very basic, click to takeoff, mouse back and forth to move and click to shoot.

Here's the start screen. The graphics are very rudimentary (prototype remember!).

Once you click, the character takes off and you can control his motion and fire bullets.

You'll notice the foreground has scrolled off screen already and there are blocks to shoot. The background scrolls much slower, and the sky doesn't scroll at all. It also gets darker as you ascend into the atmosphere.

Game Logic

I don't think I would want to build an entire game in Quartz Composer. But having a mockup like this is great for prototyping because most of the game logic is there in a clear modular structure, which makes it a great starting point for porting it to your programming language of choice. Here's a look at the patches involved in making this "game" happen.

Rather than coding blindly, many of the logic bits required for gameplay are already in place. Things like "the bullet needs to know the character's position at the time of firing" or "each 2D layer needs to know the absolute position and multiply it by some factor" or even "the clouds look good enough as a blurred semi-opaque sprite".

From Command-line to XCode

2010-12-20T16:41:00.000-08:00

I'm a command-line guy at heart when it comes to programming. It seems much more organic to know what every project file is doing and how each one contributes to the whole. Which files might break things, which files are optional and which files can be refactored or optimized.

"The Pragmatic Programmer" challenged this notion. They recommend diversity. If you only know and IDE, try command-line and if you know command-line, try an IDE. At first I was skeptical of this advice. I've come to know my command-line toolset fairly well, and IDE's feel like they add too much baggage and inflexibility. I've built Xcode projects before, and there is a lot of behind the scenes voodoo attached to the projects.

But, after pushing through my initial concerns, I've come to really enjoy using Xcode for my projects. Here are some command-line programmer concerns, and how to alleviate them in Xcode. Some may be obvious to those who are at home in their IDE, but they weren't to me.

Problem: I'm going to have to fill my project folder with unnecessary Xcode-specific files.
Solution: Make a workspace (eg. Documents > Web_development.xcworkspace). Add your project folders to this workspace(without copying the files).
Problem: My project is unique, won't I have to use the command line anyway?
Solution: This is where command-line knowledge really pays off. Just create a new "Scheme" and you can run scripts pre/post Build pre/post Run. For a simple HTML site, I just set Safari.app as the executable and my index.html file as the argument. The build scheme is your hook into all the command-line utilities at your disposal.
Problem: I prefer editing with [vim, emacs, ...].
Solution: Well, I can't really help you there. Personally, I like Vim, but I'm not a total fanboy. I'm quite happy using the standard OSX keybindings, with a few tweaks for one's I use a lot. Find & Replace works well. Split view and file navigation work well, and syntax-highlighting is fully configurable and works with every file I've thrown at it.
Problem: I can't manage my git(svn, whatever) repositories from there.
Solution: This was my own ignorance. I use git for all my projects, and Xcode support for git is actually pretty awesome. It automatically knows if a file is in a git repo, and puts an M beside the file if it's been modified.

All in all, I've really enjoyed programming in Xcode. It does take a small investment of time and imagination to build custom projects, but I think it's worth it.

Skratch Beta - Call for Testers

2010-12-01T22:05:00.000-08:00

I've finally got my online coloring book ready to unleash into the wild and I would like to ask you to try it out!

If you're willing I would ask that you answer all/some/none of the following questions and e-mail them to me (1andyvanee _at_ gmail _dot_ com )or just leave a comment.

What hardware/browser(s) did you try it on?
What was your overall impression of the app?
Have you used any comparable applications? If yes, what did you like/dislike about them?
Was there any features that seemed unexpected or strange to you?
If you could add one killer feature, what would it be?

Thanks guys, have fun with Skratch!

Download it Here

P.S. If you answer Internet Explorer to the first question, you probably won't get to the rest...

P.P.S. Confirmed, Absolutely NO IE SUPPORT.

Endangered Species and UX

2010-11-30T21:13:00.000-08:00

I've just picked up Gene Wolfe's collection of short stories, "Endangered Species" for a second read. I stopped after the introduction to think about it for a while. It's that good.

I'll outline his points first for context. Wolfe starts by saying that the most important aspect of a story is that it have a reader. He explains that he wrote these stories for me, the reader. He then goes on to describe me, the reader, in great detail. My hopes and fears, my character flaws, virtues, vices and deepest thoughts. He also tells me the reason he wrote a few of the stories particularly for me.

His examples are obviously generic enough to cover a lot of people, but it's truly remarkable how specifically he targets his audience. Here's an excerpt to give you the flavour.

"...At certain times you have feared that you are insane, at others that you are the only sane person in the world. You are patient, and yet eager..."

The reason I have found this so refreshing is that I've read a lot on User Experience(UX) lately. There is a fair amount of hype and self-importance in a lot of UX articles. The stated goal of UX is providing a pleasant experience for "The User", but most of the literature drives home the point that the average user is ill-tempered, ignorant and just plain stupid.

Gene Wolfe truly cares about his readers. He thinks highly of them. He also expects creative engagement and effort from his readers.

UX design seems to be focused on placating unwilling participants. I would argue that it should be focused on delighting the willing participants. Valuing them.

SVG Explorations

2010-11-25T11:01:00.000-08:00

SVG seems to have got buried under all the hype about the HTML5 canvas. But it's really quite cool and I'm surprised more sites aren't using it for dynamic content.

Part of the problem seems to be early and inconsistent implementation. A lot of the demos that are floating about the web are old, uninspiring and bland. Some good examples can be found on the raphael.js site and a decent example of a full page built with SVG is emacsformacosx.com.

I've built a small example using CSS styling and JavaScript mouseover events to randomly rotate each path in an image I "traced" using Illustrator.

Take a look. (Hint: Try to click the pink element!)

Update: Apparently Google Graphs(a spreadsheet option) are rendered with SVG. Cool!

Chaining in JavaScript

2010-11-24T23:16:00.000-08:00

It's been bugging me for a while that I didn't understand the method behind jQuery's function chaining. So I built the smallest functional version of it. I call it aQuery (for andy, or awesome).

This is the calling code:

window.onload = function(){
    aQ("#content").height().background().elem.innerHTML = "hello";
};

I have made two functions .height() and .background(), which do pretty much what their names suggest. The elem is the original object and so I use it to call "native" code. Maybe should have called it "target" to be consistent.

Here's the meat and potatoes of the "library":

(function() {   
    var aQ = function(selector) {
        return new aQuery.fn(selector);
    };

    var aQuery = {};
    aQuery.fn = function(selector) {
        this.elem = document.getElementById(selector.split("#")[1]);
        this.cssText = "";
        this.height = function(){
            this.cssText = this.cssText + "height: " + window.innerHeight + "px;";
            this.elem.style.cssText = this.cssText;
            return this;
        };
        this.background = function(){
            this.cssText = this.cssText + "background-color: #444;";
            this.elem.style.cssText = this.cssText;
            return this;
        };
        return this;
    };
    
    return (window.aQ = aQ);
})();

Each call to aQ makes a new object, primed with the selector string. This object has several functions and attributes, and all the functions return "this", so they can be chained together. As it stands, it's not dynamic at all but you can use your imagination!

jQuery Mobile Data Attributes

2010-11-17T22:08:00.000-08:00

In testing out the newly released jQuery Mobile library, I'm left with strong but mixed feelings about it.

First, it's a surprisingly unique solution. The jQuery team is very good at simplifying hard problems, but in some ways you come to expect that the solutions will be quite similar. I expected a bunch of jQuery-type functions that would transform your markup into the mobile look and feel. I expected jQueryUI for mobile, and was ready to start moaning about how inflexible and unsemantic it was.

Instead, their solution is to add HTML syntax and automatically wire up all of the functionality for you. Not what I was expecting. You don't have to write a single line of JavaScript for the default functionality. This has got to be the easiest solution for porting HTML content to a device. Your "pages" can all live inside one HTML file or be loaded dynamically with AJAX. All the URL's get dynamically referenced to the main page and resolve correctly. And no handwritten JavaScript or CSS in sight.

A lot of this functionality comes through the use of custom data attributes in your HTML. In my perusing of the HTML5 features I have to admit: this one seemed fairly boring. So what if I can add custom attributes and my page still validates? Why should I care?

But the guys at jQuery Mobile have capitalized heavily on this notion. By using custom attributes quite liberally to alter the semantic meaning of lists, headers and anchors, they can automate all the behind the scenes JavaScript and CSS necessary to make it work.

It's a strange reversal of the web technology where the withering old HTML document is front and center again and the CSS and JavaScript hide in the back room, pulling on the invisible strings.

For a "Navigate to Text Content" site, this is a dead-simple solution. I think a 'Project Gutenberg' site is at the top of my list. Or maybe an RSS reader.
Some negatives that I see:

Building or altering UI components is going to be a pain.
Really only scales to iPhone-size devices. (They support iPad, but who uses full-width buttons on iPad?)
The UI is a tad bland. (Erring on the side of caution, I guess)
I'm undecided on all of those data-role tags.
It's new and strange. I don't like new things. I'm scared of it. What if it doesn't like me?

Anyway...

My real takeaway is not the tool itself, but understanding how it works and how I can incorporate those ideas into my own projects. There's a lot of smart people working on projects like this, and you really only get to know them by reading their code.

Boxeddet - Another Wireframing Tool

2010-11-10T08:48:00.000-08:00

After reading the article on sixrevisions.com about wireframing I thought to myself: "What the world needs now is another wireframing tool!"

Yes, the options are staggering. But in a way they all seemed very similar. My feeling is that if you want to sell a tool to a designer, the primary goal is not adding new functionality but simply adding a bit of unique visual flair to make it "pop".

Last night I hacked up a bit of code for a different type of tool. This one produces actual HTML layouts in the grey-box style. You just draw your div's inside the browser, and the DOM is dynamically updated.

I've put the code up on github, so go ahead and clone/fork/prod it and let me know what you think!

Get The Code!

Or you can Test drive it!

jQuery Extensions?

2010-10-28T20:29:00.000-07:00

I've been working with jQuery a fair bit lately. Although it gives you a fair number of handy shortcuts, I think it's true value is as an educational tool on how to use JavaScript effectively.

This means diving in to the jQuery sources. This means learning what anonymous functions and closures are all about, letting it settle, and then learning what they're about again. It also means taking 98% of all the JavaScript tutorials on the internets and throwing them out. Well, maybe not throwing them out, but putting them in to context.

What I'm digging in to right now is programming in the jQuery paradigm. What this means is bolting all my own functions onto the jQuery object through an anonymous function. I can then call these functions just as I would call the jQuery functions, making for a cleaner and more reusable codebase.

The basic pattern is this:

jQuery.js

The jQuery library.

myLibrary.js

This is where I attach my functions to the $ object. The template goes like this:

(function ($) {
    var private_var = 1;
    var private_function = function() {return "Only this closure can call me, I'm special."};
    $.cool_function = function(){
      return "Hello jQuery.";
    }
}(jQuery));

doStuff.js

$().ready(function() {
    $("h4").html($.cool_function());
});

That's it in a nutshell. For more info on wrapping your brain around why this works, see this article. I wish I would have read it earlier.

You'll note i'm not handling input from jQuery selectors. I'm going to have to let that one bounce around in my brain a bit longer...

Happy coding!

Call For Web Standard

2010-10-07T19:40:00.000-07:00

Developing native apps is a pain. Yes, you have the luxury of fixed screen dimensions and access to hardware features that may not be available elsewhere, but when you're done, you have an app that only works on one platform.

Web apps, on the other hand, work on a lot more platforms straight out of the box. The performance may not be the greatest, but the technology is advancing at breakneck speed.

The single largest drawback of web apps is working with files. The browser has no access to the filesystem, which makes sense in a lot of ways, but I'm surprised that this hasn't been addressed more thoroughly.

What we need is a new web standard and API for remote filesystems. Unlike the popular storage solutions like Dropbox, Box.net and MobileMe, this would be a remote hard-drive that you and your apps have access to. Apps would need to authenticate to access zones of your storage space; you could make some zones public and other zones publicly accessible via login.

Again, this goes back to my idea of content control. I would far sooner use an app that writes to a simple, accessible file-type than one that uses a locked down and proprietary format. The same standard applies to web apps, but the current assumption is that your web-app files live and die with the service that created them. Even Google Docs, probably the best of it's breed, only operates ideally if you only access and edit the files through their service (downloading/uploading strips date, author, etc. ). Interoperability with other web apps? Well, that's just crazy talk!

Crazy!?!

But why is it crazy? What is it about web-services that makes us freely give up the rights to our data? If I downloaded a word processor that password encrypted every file so that only that app could read it, I certainly wouldn't use it! But we are so blinded by the sharing and publishing features of these apps that we throw interoperability and future-proofing out the window.

In fact, I think there's some merit to this idea even in the desktop computing world. By zoning the filesystem, you could download and run apps in a sandboxed environment without worrying that they are messing with your files.

Maybe something like this exists? It seems like the IETF group working on WebDAV has something like this in mind, but it still seems pretty system level, like Samba and afp.

Divide And Conquer

2010-09-15T22:33:00.001-07:00

School has started and I've been very busy with projects, but haven't had time to write. And in fact there's not a lot to write about. Sure, I've been working on a lot of cool stuff, but the big picture hasn't come into focus.

Along with getting a grip on the Adobe tools (Photoshop, Illustrator, Flash/AS), there's the programming.

One of the best insights that I've got so far is how powerful pencil and paper can be. Storyboarding, brainstorming and sketching use cases are a great way to think about a project from all angles. If you sit down and start coding without any clear direction, you are wasting a lot of thought on design and abstract principles, when you need to focus that brainpower on making sensible code. Working this way, a lot of coding design patterns start to emerge before you even begin coding.

And maybe that's the one bigger picture that is coming into view: learning to divide and conquer is vital to any project, even if it's just a one man show.

Progress

2010-09-06T23:06:00.001-07:00

Technology is advancing, but is it really advancing as fast as it seems?

I might come across as a bit of a Luddite here, but I think that a lot of the emerging software and hardware is ephemeral. And maybe that's the way consumer electronics has to be. In the pre-dawn of the internet, software was developed for aerospace, medicine and government. It had to be robust, provable and enduring.

The landscape has changed completely. Today, developers have embraced the ephemeral nature of apps, iPads and cell phones. They're not worried about building enduring artifacts and code that will be solid and transparent for a generation.

Perhaps hindsight is always myopic. There are plenty of criticisms that judge the UNIX philosophy as being hackish; a virus that spread the "Worse is Better" philosophy of the the open source movement in general. But today, the tools have been honed, pruned and documented so well that it all works together as an organic whole. An organic whole that can be extended or repurposed in ever new ways.

It is this sense of growth that seems lacking in the software and hardware landscape of today. Code and hardware now go into the landfill in a few years by default. Without the organic usefulness of command-line programs, standalone apps have no value beyond a very short life-cycle.

In a way, I think it's what had to happen. For most consumers, the only interface that will get used is the one that is blatantly obvious. They feel smart when the computer doesn't make them feel stupid. And so user interface must appeal to the lowest common denominator. I just feel a tiny shiver of remorse when I see the amount of energy poured into technology that will be junked and forgotten in months or years.

Git Whyto

2010-09-04T14:15:00.001-07:00

There are plenty of Git howto's on the internet, instead I think many people need a Git whyto. This guide is specifically focused on using Git and GitHub for personal projects like writing, publishing and creative work.

1. Complete Content Control

Git basically tracks any folder you tell it to. As you make changes to files, you can commit the changes, roll back the changes or create branches to try out alternate paths without messing with the main branch. You then push these changes to GitHub, which is simply a mirror of the files on-disk.

2. Responsibile Collaboration

It's easy to add others as collaborators on your project, and easy to see what, when and how much they changed. Any changes that you don't agree with are easily rolled back to previous versions.

3. Tracking Intentions

Each time you commit a change, you write a short description of the change. This helps you to get a sense of where the project is at, what you were thinking, and what still needs work.

4. Scalable Projects

Personal projects start out small, but there is a chance one of them may catch fire and become a big idea. Git works well with a single contributor or thousands of contributors. And if the project takes off, you will have the track record of the entire project right from the first commit.

5. Universal Appeal

Git was designed for software projects but it really has many compelling features for any project, from thesis to portfolio to presentation. System-wide backup plans like Time Machine are great on the large scale, Git allows the same kind of snapshots on a per-folder basis but adds the ability to merge, clone and modify these snapshots across many computers.

One slight problem with GitHub is that the free version only allows public projects (view and download).

If your project is private you can always just use git on your personal computer, zip the folder and back it up to Google Docs. You can then unzip the folder to any other computer and it will work with git exactly as it should. Or, alternatively, you could keep your master on an external hard drive and merge changes from your laptop, desktop and work computers without a problem.

Photo Archiving

2010-08-26T23:15:00.001-07:00

Backups and Archives are two different things. Ideally you should have a good archiving system, and your backup would be a redundant copy of that.

In photography, a robust archiving system has between two and four copies of any file. These copies are:

Camera Card
Working Copy
Source
Backup

In this case, Source and Backup are read-only copies that are exact duplicates of the file off the camera. The Backup copy should be on a seperate physical media and ideally in a seperate physical location.

The working copy is the one you edit, view and share. Once a file is edited, it is important to make a Source and Backup copy of them as well.

It is good to view these copies as layers of mutability.

Camera Card: Changes the most. Every shooting session will create and erase content from the card.

Working Copy: Can change with each editing session and these files can be safely erased once backups are made.

Source Copy: Should never change, it is the copy that is accessed for viewing and making working copies. Any Source copy that is deleted will be deleted forever.

Backup Copy: A mirror image of your Source copy. It will only be accessed in the rare event that your source copies are compromised (HD failure, fire, theft, etc.).

Implementation

The way to implement this structure will vary with the tools that you are using. The first step might be to disable automatic importing when you connect your camera. A better solution when connecting your camera is to make source and backup copies immediately.

If you are using DVD's as your Backup copy, it probably makes sense to use a USB thumb drive as an intermediate backup until you have enough to fill a DVD.

It is important though to do regular integrity checks on your Backup copies. You need to make sure that they aren't compromised without you knowing. If you use an external hard drive, and are command-line savvy you might do "diff -r PictureSource/ PictureBackup/" to compare the entire contents of the two folders. If you are using DVD's you either have to trust the quality of your media, or do periodic checks of your media.

Connect Camera.
Make Source copy, check file count.
Organize Source photos into events, deleting any Absolute Garbage.
Make Backup copies, check file count.
Make Working Copy from Source copy by importing into your editor of choice.

Resist the temptation to organize your Source folder too much. Use only large, time-based, linear chunks.

DO: Pete's Wedding, Reception, Saloman Bay Beach...
DON'T: Flowers, Trees, Close-ups, Saloman Bay Beach...

The attentive reader will notice that the beach name is both in the do and don't list. This is because in the first, I've assumed that it was an afternoon of shooting at the beach. In the second it is a couple shots of that subject interspersed with shots of other subjects.

On a related note, don't rename your files. The filenames from your camera are a very good shooting record of your camera. If you want to attach descriptions and tags, do it in the file's metadata.

Now you are ready to edit your files. A typical editing session might look like this.

Make a Working Copy from Source copy (if it's not already done).
Edit the file(s).
Make a new source copy next to the original (eg. DSC1255-edit1.jpg)
Make a Backup of what has changed in the Source.

If you are using DVD's, you probably want to make a new folder for any edited files, because the original folder might already be burned to disc. Again, use the original name with a standard suffix (Pete's Wedding-edited).

Generalized Strategy

Photos are actually quite easy to organize, compared to the myriad of other file types out there. But the systematic logic can be transferred to other types of files as well.

By being systematic, a lot of confusion can be avoided. It also makes it possible to work toward automating the process entirely. But as the number of file types increases, so does the tendency to use different sorting methods or mixing methods.

What The Internet Needs Now

2010-08-26T23:09:00.001-07:00

I've been blogging a fair bit lately about metadata, archiving and dealing with the masses of information that we produce and consume today.

It all comes from an idea that has been alternatively percolating and distilling in my mind. It's the idea of stability.

The internet age is highly energized and highly transient. The next big thing comes along, is adopted by huge masses of people, and is discarded in a matter of months or years. Examples abound, but my working example is waning interest in Facebook.

I recently printed a PDF of my entire facebook history because I'm a dork and find that type of thing interesting. I just kept scrolling down and clicking "Older Posts" until I reached the "Andy joined Facebook" post. Here are my findings:

It's only a 36 page PDF (small font though). Though it would be more.
I joined in June 2007.
My first friend is someone I haven't spoken to since.
My first status update is still one of my favorite quotes: "The chief enemy of creativity is good taste. - Pablo Picasso"
I was really into "Graffiti" in the early days. I printed those off too.
I "connected" with many people that I had forgotten about for years. Then promtly forgot about them again! Some I genuinely want to keep in touch with though, so that's good.
I wish there was a graphic scale of friend numbers. I think it would resemble a logarithmic curve, rapidly rising, but plateauing at around 220.

Okay, it's a silly game, but I find it alarming that we invest so much time into something that is probably doomed to extinction or, more likely, habituation. Facebook has become an e-mail replacement and when is the last time you were truly excited about e-mail?

The tool has become stable: less exciting, but stable. The end result is a tool that is far less robust and stable technically than the one it replaced. E-mail may be getting archaic, but it is a completely open and robust mechanism that is largely independant of any specific implementation. Facebook on the other hand is completely tied to corporate interest and a locked down API.

I think we must all make a conscious effort to aim for stability in the turbulent age we live in. To research the tools we use and what the long term strategy is for them. And for those who are in the business of creating the tools, to think long and hard about how robust the system is in the long term. Is a locked and site-specific format necessary? (no, never!) Is your user policy going to limit the long-term viability of the system?

In the case of this blog, I've made a conscious decision to keep control of the content. The site is just the publishing medium, the actual product, for me, is just a folder of plain-text files on my hard drive in chronological order. It may not look flashy with fixed-width fonts and markdown formatting, but it's simple, stable and guaranteed to work long into the future.

Auto Metadata

2010-08-24T13:51:00.001-07:00

One interesting part of the weekend photographic seminar I just got back from was the emphasis on metadata.

It's important to take great photos, but it's just as important to know how you got those results. In the days of film, A photographer would keep a shooting log, recording aperature, speed, lens, etc. A decent digital camera will record all that information for you as long as you know where to look for it.

Photography is one of the best examples of how useful metadata can be. The reason it is so useful is that it is unambiguous and immutable. Any metadata that has these qualities is easy to work with.

Pure Metadata

This is the simplest kind of metadata to work with and often the most useful. It usually has a direct connection to something tangible and concrete. Some types of pure metadata:

File size
File Attributes (resolution, color space)
Physical Settings (ISO, aperature, shutter speed)
Date*

Dates are very useful but only if they are relevant. If the date is not the actual shooting date it is actually less help than no date at all. This can happen with downloaded files or files that are saved to a new location.

Impure Metadata

File names are probably the least valuable indexing tool out there. They don't have to have any relevance to the content, an identical file can have different names and totally different files can have the same name.

Tags are good, but I don't find them as useful as they could be. Again, there isn't necessarily any correlation with the content, there is no standard set of tags or naming conventions enforced. But the primary reason it doesn't work for me is that it's a manual process. Any metadata that isn't automatically applied to all content is too much work.

Folder heirarchies are actually metadata and we use them every day. It's important to understand this when backing up or rearranging files because the heirarchy probably has meaning that would be lost if you moved those files.

The biggest problem with impure metadata, like the one's I've mentioned, is they can be very hard to normalize. There may be information attached as file names, tags and folder structures but it's all been manually assigned by what makes sense at the time. This manual metadata will always be very fragmented and incomplete.

Meota Summer Photographic Seminar

2010-08-21T22:42:00.000-07:00

Just finishing up the weekend photographic seminar at the lake. Got some good shots and some good practice at composition, lighting and rhythm.

Chaotic Data

2010-08-11T14:11:00.001-07:00

Data is chaotic. Our attempts to tame it are largely attempts at dehumanizing ourselves. Here's a transcript of a piece of cardboard on my Grandfathers wall:

 -------------------------
|                         |
|  WALLET    LIST         |
|            ------       |
|   EARS                  |
|             CANE        |
|  WATCH                  |
|                         |
|   SWIM                  |
|                         |
|_________________________|

Even by typing it here, I am imposing a fair bit more order than the original had. The original was written with a Sharpie and, although it was very legible, the intent and structure was difficult to parse. "List" may or may not have been the title or one of the items to remember. "Ears", I assume, meant his hearing aid. All the items seemed like a general checklist for leaving the house until "Swim", which makes the list seem very specific to a certain day, or day of the week. He has had this list posted by the door for quite some time.

Grandpa's list was not very Search Engine Optimized, but he didn't seem to bothered by the shortcoming.

This list has a few other interesting qualities that don't transfer well to the digital domain:

It's taped to the door. Location and size mean a lot in the physical world.
It has no definite structure. Anything could be added or crossed off the list with ease.
It's author is obvious. The handwriting is an echo of the mind of the author.

My point is not that my Grandpa has a habit of making strange artifacts, my point is that we are more unique, creative and human when we're not forced to order our data along the way.

Our attempts to structure and control our data are effectively dehumanizing us. Yes, parsing the messy world of human language and thought is not a simple task for computer systems. It is much easier to build rules and frameworks and force humans to fit their ideas into them; to meet the computer halfway. This has a few effects:

it allows the less creative minds to be efficiently less creative and feel organized
it encourages independant thinkers to constantly sabotage the system for entertainment
it erodes the capacity for beautiful, unstructured and creative enterprises.

I think we too often forget the distinction between the technologies that are built for enterprise and those built for personal use. This is probably because all of the tools are built for enterprise. All of them. The ones that aren't are garbage. Web2.0 or 3.0 or whatever, only really make sense for businesses and consumers. Not for people.

Champions Of Order

2010-08-09T09:44:00.001-07:00

Software Engineers assume that your data can be ordered logically. This assumption is built in to the file system. They assume that it's an easy task, but that it's best left to the user. Better than imposing an order that doesn't make sense for some users.

But the software engineers are wrong and here's why:

Case Study: Plain Text Library

This is the simplest of all systems to represent digitally. All the items are in the same format, each is a distinct entity. So, you start ordering your items by Author into a heirarchy. The index itself is also an item, but not like the other items, so you decide to put it at the top of the heirarchy.

It is simple and unambiguous. But, like all libraries, there will come a day when you want to extend it a bit. You might want to add a scientific paper, a scientific journal, a DVD box-set or an untitled and anonymous poem. You might want to seperate fiction from non-fiction or keep a commentary together with it's source. At every step, you must decide how to incorporate these new elements and whatever decision you make must be applied unerringly in the future.

From the start, the heirarchy must be robust enough to handle any future additions. The end result - it never works. It's hard enough for libraries with full-time staff, training and documented processes to keep things in order. The average computer user has no chance of making a logical heirarchy that will make sense now or into the future.

UNIX and the Emerging Web

2010-07-31T23:15:00.001-07:00

After reading "The Art Of UNIX Programming" again, it's got me thinking about the nature of the technological explosion that has happened since the quaint days of mainframes and text-based consoles.

The two ideas that these legendary computer scientists cared greatly about and seem to be dissolving today are ownership and simple formats.

Ownership

Web-based services take a certain level of trust. The user gives control of their data to a service provider in trust. By doing so, there is a subtle loss of control by the user. We end up with bits of content all over the web that may or may not be accessible or editable by us.

There has been some recent development to build a unified "user space" like OpenID, but in my alternate history of the internet, user profiles would have expanded beyond the workstation into a worldwide user space. Up until now, the user space was determined by email accounts, but with the shifting landscape of email services this was not really a practical solution.

In my alternate history, instead of email accounts, user profile would be generated like IP addresses, and could be linked to email accounts or any other service. Maybe this just seems too big brother for the average user. It would enable much easier tracking of the web services and content that is linked to your profile. I think this would be incredibly convenient for managing your online history; the services you've used and the content you've generated.

Simple Formats

In the UNIX days, plain text was the format of choice for almost anything. Today, raw emails remain more-or-less human readable, but that's about it. Every new web service and application introduces their own formats and interfaces.

Maybe I'm just dreaming, but I think that with some work, text formats could work in the context of most popular web services. Twitter, for example, is simply a collection of text notes with the date as metadata. So why not integrate it with the desktop workflow?

The reason is that web browsers have been designed to be very isolated from the desktop, and the only way to break this isolation is with downloadable formats or task-specific applications that can bridge the gap. Again, email clients are the classic example of a task specic application that grabs a stream of information with a secret handshake and delivers content to the user.

What I envision here is a two-way RSS folder using a universal keychain and managed with something like Git. But what this requires is that the web service and browser be given read/write access to the folder. I think this model is fairly incompatible with the current state of web technology.

Summary

In some ways, I think that the values of the UNIX crowd are coming full circle with emerging web trends. ReadWriteWeb has been blogging for years about the potential of a web that's as easy to write as it is to read. This post is a good summary of emerging trends like structured data. But I think that to really acheive this flexibility, internet and desktop architecture needs to adapt and build robust standards to allow it to happen.

The problem isn't access to services, it's the completely sandboxed environment of the web browser. If we truly want to see the potential of web applications and services, we need a way to let the web in.

Browser Review: Arora

2010-07-21T10:29:00.001-07:00

The Browser War is an ongoing battle with no obvious winners, although there is a seemingly never ending stream of competitors.

The big players at the moment seem to be converging on their basic feature set while edging out their opponents in a certain area. Firefox has extensibility, Chrome has speed, Safari has stability, Opera has security, and IE has users (yes, IE bashing is mandatory in any browser debate!).

Beyond these, there is an interesting subset of purpose built browsers: Flock does social browsing in an interesting way. Sea Monkey includes many tools specific to developers and advanced users. Dillo boasts an extremely small footprint and fast loading.

Arora

But the new browser I've been testing lately is Arora. It is still a very immature version, but I think some of the features are really well done. First of all, it looks really nice on Mac. This may seem like a silly comment, but most apps that have been developed with a "cross-platform" toolkit come out looking like cheap knockoffs with cartoon icons and ugly screen proportions. Even Chrome, with it's oh-so-clever tab bar and Firefox with it's monster icons. This is fine for Linux, which doesn't have clearly established UI conventions, but for Mac development: don't break the rules unless you have good reason.

In this sense, Arora does it right. Not that they use Mac icons, but they are minimalistic and clear, and the layout doesn't add visual clutter or break established conventions. It doesn't shout "Look at this groundbreaking layout!", and that's a good thing as far as I'm concerned. It let's you focus on the task at hand.

I downloaded the .10.1 version which had some serious bugs on the Mac: many SSL errors, no two-finger scrolling. But this is an alpha version so I gave it the benefit of the doubt. I installed the Qt SDK and built .10.2 from source and now it works great. Not really for the faint of heart, so hopefully there will be a Mac binary update soon.

Give me the Keyboard!

The first shock was that it was very keyboard friendly. This is like a +5 million points for me. The way it works is that when you're on a site, if you press the Command key all the links get letter popups beside them. You press the key and it goes to the link. How cool is that! Generally, all the links get letters based on the first letter in the link name, so if a site's top bar has links named "Home", "Reviews" and "Downloads", these will be mapped to the keys H, R and D. All other keyboard shortcuts work as you would expect on a Mac.

These guys have done their homework when it comes to keyboard accessibility. The only exception is using F11 for fullscreen, my macbook has F11 mapped to Volume-DN and fn-F11 mapped to Expose. Not a big deal, considering the wealth of things that they got right, even compared to the big browsers. For example, Cmd-L puts the cursor in the Location bar and Cmd-K jumps to the Search bar. In Safari, you have to Cmd-L to the location and then tab to the search for keyboard only input. Yes, it might be a small thing, but it's good attention to detail.

So far, there's really nothing keeping me from using this as my main browser, which is an amazing feat considering how lightweight and early in development it is. Page loading is fast and application loading is the fastest of any full-featured browser I've used. It renders with qt-webkit, which was a bit rocky to begin with, but has come a long way in a short time.

And you gotta love the polar bear on the globe icon!