Processing

This post is for people like me – programmers who want to take  advantage of Processing’s short iteration cycle, clear syntax, and simple API to use it as a prototyping environment.

First, here is what I think Processing excels at:

• Interactivity – It’s dead easy to get and use mouse and keyboard input and define event handlers.
• Graphics – “Hello world” in Processing is drawing a circle. It has a simple but powerful graphics API. 2D is really simple, and it has powerful features that can be used without any boilerplate such as 3D rendering and a matrix stack.
• Engineering flow – like I said above, the short iteration cycle, clear syntax, and simple API make it a breeze to get up and running quickly and iterating.
• Fun – Because of all of that, it’s a lot of fun to write stuff in Processing, especially for smaller projects where you are just messing around.
• Multiplatform – Out of the box, Processing 1.5 can build native applications for Windows, Mac, and Linux, native Android applications, and web applets (yuck.) Via Proccessing.js you can also supposedly get your sketches to run natively in web browsers.

Here is what I think Processing is weak at:

• Large, complex programs (they tend to get a bit unwieldy since it’s impossible to organize your classes inside a folder structure.)
• Programmer assistance (No out-of-the-box debugger. The preprocessor’s errors are often frustratingly non-specific, and can leave you blindly hunting for syntax errors across multiple files.)
• Audio (there is no out-of-the-box support, but there are libraries available) Processing now includes the excellent Minim library, but you have to explicitly import it. Audio still appears to be an afterthought in the documentation and tutorials.

I have been prototyping games and visualizers (including the one above) in Processing for several months now, and have learned some incredibly useful things that I think any programmer looking into it should know. Here they are:

Processing is Java.

This took me a really long time to realize. I mean, it really is just Java, with a slightly simpler syntax. When you hit “Run,” processing runs your .pde files through a preprocessor that converts them into one giant Java class. Your classes are all inner classes of your main class, which extends PApplet. This is how all of your custom objects can access “global” variables and functions defined in your main class, as well as all the Processing API methods. No magic.

What this means is that out-of-the-box, you can use tons of classes from the Java standard library. My absolute favorites here are Java’s generic collections… HashMap<K, V> and ArrayList<E>. In the Processing documentation, they have lots of examples using the non-generic versions of ArrayList and HashMap. This means that you have to manually unbox (cast) items coming out of those collections. What a pain. Don’t bother, just use generics!

You can also use Java’s for-each syntax to iterate through these collections rather than manually using the iterator as they describe in their documentation. For example, the following is perfectly valid in Processing, you don’t even need to import java.util (that’s part of the default Processing imports behind the scenes):

HashMap<String, String> hm = new HashMap<String,String>();
hm.put("somekey","foo");
for(String item : hm.values() ){
     println(item);
}

A trivial example, but generic collections are extremely useful, since you can lean on the compiler to keep track of the types you’re using. It’s a bit more verbose to create these objects, but it’s much less verbose to use them.

Processing is not Java.

Processing is based on Java, but because program elements in Processing are fairly simple, you can learn to use it even if you don’t know any Java. If you’re familiar with Java, it’s best to forget that Processing has anything to do with Java for a while, until you get the hang of how the API works. – Ben Fry

It really is a good idea to try to discard your preconceptions about Java when first learning Processing. You don’t need to be quite as verbose as in Java, and there’s less technical overhead.

So, I’d recommend that you forget that Processing is Java when you are first learning it. But, if you are able, keep it in the back of your mind, so you can avoid some of the simplistic kludges put forward in the Processing documentation (the main thing that comes to mind for me is using generic collections and for-each loops.) Once you are familiar with Processing idioms, then it’s great to know what’s going on under the hood so you can leverage the more powerful Java stuff.

This also leads into the next most important thing on my list:

You can use a real debugger.

I have had many headaches trying to debug runtime errors in my sketches without the assistance of a modern interactive debugger. I had started getting used to print debugging to figure out what was going on, but in something as complicated as a game engine, I was really hurting for a real debugger. Then I discovered an awesome trick.

You can use a real text editor.

In the PDE, select Preferences -> Use external editor. If you are like me and believe that mastering a powerful text editor is a worthwhile investment of your time, you are free to use your favorite editor when making Processing sketches. The built-in editor in the PDE is okay, but very basic. I am using Vim with this ftplugin for Processing. It supports syntax highlighting, indentation, and documentation lookup via shift+K. It is wonderful. I keep the PDE open in the margin of my screen to do the build and provide a place to display output, it still can be helpful sometimes in locating the line of a NullPointerException, etc.

Apparently there is also a way to call the Processing build without using the PDE but I’m sure it’s not as well tested as that little Run button, so I decided not to muck around with it.

Git is your friend.

If you are comfortable using Git, this is a no brainer. Just “git init” your sketch folder, add your code, and commit. Git is really a helpful tool when working with Processing, especially because the preprocessor is so finicky and unhelpful. I like to stash or stage good versions of my code as I develop, so I always have a version to roll back to if I’m getting syntax errors from an unknown location and the code is clearly FUBAR. Making commits during logical stopping points on features is great, and much cleaner than making a bunch of copies of a sketch. Plus, it makes backing up your sketches stupid easy if you have a remote git repo such as GitHub. I recently have used git’s best debugging tool, git bisect, to hunt down a nasty bug. Git helped me do a binary search through my revision history and track down the commit that introduced the bug, which was a different commit than I had expected. I was able to immediately hunt down and fix the bug since bisect revealed that it was data-side (all I had committed was data,) so I knew where to look.

I also use Git to painlessly manage a master development branch, and an Android branch (since Java doesn’t have #ifdefs…) More on that later.

If you don’t know Git, why not start now? Next to my text editor, it’s my favorite programming tool. I’d recommend reading the first few chapters of Hg Init and Pro Git, which are freely available online.

Android development is frictionless.

You do need to have the Android SDK installed and set up first - one-time setup instructions and more details are on the Processing wiki. That can be a bit of an adventure if you’ve never done it before. Follow the SDK install docs carefully along with the instructions on the Processing wiki and you should be okay.

After that is set up, if you’re not using any special libraries, it may be as easy as selecting “Android” from that drop-down menu in the PDE and clicking “Run.”

I was using the proxml library in my sketch that I wanted to convert to Android, so I had to comment out all of the deserialization code that used proxml. Not a big deal, as that was just for saving levels, I was using processing.xml for loading levels, so loading still works.

I did this on a separate git branch, so all the Android-specific project files live on the Android branch only, along with Android-specific input code and a handful of display code (locking screen orientation, and not specifying an explicit window size.)

After getting everything set up as described above, I have achieved pretty much frictionless multi-platform development, using Git to manage my workflow. This is going really well so far… I do my development on master, and then merge master into my Android branch. Many times the merge is automatic. Occasionally there is a minor conflict that is easy to resolve. I close the sketch in the PDE before switching branches which doesn’t take too long and ensures that it loads my Android settings.

There are a few differences when developing for Android, but overall the development experience is pretty much frictionless, and the Processing APIs are nearly identical. Again, check out the Processing wiki for more details.

Processing helps you learn Arduino.

Another fun benefit of learning Processing is that it’s very similar to the Arduino programming language. If you have ever wanted to mess around with writing software for a microcontroller that can be hooked up to all sorts of crazy electronic doodads and do really cool stuff, this is a great entry point into that whole world! It’s probably the friendliest microcontroller on the market right now and it’s pretty affordable too, well worth checking out if you have a bit of a tinkering instinct.

That’s all, for now

I hope you enjoyed this overview of advanced topics in Processing. I will continue to use Processing to prototype ideas – I like it even more now that I can use it like a modern programming language, and being able to iterate quickly on a game prototype that works on both PCs and smartphones is a boon. If I learn more, I’ll update this article or write a new one.

If you haven’t used Processing, I’m amazed you made it to the end of this article! Go check it out, learn it, and maybe come back and read this again in a few weeks when you will be more able to absorb it. Thanks, and let me know if you have any questions or comments!

For many people, it really doesn’t make a difference. But I’m a programmer. I edit code for a living. My editor is the tool of my trade and I want to use the most efficient tool available.

I’d be preaching to the choir if I was telling programmers that they should use good text editors. But I’ve noticed some patterns in the history of text editors I’ve used that have made me come up with my own philosophy about choosing an editor. My goals and usage habits are certainly not the same as everyone else’s, but I hope that in sharing them I might get you to reflect on your own habits and see if they match up to your goals.

I have used many different text editors over the years, starting with the crappy built-in editor for BASICA. I’ve used Notepad, Metapad, Pico/Nano, TextEdit, SubEthaEdit, BBEdit/TextWrangler, Notepad++, Vim, Visual Studio, and NetBeans to name a few. My reasons for switching (or using more than one editor) were various. I was moving to a different OS, or using multiple OSes. I found an editor with a cooler feature set than my current one and I wanted to check it out. The editor was recommended to me by a friend or teacher. I was learning a new programming language that was highly IDE-centric (like C# and Java) and it made sense to adopt a respective IDE.

I noticed something making all of these jumps between editors. There’s a friction that occurs when switching, a penalty. When I had been using my previous editor I was developing muscle memory from repeated use of certain keystrokes to perform a task. In my next editor I’d habitually hit whatever key I was using in my previous editor for find, find and replace, save, copy, etc. This annoyed me a bit as I had to spend a good amount of time re-training my reflexes to use different keys, to get back up to the speed and subconscious efficiency I had in my previous editor. I began realizing that it was counterproductive to my efficiency to keep switching editors every couple of years. It made sense to pick one editor as my main editor and stick to it so I could retain and build off my muscle memory, a valuable byproduct of using the same editor for a long time. Muscle memory is valuable because it allows our conscious mind to think of an action, but our reflexes to execute it instantly. It can only be acquired through repetitions of an action.

… This phenomenon is sometimes called “muscle memory”, a phrase that tellingly gives credit to an organ other than the brain. The brain is really in charge, of course, but the term rings true because the actions seem to happen without conscious thought. This is the ultimate in efficiency, leaving the higher-functioning parts of the brain free to concentrate on the task, rather than on the operation of the tool required to complete the task.

-John Siracusa

Somewhere along the line, somebody suggested I check out Vim, an improved version of the Vi editor. I ran my own Linux server so I was slightly familiar with Vi, as is any Unix user (it has a way of popping up from time to time since it’s the default editor.) I knew one command, how to quit it! I found this arcane, cryptic editor really off-putting and unnecessarily complicated. What was wrong with Nano for editing HTML pages on my server?

The guy eventually convinced me to try learning Vim using a program called vimtutor. I tried the tutorial. It still seemed arcane and cryptic, but as I worked through it more, I started seeing some crazy possibilities. Vim possesses a succinct and powerful syntax for editing text in a really unusual fashion. I saw the possibilites open up for navigating and manipulating text in powerful ways that had never even occurred to me before. I’ll spare you the details, but it was an eye-opening experience. My text editor could be a more powerful tool for accomplishing useful work than I had ever thought was possible.

I’m not here to preach Vim and get you to switch to it, or start yet another holy war with the Emacs crowd. It seems to me that Emacs is a perfectly valid alternative to Vim – I just don’t have the mental capacity to learn both editors at once.

But, I do recommend that you try out a powerful text editor – ANY powerful text editor! I think it’s an experience that every programmer should at least try. The best contenders right now in my mind are Vim, Emacs, and TextMate.

There are a few reasons why I decided to commit to Vim as my editor of choice:

• Vim is powerful. As I have increased my mastery over its complicated set of commands and modes, my efficiency in editing text has risen proportionally. This is great for me as a programmer – the less time it takes me to mechanically edit text, the more time I can spend thinking about and writing code.
• Vim is flexible. Once you speak it’s language, you can combine commands to create expressions that accomplish a very specific and complicated text editing tasks in a few keystrokes. Its interoperability with the shell is cool. For instance, it’s very easy to use Vim as a lightweight pseudo-IDE for any language that has a command-line compiler or interpreter. It has support for plugins and new syntax rules, which allows the community to contribute rules for all sorts of uncommon languages. If you know a language, there are probably Vim syntax highlighting rules for it!
• Vim is ubiquitous. It’s open source, and available on Mac, Linux, and Windows (and many other OSes including BSD!) In fact, any Unix system I’ve ever used, including Debian 0.91 from 2002 included some form of a Vi editor.
• Vim is lightweight. I think it typically weighs in at around 20 MB. Right now, MacVim is consuming 16.9 MB of memory on my computer and averages about 0-5% of my CPU. This is in contrast to a full-blown IDE like NetBeans, which is 154.8 MB in size, and can take up between 5-30% of my CPU when running and a whopping 329 MB of memory! NetBeans sometimes produces noticeable hangs on my machine.
• Vim is deep. I’ve never encountered another program with the same amount of commands, modes, etc. This put me off at first, but I’ve learned over time that each feature is present for a reason, and in mastering a new feature, I unlock new levels of efficiency in using the tool. It’s a slow burn – I’d expect that Vim has at least a 10-year, perhaps even a lifetime learning curve. (You can learn the basics of vanilla text editing in a few minutes – I’m talking about the learning curve to fully utilize the tool’s powerful features.) I can’t spend all my time learning Vim, I need to also focus on learning programming languages, new technologies, etc.

I have been using Vim as my primary editor for something like 6 years now, and I am still learning new features every day. I find the time I’ve put in to learning the editor is really starting to pay off. Recently, a coworker who was watching me edit some HTML remarked, “Whoa, I thought people only went that fast in movies!”

Here’s what I find lacking in Vim:

• The learning curve is at times ridiculously steep which can be discouraging. It often takes me months to get enough use out of a feature to remember that it exists, how to use it, and finally how to use it effectively. I think, however, that this is part of the price of having a highly powerful and advanced tool for any sort of craft.
• The documentation is dense and in many places seems to assume you are familiar with other aspects of Vim. Considering how massive the feature list of this editor is, that’s not always the case. Thankfully links are provided – but I don’t always feel like learning an entire new subsystem just to get one feature working! Google is my friend here as it’s often more clear to read somebody’s blog post than the actual Vim manual.
• The scripting language seems clunky. I have no desire to learn yet another hacky Unix scripting language – bash is enough for me! That’s okay for now since the handful of Vim plugins I actually want have already been written by someone else.
• Vim supports code completion, which is cool, but it’s a real pain in the ass to set up completion for your own code (i.e. getting a list of methods that I can call on an object of a certain class that I wrote, which is a nice feature that some IDEs provide.) I have yet to meet anyone in real life that has actually bothered to do this. It requires setting up a program called ctags (or exuberant ctags) and running that every time you want to update your completion rules, which analyzes your code and produces a ‘tags’ file. Emacs has this exact same issue by the way, it also uses ctags! This is one area where bulky IDE’s actually provide enough of a service to warrant their rather poor behavior!

With all of this in mind, I’d like to put forward my current list of qualities that an ideal code editor should have:

1. Powerful – A code editor should be able to manipulate text in powerful and context-sensitive ways.
2. Assistive – A code editor should provide facilities for code completion and inline display of documentation relevant to the code being written.
3. Multi-language – A code editor should not tie you to any particular programming language.
4. Crossplatform – A code editor should not tie you to any particular operating system.
5. Plugin architecture - A code editor should be easily extensible to work with new languages by its user base.
6. Open source - A code editor should be easily extensible to work with new operating systems by its user base.

How do some editors I know of rank against this rubric?*

*This table is highly subjective. I tried to include some popular editors that I haven’t personally used much like TextMate and Emacs and assess them by reading about them and talking to people who use them. See Wikipedia for a more exhaustive comparison of text editor features.

I was a bit hard on the IDE’s listed here. To be fair, most of them support a handful of different languages. However, I don’t think they meet the criteria of “should not tie you to any particular language” since a lot of their more useful features such as code completion would not be available in anything other than those languages even if you were to use them as a plain text editor.

There isn’t an editor that actually meets all 6 of my criteria. Vim and Emacs come the closest, but the fact that most human beings don’t actually have the patience to install and run ctags for code completion makes me rank them down in the “Assistive” category, whereas in the IDE’s this is automatic and built-in. I also ranked the IDE’s down for “Powerful” because the context-sensitive editing abilities in their default editors are tied to their particular languages, whereas these commands in Vim are generic and work in any language. Also, if you’ve used Vim, your definition of ‘powerful editing’ changes and you realize that what are typically considered powerful editing features are weak sauce.  Emacs gets a pass since it has a metric assload of modes for a bunch of different languages.

Other editors such as TextMate seem appealing, but are not crossplatform or open-source. I am a generalist; I write games for Windows, backends for Linux, and utilities for Mac OS X, among other things. Mac is my primary platform at the moment, but I feel like it would be foolish for me to put all my eggs in one basket by tying myself to a text editor whose creators openly proclaim it will never be ported to any other system.

Another enticing option is the franken-editor… people have actually written plugins for various IDE’s that allow a simulated Vi editing mode (jVi, ViEmu) or in one case, a real instance of Vim to run inside of the IDE. This actually winds up providing a great mix of editing power and assistive support from the IDE. It’s a win for Vim ubiquity since I can re-use my exact same muscle memory for editing within multiple IDEs. However, this scenario is plagued by the same issues that make the IDE fall short of perfect, by tying you to specific languages supported by the base IDE.

To wrap up, I don’t think the ideal code editor exists. If you have the good fortune to be coding in a language supported by a decent IDE, that could be the ideal editor for that language, but not others, and you’ll pay a friction penalty switching between the IDE editor and whatever other editor you use, unless you’re doubly fortunate and there’s some sort of plugin to make the IDE work like your other editor or vice versa.

Vim and Emacs are pretty dang good, but fall short of perfect in my eyes since it’s such a pain in the ass to set up ctags. If either the Vim or Emacs community was able and willing to bundle this functionality up in a more automatic and integrated way (i.e. not making me compile the damn thing for OSX or force me to ask my sysadmin to install the exuberant ctags RPM in a shared environment,) they just might fit my criteria for perfect code editor. At least until I actually used them and thought up more features I’d want.

I typically use find for two purposes:

1. To create a list of files matching some search terms.
2. To execute a batch operation on those files using the -exec argument.

In Shell Scripting: Becoming a Wizard, I layed out a recipe for productive and useful bash scripts that uses a while loop over some sort of multi-line input. This construct is more powerful than find‘s built in -exec flag, but that power is not always needed.

Here is how I use find:

Stage 1: Build up a list of files to operate on.

By default, find just prints the list of matching files, which is really useful because you can check the list of files you’ll be working with before you move on to actually doing something to them. It also makes find useful as a general purpose file searching tool that’s available on most Unix systems. Note that find does not inspect the contents of a file directly, only its external attributes. If you want to search through file contents from the command line, check out grep/fgrep or ack. (fgrep -r searchterm file1 [file2 ...])

The general form for find looks something like this:

find directory argument1 [argument2 ...]

Typically, I want to run a recursive search from my current directory (which is ‘.’ in bash convention.) I’ll include a bash prompt signified by the $ character, and example output from the find command to better illustrate what’s going on:

$ find .
.
./bar.jpg
./baz.png
./foo
./foo/foo2.jpg
./foo.jpg

This command (find .) is the simplest working invocation of find, and produces a full listing of every file and directory inside your current directory (including the current directory itself.) I rarely ever want this though, so I will begin by limiting my search. A common way I do this is by searching for filename matches. For instance:

$ find . -name '*.jpg'
./foo/foo2.jpg
./foo.jpg

Uses wildcard/glob syntax to search for only for files with names ending in .jpg. If we wanted to also include files ending in .JPG or any other case combinations, we could use the -iname (case insensitive name) expression instead.

$ find . -iname '*.jpg'
./bar.JPG
./foo/foo2.jpg
./foo.jpg

This is working well so far. Supposing we wanted to limit our recursive search to directories of only a certain depth, we can do that using the -maxdepth flag. To search only the immediate children of the current directory, we use -maxdepth 1. To search only immediate children, and the children of immediate subdirectories, use -maxdepth 2, and so on:

$ find . -iname '*.jpg' -maxdepth 1
./bar.JPG
./foo.jpg


Let’s say for the sake of example that we wanted to search based off of partial filename instead, for files called ‘foo’:

$ find . -name '*foo*'
./foo
./foo/foo2.jpg
./foo.jpg

Whoops! We don’t want any directories in this case, so let’s narrow our search criteria to files only and not directories:

$ find . -name '*foo*' -type f
./foo/foo2.jpg
./foo.jpg

Perfect! -type f selects files and -type d selects directories, respectively. In this example we could also use glob searching on the name for files such as -name ‘*foo*.jpg’. However, -type is still useful since Unix systems allow file ‘extensions’ on directory names and allow files without extensions.

How about if we only want to find files in a certain subfolder? We can use -path :

$ find . -path '*foo/*.jpg'
./foo/foo2.jpg


A trivial example, but this is really useful for searching large source code trees for a particular file, etc. For instance, if I know there’s a class that has Twitter in its filename and resides in some subfolder of Services, a search for -path ‘*Services*Twitter*’ should turn it up. A case insensitive version of this flag is -iwholename.

What if we want to use some logic in our search? find supports that. For instance, we can search for all jpg files that do NOT contain ‘foo’ in their name using the ! operator:

$ find . -iname '*.jpg' ! -name '*foo*'
./bar.JPG


How about if we want to find all jpg and png files? Use -o for logical or. It will be true if either the previous flag or the next flag is true (but not if both are):

find . -iname '*.jpg' -o -iname '*.png'
./bar.JPG
./baz.png
./foo/foo2.jpg
./foo.jpg

Tons of other functionality exists in find, although the above are all the commands I use on a daily basis. find can filter on modification/creation time, ownership, more unusual file types like FIFOs, and use other types of logic such as AND. Run man find for the full list.

Stage 2: Execute a command across all selected files.

Now that we have a list of files we want to use, we can take advantage of find‘s most useful and powerful flag, -exec. The syntax for -exec seems cryptic at first, but comes to make sense with repeated use.

There are two forms I typically use when calling -exec:

-exec somecommand {} \;

and

-exec somecommand {} +

The former will execute some command once for each file in our search results. The latter will execute one command once (or as seldom as possible) and pass the files in as multiple arguments. For example:

$ find . -iname '*.jpg' -maxdepth 1 -exec echo file {} \;
file ./bar.JPG
file ./foo.jpg

The command echo file {} is being run for each result, with the path to the result being substituted where the {} is. \; is just a terminator character to let find know that that’s the end of the command we want to run. If we get rid of the echo (a debugging trick I use to preview a list of batch commands) the file command will actually be run against all these files, printing out their true type for us:

$ find . -iname '*.jpg' -maxdepth 1 -exec file {} \;
./bar.JPG: PNG image, 64 x 32, 8-bit/color RGBA, non-interlaced
./foo.jpg: JPEG image data, EXIF standard

Good thing I checked – looks like bar.JPG is really a PNG with an incorrect extension!

The file utility is even kind enough to print out the file name of the file it’s talking about. It’s also spiffy enough to accept a space delimited list of files to parse (as do many command line utilities) so we can improve the performance of this batch operation by reducing the invocations of file:

$ find . -iname '*.jpg' -maxdepth 1 -exec echo file {} +
file ./bar.JPG ./foo.jpg

Rather than file getting called twice (once for each result) it’s only called once, with each result being passed as an argument. In this trivial example this behaves the same. But if you need to run a batch operation over many files, it can make a difference in speed.

Also, if your batch operation involves something like gluing a bunch of files together (like with cat to concatenate files) or opening them all at once in one editor, the + style invocation is necessary. It basically serves the same function as xargs, so there’s no reason to pipe the output of find to xargs. Using -exec {} + is safer and faster. xargs is still useful when taking input from another source such as a text file, but be aware of paths with spaces in them, they need to be escaped properly.

Limitations

find‘s -exec flag is only designed to run one command at a time, and doesn’t support fancy bash tricks such as output substitution and piping. If you need those, use the bash while loop outlined in Shell Scripting: Becoming a Wizard. You can still use find to create the list of files you want to operate on, just pipe the output of find into the bash while loop.

Note that the -exec {} + style invocation may not be present in older versions of find. If that’s the case and you have no choice but to use xargs, avoid nasty escaping issues by telling both tools to use the null character as a filename delimiter instead of newlines using find -print0 and xargs -0, like so:

find directory -print0 [argument2 ...] | xargs -0 somecommand

Conclusion

I hope you learned a thing or two about find today. Remember, practice! I always had to look up the -exec syntax the first several times I used it, or if I haven’t used it in a while (which is no longer likely since I now use it almost every day!)

After reading this article and experimenting a bit, you should now be able to understand the find examples for doing useful work I mentioned in my article on The Shell and Coreutils. I hope this is enough to spark your imagination for the massively powerful automation techniques available by coupling find with other command line utilities.

Other resources

Here are some great resources to learn more:

• man find – The find manpage on your system has a definitive list of valid expressions and operators.
• Greg’s Wiki – UsingFind – This is where I learned most of what I know about find along with the man page.

Here is a video of a game prototype I was working on in UDK. My idea is to express the feeling of being in zero gravity in a way that hasn’t been done in games before (i.e. no jetpack.) You move by pushing yourself along surfaces and kicking off them towards other surfaces. Once airborne, you can’t alter your velocity without grabbing on or bumping in to something (or possibly throwing stuff out of your pockets.) I hope someday to make a real game from this concept.

I reflect on this prototype, why it went to the back burner, and what I learned after the jump.

Why it’s back-burnered: I got bogged down in a lot of difficult 3D math. Also, I’m finding it a pain to just figure out how to do what I want in UnrealScript due to lack of documentation. The last time I worked on this, I had just finished implementing a custom six degrees of freedom (6dof) camera that uses quaternions to represent rotation instead of the default euler angles. After jamming on this for several weekends, I sort of lost steam on the project.

I have inklings of a vision for a larger exploration/puzzle game in zero gravity, but I’m not quite sure how I want to proceed. Also, I think this idea of mine might be too ambitious in scope.

Actually building this prototype helped me immensely to understand the issues that would arise in designing this sort of game. I hit an issue with 6dof cameras that I have not seen solved yet by anyone who has attempted this. Notice that if you rotate the camera in small circles in any game with a 6dof camera (say, in Descent, or Shattered Horizon when you’re not walking on a surface) you’ll see it start to roll. The larger the circle, the more exaggerated the motion is. This means that if, for instance, you yaw left by 90 degrees (i.e. look along your left shoulder), then pitch up by 90 degrees (look straight up from there) and then yaw right by 90 degrees (move your head to the right) you will be looking forward again, but the world will have rolled by 90 degrees! This feels really unnatural in a 1st-person game where the mouse is supposed to control head movement. (Below is a diagram of roll, pitch, and yaw axes if you are unfamiliar with the terms.)

The only alternative I can think of, that I was working on coding before I back-burnered this project, is a system where you can control the position of your body seperately from your head. In that case, you would be able to orient your body with six degrees of freedom, and could in fact do the rotation described above, and it would be OK since you are in zero gravity and deliberately re-orienting yourself. But to keep it so that the situation described above doesn’t happen when you are just casually looking around, the roll on your head would be locked, so if you did the rotation described above, your view would not roll (this is how regular FPS cameras work.) If anyone is interested, I can make some videos that illustrate the issue I am describing here.

Music: TiLT – Nova Lab Background Theme
http://www.medievalfuture.com

Nidhogg is a brilliant fighting game for two players created by Messhoff (Mark Essen.)

I suck at most fighting games, although I have a lot of respect for the people that play them. Watching my old roomate Kilo play Super Smash Bros. Melee and BlazBlue and talking with him about it helped me to understand the deep nature of these games. So did the excellent documentary I Got Next. Key to these games is the idea of a metagame, and playing mindgames with your opponent, occurring on different layers of anticipating your opponent, called yomi. Check out David Sirlin’s excellent article for an explanation of the layers of yomi in mindgames. The basic idea is that you have a certain set of moves, some of which can be used to counter others. Some provide a greater reward (more damage, a stun, etc.) but at higher risk. Some are lower reward but also lower risk (low damage, shielding, etc.) Effectively mindgaming your opponent relies on knowing what your opponent thinks you will do, and playing off those expectations. A noob will try to use his most powerful move at all times if he knows how good it is, but this move is risky, so a pro will know that and counter it. But if a pro is playing another pro, he might also use the powerful move, precisely because the other player wouldn’t expect him to since it’s typically too risky.

Sirlin’s game Kongai did an excellent job of bringing these mindgames to a wider audience by removing the need for the millisecond reflexes that pro fighting game players must have, following in the vein of stuff like the battle game from Pokemon.

Nidhogg has accomplished something fantastic, in that it retains the real-time thrill of a classic fighting game, but provides a relatively small amount of actions that the player can take, which makes the game easier to pick up and play for the average gamer without having to learn a bunch of different combos for each character. However, the design is elegant, as these moves are more than enough to create interesting mindgames.

More about my thoughts on Nidhogg after the jump.

The objective of the game is to get past your opponent. You need to go several screens in one direction, and your opponent needs to go several screens in the other direction to win. When you kill your opponent or pass them onto the next screen, they respawn pretty quickly, a bit of a ways in front of you. The only difference between the two players is that one is orange and one is yellow–their abilities are identical.

You can run, jump, duck. If you duck while running, you do sort of a duck-slide. When you are walking with a sword and near your opponent, you automatically take a fencing stance. You can position the height of your sword, move backwards and forwards, and do a short jab with your sword at whatever height it’s at. If you are running and try to jab your sword, you will throw it. A thrown sword can be deflected if you are holding a sword out and facing it. If you don’t have a sword, you can do hand-to-hand kicks and punches, although these are closer range and do a lot less damage, whereas a sword kills in one hit.

That’s pretty much it for controls, as far as I can remember. Just a directional control and two buttons. I was able to play the game equally well on an Xbox 360 controller and an NES gamepad.

The most interesting part of the game, of course, is the mindgames that arise. Throwing your sword is a risky, but awesome move. If it hits, you kill your opponent, have a few seconds to run and pick up your sword again, and advance in the game. If your opponent dodges or deflects the thrown sword, you just made a huge mistake. An opponent with a sword can easily overpower one without a sword, so if you are standing in the way of the other player you’ll likely need to try dodging them when they come in to stab you or throw their sword at your squishy unprotected body, or try to find another sword lying around to defend yourself with.

A great move in the game is the duck-slide (that’s what I’m calling it, anyway.) If an opponent holds his sword at waist height, you can slide right under it and get past him quickly. But if he holds his sword low to the ground, you get skewered. Also, I think if he holds his sword to the ground, he leaves himself open for an attack by a thrown sword. So when one armed opponent is running at another, it’s always thrilling to see the outcome. You can also approach each other slowly like fencers, trying to jab/block your opponent.

The game lends itself to many tense, unexpected, and hilarious situations. Playing it at a packed GDC party really added to the excitement and atmosphere. It was awesome to see a player execute a duck-slide under his opponent’s sword and run all the way to the end of the hall, only to be cut down at the very last second by a sword thrown hail mary style from halfway across the room, to the cheers and laughs of the crowd. Or two guys brawling it out over a sheer drop, one guy kicking the other guy off the ledge, and trying to jump over but missing it and tumbling down himself. The fast pace of the game and constant respawning means the tide can be turned at any moment. From moment to moment, both players can experience swings between what you might call epic win and epic fail.

I don’t know how Nidhogg will be received by the fighting game community, or if most people there will even hear about it. Perhaps the mindgames won’t be deep enough. Without being a fighting game player myself and only having played a handful of rounds, I don’t know the answer. But the potential is there, and it’s palpable. Undeniably, Nidhogg is a wonderfully entertaining party game – watching two people who have never played before duke it out is just as entertaining as watching players who seem to have a better idea of what they’re doing.

It looks as if Nidhogg might have a single-player mode as well, but I didn’t get a chance to see it.

I got a chance to chat with Messhoff briefly about the game. I asked the perennial question, when is Nidhogg coming out? He said he had been hoping to release it at GDC, but that it just isn’t ready yet. But hopefully, it will be released “soon.” It goes without saying, but I’m brimming with anticipation.

Tee is a wonderful but often-overlooked core utility. It’s purpose is to intercept a pipeline, write the output to a file, and pass the output along the pipe. It can be used to easily add output logging to any command pipeline or script.

To get the most out of tee, it pays to read up on how input, output, and errors are handled by the shell. (As a byproduct, a good understanding of bash I/O redirection will make you a more efficient bash user and scripter!)

There are three standard streams that can be used by any unix program: stdin, stdout, and stderr. Each of these can be thought of as a file. By default, all unix programs read from stdin, write normal output to stdout, and write error messages to stderr.  The shell’s job is to take things being read or written to one of the standard streams and redirect them through pipes that you specify. By default, the shell will take the characters you type into the terminal while a program is running and pass them to that program’s stdin. It will take the characters written by the program to stderr and stdout and print them to your terminal.

If you instead insert a pipe, like so:

command1 | command2

the shell will redirect the characters written to command1′s stdout to command2′s stdin via a pipe. Command1′s stderr will still get written to your terminal.

Bash has syntax for specifying custom redirections. For instance, you can redirect stdout to a file instead of your terminal:

command1 > file

This is where tee comes in. If we want to both log something to a file and pass it along a pipe, or just have it written to the terminal (by leaving tee as the last command in the pipe,) we can insert tee into our pipe:

command1 | tee file | command2

If you want to do something other than use tee to log just stdout, you should read more about bash I/O redirection. Here are some resources:

• Advanced Bash-Scripting Guide: A Detailed Introduction to I/O and I/O Redirection
• Advanced Bash-scripting guide: I/O Redirection
• Greg’s Wiki: How can I redirect stderr to a pipe?
• Greg’s Wiki: How can I redirect the output of multiple commands at once?

Typical invocation:

some command that produces output | tee file

Invocation:

file file1 [file2 file3 ...]

This is where scripting comes in. You can write scripts which are capable of performing a lengthy and detailed series of operations in order. Your computer can do these tasks over and over and over, very quickly. Unlike you, your computer doesn’t need to sleep or eat. It will perform just as well at 3am as it will at noon. All you need to do to take advantage of this amazing machine is to learn a language that it understands so you can instruct it on what you want it to do.

The awesome thing is, if you’ve been using your shell, you already know a scripting language. Every command you type at the command line can be thought of as a shell script.

I provide an overview of my favorite bash scripting facilities after the jump.

I will use bash as my example here since I am most familiar with it, and it’s likely to be the shell you are using as well. Instead of typing commands into the terminal, we will save them to a file. I recommend a text editor such as the free TextWrangler. You can use TextEdit in plain text mode, but beware that it can sometimes behave poorly and cause unforeseen consequences due to how it handles formatting of line endings. Mac OS X (and almost every other Unix system) ships with a variant of an editor called vi which runs in the terminal and is incredibly powerful. vim is a vi variant and my editor of choice. However, the learning curve for vi is really steep, so I wouldn’t recommend it for beginners, unless you need to do a lot of work on a remote Unix webserver, are an efficiency freak, or simply want nerd street cred like me.

So, open up a file in your text editor. Type in some commands. Each line of the file represents one command, or command pipeline. So we might type:

echo "Hello world! Here are your files:"
ls

If we save this file to our home directory as “myscript.sh” and then open Terminal and type the following:

bash ~/myscript.sh

We should see the script run! It should print “Hello world! Here are your files:”, and then a list of files in the current directory.

This is nice. Just using this method, we can consolidate several separate commands into a single script which will run them all in sequence.

However, bash provides us with many language features that allow us to give our computer much more flexible instructions. We can tell it to repeat a segment of instructions an arbitrary number of times (a ‘for loop’ or ‘while loop.’) We can give it logic to determine what to do in a number of different situations (by using ‘if statements.’) We can even have our scripts take flags and arguments from the shell itself, like any other command line utility! In effect, we can complement the coreutils by writing our own utilities in bash. Ultimately, this allows us to make our computer ‘smart’ enough to accomplish large and tedious tasks that would otherwise require an intelligent person like you!

Now, I am going to give you some magic sauce. Remember this later, once you have become more familiar with bash scripting. This little template for a bash script has proven more useful to me recently than anything else. It’s in pseudo-code so you’ll need to swap out a few things to make it work.

something that produces output | while read index
do
  some command on $index
done

This can also be written on one line. It’s less readable, but can be useful since you can test it out directly in the shell without having to save it to a file:

 something that produces output | while read index; do some command on $index; done 

This is the key to real ultimate power. Why? Two reasons:

1. The input can come from ANY command or pipeline of commands. It can come from a file. It doesn’t matter. Our script will go over the input line by line and do something related to it.
2. The current line of input we’re on is stored as a variable, here I’ve called it index. A variable is just a container for some value, in this case, the contents of the current input line. This allows us to run the same block of code on many different lines of input, munge them around in some way, and execute the result as a command!

Here is a specific example of a script written using the above template.

find . -name '*.jpg' | while read index
do
	echo mv $index ~/photos/PHOTO_`basename $index`
done

What this script will do is create a list of commands to move all the files ending in .jpg in the current directory or any subdirectories to the folder ~/photos, and prefix PHOTO_ to the file name. It’s “nerfed” in the sense that it won’t actually execute this set of commands, I did this just in case you wanted to copy this onto your machine so it wouldn’t mess up your files.

It’s easy to “un-nerf” this script – either delete the “echo” – so that the command being run is mv, not echo (which was just printing the mv command.) You could also pipe the output of this script into a new shell, like so. The second instance of “bash” will read the input line by line and execute it.

bash script.sh | bash

This script also takes advantage of another powerful bash feature called command substitution. You will notice a command enclosed in `bacticks` (that’s what happens when you press the tilde/~ key without pressing shift.) This tells bash to take the output of whatever is inside the backticks, and use that in the final command. So in our example, we’re calling the basename utility, which in our example gives us the base name of the file path stored in $index (just the file name without any path information.) Since it’s enclosed in backticks, the output of the command will get added to the echo mv command. So in effect, each time the while loop runs, it  will execute a command that’s  something like “echo mv /path/to/file.jpg ~/photos/PHOTO_file.jpg”.

Perhaps not a super exciting example, but it demonstrates the additional flexibility we have with this method over just using utilities such as find and xargs. With those tools, it gets rather convoluted if you want to do something like move and rename a file based on a pattern. The beauty of this method is that since we, the scripter, control the logic entirely, we can make anything happen inside this loop. We can iterate over a list of file name fragments, with each line becoming the next index variable, and then call find from within the loop to try and locate files which match the fragment, and do something to them.

A corollary to the above is the bash for loop, which can loop over the contents of a directory.

Here are some examples of scripts that I have written entirely in bash:

• Do a mass search-and-replace in ALL the files in a directory
• Zip up a series of directories one-by-one, and submit the zips to a web application along with a bunch of data gleaned from reading the files in the folder
• Search through my server logs, count accesses to a certain resource, and give me a daily count
• Read in a list of folder name fragments, and do an svn-move of matching folders to a different location in the svn repository.
• Use sed to batch-replace a value in a file with something else, and bash file I/O and the mv command to replace the original files

Here are some resources for learning more about bash scripting:

• Linuxcommand.org – Writing Shell Scripts
• Advanced Bash Scripting Guide
• Writing Robust Shell Scripts – A must-read once you’ve finished making toy scripts and want to move on to things more powerful, and therefore dangerous.
• man - Many of bash’s functions and syntax are documented in man pages. Check ”man bash” and “man test” for starters.
• Google – As always, if you’re stuck, google it!

A word of warning: I find bash an excellent scripting language for tasks of low to moderate complexity.  However, the syntax is a bit rigid and opaque compared to recent higher-level languages suitable for scripting, such as PHP and Ruby. After taking an upload script I had written in bash and adding all sorts of features such as command-line flags, error and output logging, etc, I soon regretted it as the 100+ line script became unwieldy. I rewrote the script completely in PHP to make it cleaner. I plan on writing a brief introduction to scripting in high-level languages soon.

It’s certainly possible to write clean and modular bash programs for larger applications, but I find the syntax of other languages more comfortable when I need complex control logic, data parsing, etc.

However, the ease of converting a shell command into a simple bash script, and the brevity of its syntax still makes bash my language of choice for relatively simple scripts.

Up Next:

• Curl: Internet Magic Sauce
• Scripting in high-level languages

Previously in this series:

• Better living through automation, pt. 1 (Automator)
• The Shell and Coreutils

Every operating system in the Unix family (including Mac OS X) ships with a set of core utilities, or coreutils. These are small, fast programs which are invoked via the command line. The coreutils are truly the holy grail of automation.

By design, each utility is very, very good at one specific thing. For instance, cut will print out one or several slices of the text you pass into it, as determined by a “delimiter character” that you specify. paste, on the other hand, will join two files together line-by-line. Together, these programs can be used, for example, to combine reports from multiple files and format the output how you want.

This design is known as the Unix “toolbox” philosophy. The analogy is that a swiss army knife is supposed to be good at everything. But you can’t build a house with a swiss army knife. Professional contractors have a toolbox full of highly specialized tools that can only do one thing, but are extremely good at that one thing. The same goes for each of the core utilities. Some software tries to be a swiss army knife and often winds up as the “jack of all trades, master of none.” The core utilities are like the contractor’s toolbox. By using several specialized tools in combination, you can achieve quick and powerful results.

The glue for the coreutils that allows you to use them together is called the shell. The shell is what you are interacting with when you start your “terminal” program, such as the one provided by Mac OS X located under /Applications/Utilities/Terminal.app. Bash is the most popular shell today, and is the default shell on most modern systems including recent versions of Mac OS X.

The shell interprets commands that you type into it to construct a pipeline between the different programs you are using. So, if I type:

ls

The ls program lists the files in the current directory and returns them to the shell, which prints them to the screen. But now, if I type

ls | grep "MyFile"

the shell is taking the output of ls, and instead of printing it out, it passes it to another program called grep. grep searches through the output of ls for the text “MyFile”, and sends only the lines that contain it back to the shell. The shell prints those lines to the screen.

Where we really get cooking is when we start using output from one command as instructions for another command. The find utility is great for this, as is the xargs utility. Find will search recursively in the folder you specify for a set of files which match the parameters you give it. Then, using the -exec flag, you can tell find to execute any command on each one of those files. xargs does something similar, but it can take any text as an input (typically, file names) and provide them as arguments to another program. find / xargs are designed to control other programs, and can be used for tasks such as a batch move, batch rename, batch conversion, etc.

so in keeping with our current example:

ls | grep "MyFile" | xargs open

The output of grep, those files or folders containing the word “MyFile”, is passed by the shell to xargs. xargs passes them to the open command, a Mac OS X utility that will try to use Finder to open each file. Simple functionality like this can be more elegantly achieved using only the find command. See my tutorial on using find to learn more about this incredible tool.

Also worth mentioning are sed and awk. These programs are used to perform manipulation of text inside a pipeline.

There is a learning curve here, but the more I learned about how to effectively use the command line and the coreutils, the more my productivity increased.

Mastery of your shell’s syntax is crucial in becoming effective. Using the shell, you can construct complicated pipes which branch and merge data and execute commands to do practically anything on a massive scale.

To become a master of using the shell to automate things, I recommend that you practice doing more and more things from the shell instead of from a GUI. You need to understand what the most common utilities do, and the common arguments/flags that you can pass them in order to get their full range of functionality. If/when you move on to writing scripts, this knowledge will help you take advantage of built-in functionality instead of trying to implement it yourself. It becomes easier to practice shell skills if you run a website, since you often have to log in via ssh to a webserver in order to install software and make configuration changes on the webserver. I picked up most of my basic shell skills through running my personal website.

A word of warning: Make sure you are familiar with using a command normally before invoking it with find or xargs. This type of usage requires knowledge of how the shell and different programs handle spaces, etc. You can cause damage or data loss if you don’t know what you’re doing. So learn the basics first. Test out your batch commands by adding echo in front of the command, so it will print instead of execute and you can inspect it. Test out your execution on one or two files before you execute the whole batch.

Here are some resources for learning more about the command line:

• LinuxCommand.org – Excellent in-depth overview of the command line, suitable for total beginners
• Bash Guide on Greg’s Wiki
• man - Take advantage of the built-in manuals for each command on your system by typing “man somecommand”. Type q to exit.
• Built-in help: Many programs have built-in help. This is usually available by invoking the program with the -h or –help flag, such as “mv –help”.
• Google – I can’t stress enough how useful it is to just look something up on Google when you’re stuck. Chances are, somebody else was having this exact same problem and found a solution!

Here are some examples of tasks and automation that I frequently perform using the shell and some utilities. These are provided not as a tutorial, but an example of the kind of powerful automation you can achieve from a well-crafted command.

I’ve added echo statements to all the examples that will cause a change so they will print the commands they would execute instead of actually executing them. I’m worried about people randomly pasting commands into their terminal and running them. Here’s a hint: DON’T DO THAT! With great power comes great responsibility. Make sure you understand what a command does before you execute it on your system so you don’t get owned! Note that find (and other) syntax can vary from system to system and shell to shell so these may not all work in your environment.

Move a bunch of files matching my search term to a different location using find:

find . -name 'test*' -exec echo mv {} ../foodir \;

Batch-convert a bunch of files matching my search term using find and a converter program (imagemagick in this example):

find . -name '*.jpg' -exec echo convert {} {}.png \;

Tidy up the output of a monitoring program like ps to only contain the information i’m interested in, using head, grep, cat, and bash input substitution syntax:

cat <(ps|head -1) <(ps|grep bash)

Use diff, comm, and other programs to find the differences between two files/directories, or perform set theory operations

There’s a whole in-depth article on this!

I hope I have whetted your appetite a bit to go learn more about using the command line. With practice, you can accomplish in a few keystrokes what could take tons of clicking and dragging in a GUI.

Up next:

• Shell Scripting: Becoming a Wizard
• Curl: Internet Magic Sauce

Previously in this series:

• Better living through automation, pt. 1 (Automator)

Anything that’s tedious is repeatable; anything that’s repeatable should be done by a computer.

-Robert Muller

I spend a lot of time working on my computer every day. Many of the tasks that I need to perform on a computer can be incredibly tedious. Copying and pasting text from a website into a spreadsheet. Renaming a bunch of files and moving them to a new location. Zipping a bunch of directories and uploading them to a website one by one. From an early age, I was always frustrated at these kinds of tasks. It’s a tragic waste that we have commodity PC’s capable of performing billions of calculations a second, but are still left doing the same tedious crap that we were doing 10 years ago.

Clearly, there’s no lack of machine ‘brainpower.’ What most people lack is the knowledge required to effectively leverage their personal computers to automatically perform tedious tasks for them. I have successfully automated all of the tedious tasks I mentioned above in my own workflow, and many more.

My mission with this series of articles is to provide a high-level overview of the most effective technologies for automation that I have used, and share some pointers that have been key to my success. There are already many great tutorials that go in depth into how to use these particular technologies. I will try to provide links where appropriate, but a quick Google search is usually sufficient to turn up lots of great resources on any particular technology or program. If a technology I describe sounds useful and interesting to you, I encourage you to go try it! Monkey around with it, Google when you get stuck, and try to solve at least a small part of a real problem you are having that could use some automation. That’s how I learned everything that I know now.

If there’s anything you’d like me to explain in more depth, please leave a comment and let me know what it is. I will try to help, or at least point you to where you can find more information.

I’d like to talk a little bit about Mac OS X. At this time of writing, Mac OS X is my favorite desktop operating system. Here’s why:

1. By default, everything works pretty much how I want it to.
2. Because Mac OS X is built on top of Unix, there’s a ton of flexibility built in. At any time, I can drop down to the command line to accomplish what I need to.

Granted, Mac OS X is nowhere near as configurable and customizable as a modern desktop GNU/Linux system. However, even with the advent of fairly easy-to-use Linux distributions like Ubuntu, I think the barrier to entry might still be a little bit too high for the average user. We’ll get there eventually.

Automator

From an automation standpoint, Mac OS X is at the top of its game. Included with the OS is a really slick application called Automator. Automator allows you to “snap together” instructions from a library of parts to create a workflow describing what you want the computer to do. Essentially, it allows you to create your own scripts without writing a line of code. You can even save them as application bundles and send them to other people. This is a great place for someone who wants to dive in to the world of automation without any prior experience. The ability to install custom “actions” created by others, and write your own actions in AppleScript, allows room to expand the capabilities of Automator. Some tasks I have used Automator for include:

• Re-sizing, renaming, or moving batches of photos (I recommend the Proper Cropper action for cropping instead of Apple’s default.)
• Performing a series of repetitive keystrokes in an application to load files in by the batch (this made use of a custom AppleScript module to simulate key presses)

One of the reasons I don’t like Automator and don’t use it more often is the fact that, as far as I can tell, you have to write your custom code in AppleScript. (Edit: I recently discovered that Python and Ruby are available as alternatives, see my comment below.) AppleScript tries to appear like natural language, but it’s just a trick. It’s just as rigid as any other programming language, but the syntax is just kind of weird and occasionally ambiguous. You are also limited to whatever actions ship with Automator, or whatever ones you can find online or write yourself using AppleScript. In the end, I found Automator a great way to interface specifically with Mac OS X GUI applications and do certain common tasks, but that I had to turn elsewhere for real ultimate power.

Up next:

• The Shell and Coreutils
• Shell Scripting: Becoming a Wizard
• Curl: Internet Magic Sauce

Stay tuned!