When applications are open to clearly-expressed control, it’s easier than ever - easier than with MIDI - to navigate their interfaces and make them express what you want. Using OSC (rather inaccurately named Open “Sound” Control), interacting with live visuals is getting a lot more powerful. Case in point: thanks to hacks, we’re getting affordable multitouch control of Modul8 with an iPhone or iPod touch, and even a new implementation of “chaos mode” on Resolume Avenue 3. There are free downloads for these projects, too - no super-secret special sauce; these visualists are happy to share. And while some of the hacks are specific to these tools, you could apply the lessons elsewhere.

At top, Spanish Vj Chinowski has done some elaborate TouchOSC layouts for controlling Modul8. (Chinowski tells us about them on Twitter.) Various OSC apps have appeared on the Apple mobile platform, but this particular application is enabled by the new editor for custom layouts TouchOSC’s developer recently released. What’s also nice about the TouchOSC layouts is that they don’t use Apple’s stock widgets, which aren’t always best for control. (My own Android OSC app is delayed for the same reason; I’m doing some research into what’s best for control and what yields the greatest performance and writing the widgets from scratch.)

Chinowski is an accomplished visualist with a huge resume, worth checking out. The site is in Spanish but should be fairly easy to follow even if you don’t speak the language, or skip straight to the download page.

http://www.lasonora.net/VjChinowski/VjChinowski.html

In other news, our friend Gian Pablo Villamil in New York has worked up a full-blown Max 5 patch that produces a “chaos mode” for use with Resolume Avenue 3.

It’s a good demonstration of what’s possible with Resolume’s OSC implementation, and could lead to other things. And, of course, with OSC you have your choice of whatever development environment you like - Processing, Python, or Pd, for instance (just to take the things that start with the letter ‘P’).

Included features:

• BPM configuration
• Different random modes: in sequence, random, and the “drunken walk” (I’ve occasionally taken one of those myself)
• Preset storage and recall
• Layer-by-layer control

You get the idea - not only is this immediately cool and useful out of the box, but if you fancy trying ideas yourself, this should get you started.

Explanation, installation instructions, and further discussion on the Resolume forum:

Chaos Mode for Resolume Avenue - Max patch

Now, I’ll shut up - let’s go play!

Since the very beginning, Vimeo has retained the original video files that you upload, and allowed you to grant people permission to download those files. We always take these original files and convert them so that they will play online and in the Vimeo player. We kept the original files for download because we wanted you to be able to save your video exactly the way it looked when it left your hard drive. This aspect of the service was not a huge burden on us when the site was younger, but we’ve had to take another look at what we are realistically capable of offering for years to come, while making sure the site stays on budget. Original file storage of every file for every user is a massive cost, and we have noticed that only a very small set of users actually ever download their own files. We want to keep original file storage around as a feature for people who use it, but we can’t continue to do it for everybody.

This has actually prompted me to finally make the upgrade to Vimeo Plus. The upgrade had a slight at the point of a rapier feel to it, but because the strongest impetus I had was to keep my original file uploads, I had a quick look through and discovered that I have over 3GB of source files sitting on Vimeo’s servers. $60/year isn’t such a bad deal for this kind of storage volume. It’s in line with online storage services such as box.net, or normal web hosting accounts, and of course Vimeo’s only limit is the 5GB/week upload. So even just as a file hosting service it’s a reasonable investment.

Vimeo Plus Tutorial from Vimeo Staff on Vimeo.

Of course there’s the other Plus features, most attractive to me being pre- and post-roll customization and unlimited Groups/Albums etc. Personally I don’t understand the rationale of sites - Flickr being another one - which nerf their free accounts in this way. Albums, Groups, Sets and other playlist-style features make media sharing sites considerably more useful and navigable, so limiting this functionality to a small subset of users limits the overall usefulness of the site.

I’ve also heard from another social video service, Tubemogul (which allows seamless uploading across multiple video networks, more on this soon), that Vimeo are considering blocking Tubemogul access for Vimeo free accounts. This seems a little anti-competitive and myspace-ish, blocking 3rd party tools that make your site more efficient.

Pettiness aside, I’m still loving Vimeo as a service, and the community seems to be holding off the Youtube-like hoards or trollish commenters. Let’s see how my first year as a Plus member pans out.

Ed.: Vimeo still looks like a top choice to me. We’re working on publishing more CDM’s own videos to Blip.tv - we plan on making video a bigger part of the CDMs, and Blip has a more liberal ad policy, plus easier cross-posting to things like iTunes. But Vimeo is terrific for sharing your work. I’m also intrigued by dailymotion, which recently had a big presence at the Open Video Conference and is the first to do a broad test of OGG and HTML5 for Flash-less video playback, something critical to the future of the open Web. Blip founders were at that conference, and it’d be great to see a Blip or Vimeo join in. Thoughts on any of these services? We’d love to hear them. -PK

No, this is not Processing.

Looking for inspiration only within our computer boxes is limiting. Want a fresh perspective? Check out the physical world. And yes, you’ll find even alien-looking patterns of particles out there.

I recently read a post by talented digital artist Golan Levin pointing to the non-digital work of Ward Fleming. According to Fleming, “you’re seeing 40,000 black acrylic spheres 0.125 in. dia. vibrating/contained on an off level glass plate horizontally mounted and back lit.” I also love the more poetic description from YouTube:

the agony of particle behavior. struggling to express consciousness in a world animated by mechanical vibration. a truly empathetic study of particle emotion expressed as fluid/crystal bipolarism.

You’ll find more examples. Be sure to look at these in HD for the full effect.

More information on the 3.5×8-foot pinscreen Frieze Machine by Ward Fleming is available in a separate video. You get to see him making the work, and then watch the beautiful image of a human silhouette against the screen. (Note: pinscreen-modeled nudity.)

Looks like it’s time to write a pinscreen shader.

You can actually buy this kind of work:

Areaware / Atomix

That site notes that this design by Francois Dallegret is not new:

Dating back to the 1960’s, the designer has re-introduced his artful plaything for a 21st century audience. Made from 6000 high precision stainless steel balls, Atomix creates an infinite number of fractal patterns when shaken, tilted, or rotated.

I’ve been blogging effectively every day for nearly five years, so it’s hard to avoid novelty - that’s kind of what we publish. The question is, is it the novelty that’s important, or do you see these as little steps toward something greater that hasn’t happened yet? I tend to favor the latter.

Here’s the question: can augmented reality - using computer vision tools to mix computer graphics with stuff in the real world - become a real medium and not just a gimmick?

If it works in a game, you begin to think there’s really something to an interactive design. I’m not certain yet about the upcoming PSP game from giant Sony. Georgia Tech researchers, on the other hand, are coming up with stuff I really want to play.

Oh, yeah - and it makes me crave some Skittles. (Product integration, anyone?)

About the game:

ARhrrrr is an augmented reality shooter for mobile camera-phones, created at Georgia Tech Augmented Environments Lab and the Savannah College of Art and Design (SCAD-Atlanta). The phone provides a window into a 3d town overrun with zombies. Point the camera at our special game map to mix virtual and real world content. Civilians are trapped in the town, and must escape before the zombies eat them! From your vantage point in a helicopter overhead, you must shoot the zombies to clear the path for the civilians to get out. Watch out though as the zombies will fight back, throwing bloody organs to bring down your copter. Move the phone quickly to dodge them. You can also use Skittles as tangible inputs to the game, placing one on the board and shooting it to trigger an explosion.

Skittles could be the computer interface of the future. See also a music sequencer made with Skittles. (The only problem, as your inner Cookie Monster might quickly point out, is when you eat the interface.)

It’s still just an experiment, but the interaction itself starts to really require the technique - and, in turn, it becomes something that actually gets you closer to the game.

For visualist artists and performers, there’s something really powerful and expressive here, too, which is the opportunity to remix “reality” itself (or at least what you can see through a live video feed) with computer-imagined graphics, all live.

It’s even more impressive to see this running on mobile devices. And we’re not far from this kind of performance on current mobiles, either.

You’ll find lots of other projects on the Augmented Environments Lab site, and plenty of other ideas. There are technical solutions, like building an augmented reality scenegraph for easier development. There are concepts for integration, like bringing augmented reality into the virtual reality world of Second Life. There are virtual reality pets and mixed reality games, too - and even a concept working with the MIT Media Lab’s Scratch that makes augmented reality serve teaching programming in a more tangible way for kids. (Somewhere, game developers are looking at it thinking it could be a great way for grown-ups to prototype games.)

AR Scratch: A Tangible Programming Environment for Children

Developers, there really are some great concepts there in terms of how to develop both programming concepts and mapping of imagery to the AR-tagged objects. Not only does this have educational potential, but it could help develop notions of how interactive programming in general could work, and how our closed digital boxes relate to the world around them. Figuring out just how that works is not a small problem - it’s the kind of problem you could spend a lifetime thinking about and developing, on a timescale we can easily miss in the blogosphere.

But I’ll close with a project called Joe Warpin, from Georgia Tech’s handheld augmented reality class, the inspiration for the zombie game. It reminds me of the days as a young kid when I spent hours roaming the universe inside a room in my house with Matchbox cars and toy planes. That brings up an important question that often gets left out: the question of imagination. On one hand, you could say that getting overly focused on augmented reality takes out the ingredient of imagination, by working too hard to substitute technology for the imagined virtual reality in a child’s mind.

I’d say just the opposite: I think imagination is what gives augmented reality real potential. If you can meet the user halfway - actually stimulate their imagination - and leave just enough to their head, I think AR could be not only successful, but addictive.

APPARATI EFFIMERI Tetragram for Enlargement from Apparati Effimeri on Vimeo.

Apparati Effimeri write to let us know about their latest visualist projection mapping project, “TETRAGRAM FOR ENLARGEMENT.” Watching abstract patterns wend their way across the geometries of the castle-like building is hypnotic, to be sure. But it’s as the visuals make the building seem to melt, collapse, and reconstruct itself into fluid shapes worthy of Dali that things
get really interesting.

Blog post + stills

I have to ask: is it really club culture that has “failed” the visualist, or just the limited architectures of the clubs we have now? Projection mapping has tended to be the domain of “art” visuals. But I wonder if music and visuals alike might benefit from sharing new spaces. Heck, I’d be happy to go dance outdoors to adventurous music programming while visuals crept around the buildings around me.

So, do spread this stuff around, because sometimes the live event metaphor in which a lot of us find ourselves is something very different — the silo.

You’ve seen the demos. You like the idea of tracking tags in the real world to create visuals. And now you want to try augmented reality for yourself - and, incidentally, you’re a Flash developer.

Reader Eric Socolofsky writes to share a framework he’s created that makes it much easier to work with the Flash-based, open source FLARToolkit, called FLARManager. Version 0.4 is just released:

http://words.transmote.com/wp/20090618/flarmanager-v04/

FLARManager has a number of features that improve upon the existing work done by FLARToolkit:

• Building the apps themselves is easier. Fire up the framework with Flex Builder (or Flash, or Eclipse, or FlashDevelop), and you have access to all the libraries you need, so you can start playing more or less out of the box. Hello, world, indeed.
• You don’t have to rely on Papervision if you don’t want to. Papervision, the faux-3D library for Flash, is included with the distribution. But marker tracking is decoupled from Papervision, so you don’t have to use it if you don’t need it.
• Better event management. Marker adding, updating, and removal, multiple pattern detection and management, and the like are all extended in FLARManager.
• Great documentation. Eric has taken the time to read some fantastic getting started tutorials, all accessible from the site above so you can go play.

Now, you wouldn’t pick Flash for speed - that’s not the idea.

This is about the slowest implementation of ARToolkit you’ll find. But you’d use it for compatibility, because of easy deployment to the browser. Speaking of speed, the NyARToolkit Java implementation actually outperforms the original C version. I’m the last person you should talk to about writing efficient, optimized code, but I can tell you that the notion that Java is “slower” than C is simply wrong. There are a great many other, more important variables, and in some cases Java can in fact outperform C. That doesn’t mean that Java is always the right tool for the job any more than C is, though, and in fact because Java’s Garbage Collector and event scheduling aren’t really built for real-time performance, and because “native” code is suited to certain situations, there are in fact times when you wouldn’t want to use Java. Understanding the application is what really matters - and that’s why it’s nice that NyARToolkit and siblings are available for AS3, C#, the mobile Android platform, iPhone, and others.

Eric doesn’t just do this to fool around, either; he works in interactive design for museums, and has what has to be one of the world’s sweetest “day jobs” - working for the legendary Exploratorium. He’s also working with Processing and the wonderful reacTIVision library on a separate project that’s working with tangible table-based interfaces; more on that separately. Thanks, Eric!

If you get interesting work in augmented reality going, let us know. And if you need some inspiration, my current favorite is from our friend Marco Tempest, working with Zach Lieberman in OpenFrameworks on augmented magic.

MSA Remote + VDMX + Ableton Live from Memo Akten on Vimeo.

Imagine what’s now possible with a mobile phone: anyone with a supported device can jam with other artists, walk up to an installation, connect to other creators and other software, all using supported protocols. Leaving behind the days of painstaking manual adjustment of MIDI commands and obscure drivers, and even the act of having to physically connect gear, software - and with it, digital art - can simply talk to each other in standard ways.

That’s why we’re excited about software like Memo Atken’s MSA Remote. It uses the standardization provided by the network-savvy, open protocol OSC, with additional plug-and-play (or, erm, don’t plug, do play) functionality from the TUIO protocol. OSC provides the communication; TUIO makes the messages standardized.

To avoid confusion: You do NOT need a Mac to use OSC. OSCulator is a cool app - and makes bridging to MIDI easier - but it’s just one tool among many. You can use this app with Windows and Linux, too, and visual apps like VDMX, Resolume Avenue, Pd/GEM, Processing… the list goes on. In fact, almost every visual app today worth using uses OSC, even as the music world is painfully slow to catch on.

Features:

- Multitouch information sent using standard TUIO protocol for instant integration with existing TUIO clients
- Accelerometer data for each axis (x, y, z) is sent
- 64 faders (8 pages of 8 faders)
- 64 triggers (8 pages of 8 triggers)
- 108 key (9 octaves) VELOCITY SENSITIVE polyphonic keyboard. Yes, the harder you hit the keys, the greater the velocity.
- Settings are automatically saved and restored
- Multitouch area orientation can be set as desired
- All information on protocols are documented in the app

I’m hoping MSA Remote is just the beginning of apps out there - and that we see support for slick, new devices from Nokia, Palm, HTC, and others (and for Android). But it’s especially good news to see MSA Remote available on the iTunes Store, where it was initially rejected. I still think that Apple’s one-store-only approach to distribution is a liability, and since thy’re committed to it, I’d like to see them improve the approval process. But that doesn’t mean I’m any less excited that the app is finally here.

Check it out:
MSA Remote for iPhone

That page also details what TUIO is about and with which apps you can use this. I’m doing some work myself on some of this, so stay tuned for more. And while multi-touch is cool, I think there’s an opportunity to keep standardizing other messages, too, like pen input, location sensors, and other data. It should also be possible to make software smarter about conncting to other OSC apps and hardware. (Oh, yeah, and while we’re at it, we could use more dedicated OSC hardware!)

But thanks for the great work, Memo - and for the acknowledgment. And thanks to everyone who (politely, I hope) communicated to Apple that this was an app we need.

Graffiti Wall meets MSA Remote from Alex Beim on Vimeo.

Technology still has the power to appear like magic. And one place we may desperately need magic: straightening out our horribly shaky, handheld video shots. Software makers like Apple have already offered up some techniques for doing this - in the case of Apple’s Final Cut Studio, optical flow analysis attempts to track the image as it shakes around the screen and compensates by adjusting the orientation of the frame. But a research team at the University of Wisconsin, partnering with Adobe, will present a new approach at the legendary graphics-geeky SIGGRAPH conference in August. They go one step further, applying a 3D mesh to the image to warp your image three-dimensionally to make the stabilization even more seamless.

Me writing about it is basically useless. Check out the mind-blowing results in the video. From the description:

In this paper, we describe a technique that transforms a video from a hand-held video camera so that it appears as if it were taken with a directed camera motion. Our method can adjust the video to appear as if it were taken from nearby viewpoints, allowing for 3D camera movements to be simulated. By aiming only for perceptual plausibility, rather than accurate reconstruction, we are able to develop algorithms that can effectively recreate dynamic scenes from a single source video. Our technique first recovers the original 3D camera motion and a sparse set of 3D, static scene points using an off-the-shelf structure-from-motion system. Then, a desired camera path is computed either automatically (e.g., by fitting a linear or quadratic path) or interactively. Finally, our technique performs a least-squares optimization that computes a spatially-varying warp from each input video frame into an output frame. The warp is computed to both follow the sparse displacements suggested by the recovered 3D structure, and avoid deforming the content in the video frame. Our experiments on stabilizing challenging videos of dynamic scenes demonstrate the effectiveness of our technique.

The research, at the University of Wisconsin-Madison:
Content-Preserving Warps for 3D Video Stabilization

You can view all the techie details there, as well as many more demo videos. This is promising stuff, and we’ve seen in recent years a vast acceleration of the time between academic research and shipping commercial products — especially with cheap computational power on home computers to play around with, and increasing challenges for software vendors to differentiate what they’re doing in a mature application space.

Side note: boy, do I want to go to SIGGRAPH this year.

Also along these lines: Spacetime Fusion, tests of Final Cut’s SmootCam feature, more SmoothCam tests

For those of you purists, yes, it’s still worth considering the art of steadicam shots - at least before technology obliterates it for us clueless masses. Previously: B&H Interviews Steadicam Inventor: Shooting is Like Dancing

Via Nat Lecude, here’s what the Fingertapps application platform looks like on Dell’s obscenely-affordable Studio One 19 desktop. (Try a whole computer with a multi-touch screen for US$849 and up.)

There are a few concerns here:

• I’m not quite sure why there appears to be so much latency in the demo. That could have any number of sources - latency is a complex issue - or could simply be intentional interpolation on the part of the software.
• I’d love to be able to take off the Dell’s stand and use this at a more humane 30-45-degree viewing angle — which, if you think about it, is what we should be doing anyway, for the sake of our backs and necks!
• Of course, open source frameworks are preferred.

All of that said, though, there are some really compelling ideas here. And imagine having physical hardware controllers combined with touch manipulation in your visual rig. Okay, at least, I’m drooling here. Heck, if you need more horsepower than the Studio One, you could connect it to a rack with more muscle. (The Studio One itself ain’t bad, with up to a quad-core CPU and standard 7200 rpm drives even on the cheaper multi-touch models. Now, if you only weren’t limited to just the 9400M for graphics, it’d almost be a must-buy.)

You can bet we’ll be watching this evolution. Anyone who doubts it, multitouch is coming fast, after a long wait.

Another video for you of Fingertapps:

When it comes to controlling software, let’s put it bluntly: OSC good, MIDI bad. With OSC, it’s possible to control the array of things software might do, with easy use of high-resolution data, descriptive names in plain English (or your language of choice), a path hierarchy that makes it easier to structure messages in modular software, and smart networking features that makes assignment and communication a breeze. With MIDI, um… well, prepare for lots of mucking around.

Happily, visual software developers proprietary and open source alike have done what music developers generally haven’t - embrace OSC. Thanks to the fact that this community is unburdened by tradition and commercial development tends to involve small, responsive teams, change hasn’t been so tough.

So, visualists, it’s time to reap the fruit of that development work, and make the live performance rig work the way you’ve always dreamt it should work. Our friend Gian Pablo (check out his fantastic blog) clues us in to some recent developments with Resolume 3 “Avenue.”

First stop: be sure to read the full OSC thread on the forums, starting with how Resolume Avenue’s OSC receive implementation works:

OSC manual
OSC section of the manual

That’ll get you started controlling Resolume Avenue with Processing, which could make for some cool integration of your custom-coded Processing visuals with a more traditional Resolume visual or A/V set.

Last week, though, we got into some other interesting details in that thread, like how to control the BPM thread and an undocumented feature:

Page 3 of the thread

Specifically:

There is an undocumented feature that enables you to send OSC messages from the mapping panel.
This way you can quickly find out what messages you’ll need to send to specific objects.
1. Shutdown Resolume if running
2.Open the ‘config.xml’ file in ‘My Documents\Resolume Avenue 3\preferences\’ and look for the OSC section.
3. Change the section:
4. Change the port and address to whatever is appropiate for your setup.

That also holds the promise of Resolume sending as much as it receives - no reason visual software has to be receive-only. Also, this illustrates that we need a basic protocol that allows auto-discovery of OSC-controllable parameters, but that’s a separate discussion.

Have a look and let us know what you think. And feel free to share your results across different applications (not just Resolume) on Noisepages — even if it’s a quick message to say, “darnit, I’m confused about all this and don’t know where to start.”

Noisepages OSC Group
[Warning: Noisepages is in beta - we need your participation to see how it's used and to fix bugs, so join the Help & Development group to provide feedback, and expect more features / slicker look in the coming weeks and months]