The first video is an openFrameworks visual I’m working on, controlled here with the touchpad on my MacBook Pro, which can read multiple fingers at once.

The second video is FiddlyBits running on an iPad2. You can get it free at fiddlybits.mmmlabs.com.

The third video is Animata, controlled by a Kinect and OSCeleton. I got all the info on how to do this from github.com/?Sensebloom/?OSCeleton

The music is the CC-Licensed track “Lightyear” by Anitek
.

Quartz Composer has always had an easy way to play movies. You drag and drop your movie into the editor, connect the image ports and the movie plays. However, its control over movies has left much to be desired. We can set the timebase to external for a movie, which the exposes the Patch Time parameter. If we change that the movie shuttles to our new position, but we can’t get it playing again without changing our timebase. And you can’t change the timebase with an external controller, so this method is out for live performance.

Also, to play the audio track of a movie, you have to switch to Asynchronous mode. In Leopard, this also meant losing any control over playback.

The movie loader in Snow Leopard changes all of that. We can zoom between positions in our movie, while the video and audio play continuously.

Let’s take a look.

Asynchronous mode in Snow Leopard gives us a lot more options than it did before. Here are the new parameters we’ll focus on:

Reset Signal:

Shuttles the movie back to the Selection Start and continues playing.

Selection Start:

The playhead, effectively. Changing this will not affect current position until Reset = True.

Rate:

Playback speed for audio and video. Slow-mo effect also pitches audio up and down. Works in reverse.

Building it Yourself

In this short tutorial, you’ll learn how to use these parameters to control your video and audio.

1. Start a new composition and add a Clear patch and a Billboard.
2. Set the Billboard’s Width to 2.
3. Drag in a movie from your hard disk (an SSD is highly recommended). You can download a free high-quality countdown movie at the end of this article.
4. Connect the Image output of your Movie Loader to the Image input of your Billboard.
5. Use the Inspector (Command-i) on your Movie Loader to turn on Asynchronous mode.
6. Set the Movie Loader’s Rate to 0 to stop playback.
7. Change the Selection Start to an arbitrary amount and check the Reset box to see your movie jump to the new point. Note that you have to uncheck and then recheck the box to get it to jump again.

Now we just have to combine these actions – setting a new Selection Start and resetting the movie. I’ve got a finished version of this page available for download at the end of this article. I recommend you download it and open it up.

Playing With My Version

In this setup,  we still have the Movie connecting to the Billboard, and we’ve got two new additional patches. We’ve got the Keyboard patch, which sends key signals through the Javascript patch, which interprets them, sends the Position data out to the Selection Start on the Movie, and also sends a Reset signal.

Notice that the Movie’s Duration is being fed into the Javascript patch. This is how the Javascript patch knows where in the movie to jump to. For the curious, here’s the Javascript code:

Using the home row keys on your keyboard (a, s, d, f, j, k, l, ;), you can jump to 8 different spots in the movie. You can also change the Rate for some fun time-bending effects. So there you have it – controlling a movie – With Audio – in the new Quartz Composer for Snow Leopard.

Next time we’ll take a look at swapping out this keyboard patch for a Midi Notes receiver so we can control our videos with Ableton Live.

Coming in early 2010: Realtime Graphics With Quartz Composer

The first how-to book on using this awesome language. Co-authored by Surya Buchwald (Momo the Monster) and Graham Robinson (VJ Shakinda).

For my first foray, I’m working on a Four Layer system.

1. Background – Scene-setting clips.
2. Playground – Video Instruments (Quartz Composer)
3. Foreground – Elements on black, textures.
4. Overlay – Logos, text.

Take Control

I’m a gear-hound. It used to take me HOURS to get ready for a gig as I packed every device, cable, work surface, and display device into my poor old ’85 Corolla. One of the advantages to sussing out a system like this is that I get a chance to refine my physical interface.

Software: VDMX

I chose VDMX for this setup because of its tight integration with Quartz Composer, which is integral to this particular structure. It can read my native QTZ files,  generates GUI elements for published inputs, and can use QC compositions as pass-through FX. Don’t worry if that’s all gobbledy-gook to you for now. Just trust me when I say it’s the bee’s knees for QC integration.

Faders/Knobs: Evolution UC-33

The (now-discontinued) UC-33 packs a metric tonne (or so) of faders and knobs into a reasonably compact, bus-powered interface. Yes, the knobs are a bit close together and the faders certainly aren’t going to win you the DMC – but it will get this jobbie done.

Buttons: Monome 128

The patron saints of Compactness, Versatility, Minimalism and Sustainability had at each other one wine-filled night, and the Monome is the only earthly record. Or it’s just a box of buttons that can light up. Thanks to some beyond-the-call-of-duty programming by Dave Lublin, I was able to use the Monome to fully control my clips. 4×4 groups of buttons control 4×4 grids of clips, with additional controls for switching Media Bins on the fly and scrolling up and down through larger bins. The Monome also acts as a 12-key MIDI keyboard for the Quartz Composer patches that will be looking for one.

Fun Factor: Logitech Gamepad

I love working with a gamepad. It takes two hands, which can be tricky when you’re trading to fade layers in and out. I’m still working on that part. But it’s a barrel of monkeys  to work with (translation: fun), and introduces another interesting structure to work with. Also, it’s hella cheap. No worries about handing one of these to the 5-year-old kid at Maker Faire (just keep yer paws off my Monome).

The Verdict: Oh yeah.

I built this system just in time for the ’08 Decibel Festival. Even though a lot of the programming didn’t come together until the last minute, I had a solid, fun show. It certainly helps that I was playing with an awesome band, but having that structure there really gave me something to focus on. And I’ve got a host of ideas for real 4-layer scenes I want to build. For the sake of the tutorial (and the performance), I just wedged my existing content into this new framework. I think once I’ve had some time to build fresh content for this system, I’m going to have a really solid, extensible show on my hands.

Has anyone else thought through their performances like this?

If you’re not familiar with Lucifer, it’s a VST/AU plugin for realtime beat-based cutup/repeats of audio. If you’re not already a Lucifer user, this article may not interest you much – but I do suggest you check out the video to see what’s possible.

What you’re going to do is route the MIDI from Lucifer out to another program that will do Video cutups. This is useful for more than just video – with the MIDI signals coming out of Lucifer, you can control and trigger and MIDI-capable software and hardware. But if you’re still reading at this point, you probably already knew that.

The first thing you do is set up a new graph in Lucifer. Under the ‘Effects’ tab, select ‘MIDI CCs LAT’ and turn on the effect by clicking the little red button just to the left of the dropdown menu. Now for video use, we want a perfect rising graph in 64th-note intervals. Yes, you’ll have to draw this by hand – but it’s easier if you start with bigger chunks and work your way up to the finer control of the 64th notes. Update: Steve Duda tells me that you can use the ‘shape’ tool to draw the graph automatically. Thanks, Steve!

Then copy-and-paste this graph to all of the other graph sets for which you want MIDI out.

The last step within Lucifer is to set the MIDI Control that will be sent out by doing a click-and-drag in the box just to the right of this MIDI symbol. In my example, I use CC#125.
The last bits you need to do are in Live. Set up a MIDI track, turn its monitor to IN, and set it to receive MIDI FROM whatever track Lucifer is on. If Lucifer is the only MIDI item in that track, it should come up in a little box below the Track Name you’ve selected. If not, the select it, silly.

Now you set the track’s MIDI TO port to send wherever we like. On a Mac, you can use the IAC bus. In this case, I used Isadora Virtual In, which is generated by Isadora whenever it’s running.

This tutorial has covered enough ground without going into a step-by-step Isadora tutorial. I’ll give you the general idea of what I did to get the video working. If you’ve used any node-based languages: Max/MSP, Quartz Composer, Reaktor, Cinema 4D Expresso – Isadora is quite similar, it just has different names for things.
In Isadora, I have a MIDI CC Watcher that routes the value information from Lucifer into a Movie Player, hooked up to a Projector for playback. I also use a Real-Time Watcher to observe the incoming MIDI Clock information from Ableton Live and calculate the Loop Length needed to play the video at 1x speed in time with my loops from Live.

If you want any more info, or some clarification – leave a comment and I’ll see what I can do to satisfy you.

Roadmap:

• Only allow control via Lucifer while a modifier key is held down (otherwise the Audio in Isadora suffers)
• Map Lucifer modifer and gridset keys to Monome.
• Map ‘Reverse’ key in Lucifer to also reverse video in Isadora.
• Your suggestion?

In Leopard, we need to open the Patch Creator. In Tiger, this is always open, and called the Patch Library, found anchored to the left side of the Editor. Type ‘Clear’ into the search field to drill down to the patch we want. Drag-and-drop the Clear patch to the Editor.

Now we’ll do the same with a cube. Type ‘cube’ into the search field, and then drag the patch into the Editor.

Select the Cube and press Apple-i to bring up the Inspector. This is where we can hand-adjust all the aspects of our Cube.

The cube is a little too big to work with, so we’ll take it down in size. Type 0.25 into the Width, Height and Depth fields.

Now let’s play with the X, Y and Z positions of the cube. Spin the dials to the right of the text fields labeled X Position, Y Position and Z Position and watch how the cube moves around in space. To change the values, you need to click and hold on a dial, and the move the mouse in small clockwise circles to raise the value, and counter-clockwise circles to lower it.

Let’s reset these values to 0 by typing directly into the fields.

Now let’s play with the X Rotation property. You’ll notice that the changes we’re making are very small. This is because Rotation is measured in degrees, so to go all the way around once, we need to increment from 0 to 360. Hold down the Shift key for faster rotation. We can recognize that the Cube is three-dimensional, but it’s hard to tell, and not very pleasing to look at. Let’s add some lighting to flesh out the look.

Type ‘lighting’ into the patch search, then drag and drop a Lighting environment into your composition. Notice the square edges on this patch. Our previous two patches, which are both Renderers, have rounded edges. The squared edges of this Lighting Environment patch tell us that we can put other patches inside of it.

Select the cube and press Apple-X to Cut it to your clipboard just like you would with a Word Processor. Double-click on the Lighting patch and press Apple-V to paste your Cube back inside. In Leopard, you’ll see the above breadcrumb trail letting you know that we’re inside the Lighting Patch, which is inside the Root patch of our composition. In Tiger, you may have to click on the ‘Hierarchy Browser’ to see this. Click ‘Edit Parent’ to return to our Root Patch.

Back in our Root Patch, click on Lighting and press Apple-i to bring up the Inspector. Change Attenuation to 0.1 to smooth out the lighting, and play around with the Light 1 XY and Z Position to see how it affects the composition.

Click on the color box next to Light 1 Color to bring up a color wheel. Pick a new color and observe the changes in your viewer.

Next we’re going to bring in an Interpolation patch which will provide some automated control.

In the Inspector for the Interpolation patch, choose a Start and End Value of 0 and 360, respectively. This tells the Interpolator what values we want it to output.

Connect the Result from the Interpolation patch to Y Rotation on Cube. Now the Interpolation patch is counting from 0 to 360 (using decimals in-between for a smooth output) and we’re using those numbers to change the Y Rotation on the cube. Change the Duration of your Interpolation from 1 to 0.1 and watch it go freakishly fast. Change it to 10 and observe it slow down. Revert it to 1 second and let’s move on.

Next we’ll add an Audio Input patch so we can interact with our composition.

Connect the ‘Volume Peak’ output to the ‘Height’ input on the Cube.
In your viewer, your cube should now be responding to your default system input – on a laptop, this will be your internal Mic. On a desktop, you may need to plug in a Mic or some other External Audio source to see it respond.

NOTE: It is possible to redirect iTunes to your QC comp using a utility called Soundflower – look for a tutorial on this topic soon.

Looking at the viewer, you’ll notice that our Cube is not hitting the top and bottom of the display area. In order to change this, we’ll add a Math patch. We’ll route the signal through the Math patch (as pictured above) before sending it along to the cube.

In the Math Patch Inspector, we’ll change the Operation to Multiply, and try ’4′ for the amount. In Tiger you will default to having two operations – you can leave the other one blank.

Now we should see the cube stretch all the way to the top and bottom of the screen. We can play with the multiplication amount until we find a spot we like.

Now it’s time to experiment on your own. Open up patches and change some values, see what happens. When you find a change you like, hook up an Interpolation or Audio Input patch to it. Stick Math patches in your routing streams to take control of your values. Experimenting like this works to cement the ideas you learn, and steadily increase your understanding of the Quartz Composer System.

So for this first part, I had to find something related to ‘water’ to videotape. Thankfully, there’s a fountain right near my office (the one at the Grove for any Los Angeles readers). I always carry my little camera around in my man-bag, so on the way to work one morning I stopped by the fountain and videotaped it like all the other tourists.

After work, I transferred the vids over to my laptop, brought them into Final Cut and started cutting them up to short clips – I specifically looked for ones that had rather defined movement, which would help in matching them up with audio.

Next, I decided to spice up my clips a little bit. The colors on my lil camera are alright, but nothing to write home about. I fired up some color-correction tools and made the water a bit more aqua to help it pop on-screen. Then I used a combination of the contrast tool and the trusty ol’ Kaleidescope to abstract the clips a bit.

Since I took the video in a public place, I knew on-location audio recording was out. So once I had a nice selection of clip possibilities, I opened up Ableton Live.

In Ableton, I fooled around with a free subtractive synth I’d found called Automat. Once I had a sound I liked, I made up a few patterns, and then started flipping back and forth between Final Cut and Ableton live to see what might match up. When I found a pair that looked close, I would fine-tune it a bit in Ableton, and then export it to Final Cut. Rinse and Repeat.

After about an hour, I had sounds for all my Final Cut clips. I lined them up, trimmed the clips to match the sound lengths (making sure that the first frame of video started on the sound) and set up a Batch Export.

My specs:
640×480
Motion-JPEG B at about 85% Quality
Linear PCM Audio

Once done, I uploaded them to archive.org and compiled all the clips into the AV Challenge #2 Clip Packs.

In Part Two (coming soon) – I’ll load the clips up and start making beats and melodies!

This is a tutorial on how to mix video using a Nintendo Wii controller and VJ Software such as Resolume or Isadora for your PC. I used a Dell Inspiron 8100 with a gig of ram running WinBlows XP, some kinda ATI video card that has disappointed me in the past, and a 2.4 MHZ processor. Running Vista? Haven’t tried it yet. If you have Vista and this walkthrough works for you, let me know.

About the Wii (from the excellent wii to midi walkthru by ANTIVJ.com)
In addition to the traditional buttons, force feedback and the integrated mini-speaker, the true innovation is the presence of a gravity and acceleration sensor (the ADXL330, used in the army or automobile safety, for airbags systems) inside the Wiimote which collects the movements and acceleration of the Wiimote on each of the 3 axis, X, Y et Z.The good news is that rather than using a proprietary protocol to transmit the infos from the Wiimote to the Wii, and then lock its hardware to keep an exclusive use of it, Nintendo has made the excellent decision (probably by commercial will) to use the Bluetooth protocol, standard of communication on PC and Mac for several years. This means that you can connect your Wiimote to your computer, and program it rather simply for any use you want.

About this tutorial
Use this tutorial however you want, feel free to add to it, be sure to forward it to whomever and give it away for free. If you charge some lazy bastard for this tutorial, you suck. If you use it for a performance and you make money– you rock!

What you will need
- A Wiimote, of course (approximately $35 online or at your local video games shop).
- A Bluetooth Dongle (if your computer doesn’t have Bluetooth already, about $30).
- Bluesoleil (shareware, it usually comes for free with Bluetooth dongles).
- GlovePie (freeware). Converts the signals sent from your wiimote to Midi (or anything else).
- MidiOX (freeware). Manages MIDI. Although I can use it to route my in and outputs, I really only use it as a MIDI monitor. You can contribute to MIDIOX thru Amazon.com– I gave them 10 bucks.

Other resources:
Using the wiimote with Resolume (very excellent walkthru, complete with midi scripts for RSLume):
http://www.resolume.com/forum/viewthread.php?tid=3102
Wiimote to Midi tutorial from AntiVJ:
http://crustea.vjfrance.com/article-130714.html

STEPS:
1) Connect your wiimote to your pc
2) Convert the signals the wiimote sends to MIDI
3) Rock it, rockstar!

STEP ONE: Connect your wiimote to the PC

If you have Bluetooth built into your computer skip to part 2. If you dont, you may take this opportunity to install a ridiculously expensive Bluetooth adapter and construct a delicate web of buggy freeware– PC only, of course.

A. Install your Bluetooth USB adapter (I used a $30 Kensington dongle from Best Buy). I had to install the drivers for a minute, plug the thing in, etc. Eventually you should get that little message that says “your hardware is ready to use”. If not, re-install, unplug it then plug it back in, go to the manufacturer’s website and try to install the drivers there, make sure your adapter is compatible with your PC and with a wiimote– you know the drill. I have heard of people using Bluesoleil software to access their Bluetooth, but I always like to use the software that comes with the device, and I thought Bluesoleil was kinda buggy.

B. Configure your Bluetooth adapter. Once your driver and the Bluetooth adapter is installed, you should see the funky blue Bluetooth logo in your toolbar. Right click on it (or go thru your Start menu) and do a “Explore My Bluetooth Places”. A folder will open up, click on “Bluetooth Setup Wizard”, and you will get this dialog box.Click on the second radio button to “find a specific Bluetooth device and configure how this computer will use its services”. Hit the 1 and 2 buttons simutaneously on your wiimote (switches it to “discoverable mode”) then hit “next” in the dialog box. Watch the flashing blue lights while the wiimote tries to talk to your Bluetooth adapter!

C. Select your Bluetooth device (in this case, a wiimote). Now the dialog box will say “Select a Device” and there will (hopefully) be a picture of your wiimote to select. Select it, then hit “Next!” If your wiimote does not show up, try to re-initiate: hit “search again” in the dialog box, switch to “discoverable mode” on your wiimote (hit the 1 and 2 buttons simultaneously), etc.Once you select your wiimote and hit “next”, you will get to the “pairing” screen. Ignore it– hit “skip pairing”. I tried to initiate pairing, but I couldn’t find a number on my wiimote, except some weird serial number that didn’t seem to do anything. What is pairing? I don’t know. This is taking too long.

D. Configure your wiimote. Now that your computer sees your wiimote, you need to select it. Again. It seems like I needed to constantly switch to “discoverable mode” on my wiimote (hit the 1 and 2 buttons simultaneously) during this entire process.Hit “Finish”. Your Bluetooth logo in your toolbar should turn green. Your wiimote is now connected to your PC!At one point, when installing on a Dell Inspirion5100 this step did not work– I went back, uninstalled, re-installed, and it worked fine. Also, these type of networky softwares seem to work poorly with P2P software such as Limewire and Bittorrent- be warned!

STEP TWO: Convert the signals the wiimote sends to MIDI

Now that your wiimote is talking to your computer, it’s time to configure some softwares to listen to what it says! You will be using GlovePie, a program that was originally created to work with the Essential Reality P5 Glove– classic! This software developed by Carl Kenner has the role of converting any type of signal (from a mouse, Joystick, a P5 glove) into another signal (MIDI, variables…) using a simple scripting language and/or a GUI.

A) Start up GlovePie. I don’t recall if it even needs an install, but I remember having some problems when running it off my desktop, then moving it to my Programs folder, then running it again.Anyway, when you open it up it displays an empty script editor, basically you type in the commands you want your control module (in this case, the wiimote) to execute and hit “Run”. There’s an autogenerate, but I have not really experimented with it yet.Raw coding is intimidating, but fortunately there are loads of scripts provided with the install: for example, go to the “File” dropdown menu, then open “WiimoteScripts/WiiMouse 0.1.PIE”, and click on RUN, and you can now control your mouse cursor with your Wiimote.This is what I used to make sure that the wiimote is actually working– up untill now, the only proof you have that it works is a stupid dialog box. If you run “WiiMouse 0.1.PIE” and your cursor drifts up to the top left of your screen– your wiimote has not been detected properly. Go back to some previous steps, try again! I sometimes use “WiimoteIdentifier.PIE” to check if it sees any wiimotes, but it seems like it can “see” the wiimote without “hearing” the data. If you are dancing your mouse around the screen with your wiimote, you are almost done!

B) Load the Wiimote to Midi script into GlovePie. Antivj.com has the newest version of the script here, or you can copy, paste and RUN the script below. After you copy and paste this code into GlovePie, be sure to save it for later– call it something like “WIItoMIDI.PIE”. Be sure to uncomment/comment out the appropriate sections to disable key commands from your wiimote if you are planning to use it more… unconventionally. This script ( beta0.1), you get the value of X, Y and Z slope value from your Wiimote, which are converted into #CC values 30, 31 and 32, variables from 0 to 127. Sometimes this seems to make sense to me, sometimes not. Are we done yet?
/********************************
Wiimote to midi, beta 0.1
Use your Wiimote to control Ableton live, Arkaos midi..
contact: crustea@gmail.com
www.antivj.com
********************************/
/********************************
Wiimote’s buttons
********************************///Keyboard notes
//To send “keyboard events” instead of “midi notes”, uncomment this part, and comment “Midi notes” part above.
//Values before the “=” caracter are the keyboard events that you emulate./*
Up = Wiimote.Up or Wiimote.Classic.Up
Down = Wiimote.Down or Wiimote.Classic.Down
Left = Wiimote.Left or Wiimote.Classic.Left
Right = Wiimote.Right or Wiimote.Classic.Right
Q = Wiimote.A or Wiimote.Classic.a
W = Wiimote.B or Wiimote.Classic.bOne = Wiimote.One
Two = Wiimote.Two
Home = Wiimote.Home or Wiimote.Classic.Home
Minus = Wiimote.Minus or Wiimote.Classic.Minus
NumPadPlus = Wiimote.Plus or Wiimote.Classic.PlusX = Wiimote.Classic.x
Y = Wiimote.Classic.y
v = Wiimote.Nunchuk.ZButton or Wiimote.Classic.ZL or Wiimote.Classic.ZR
L = Wiimote.Classic.L
R = Wiimote.Classic.R
C = Wiimote.Nunchuk.CButton
*/

//Midi notes
//To send MIDI notes, uncomment this part (done by default), and comment the “Keyboard notes” part
//The value before the “=” caracter is the midi note.

midi2.C0 = Wiimote.Up or Wiimote.Classic.Up
midi2.D0 = Wiimote.Left or Wiimote.Classic.Left
midi2.E0 = Wiimote.Right or Wiimote.Classic.Right
midi2.F0 = Wiimote.Down or Wiimote.Classic.Down

midi2.G0 = Wiimote.A or Wiimote.Classic.a
midi2.A0 = Wiimote.B or Wiimote.Classic.b

midi2.B0 = Wiimote.Minus or Wiimote.Classic.Minus
midi2.C1 = Wiimote.Home or Wiimote.Classic.Home
midi2.D1 = Wiimote.Plus or Wiimote.Classic.Plus

midi2.E1 = Wiimote.One
midi2.F1 = Wiimote.Two
//midi2.A2 = Wiimote.Nunchuk.CButton
//midi2.B2 = Wiimote.Classic.x
//midi2.C2 = Wiimote.Classic.y
//midi2.D2 = Wiimote.Nunchuk.ZButton or Wiimote.Classic.ZL or Wiimote.Classic.ZR
//midi2.E2 = Wiimote.Classic.L
//midi2.F2 = Wiimote.Classic.R

/********************************
Setup for gravity values
********************************/

// these are the magic line, which converts X, Y and Z rotation values to Midi #CC 31, 32 and 33.
var.xOffset = 1
var.yOffset = -28
var.zOffset = 2

var.xRot = Wiimote.RawForceX + var.xOffset
var.yRot = Wiimote.RawForceY + var.yOffset
var.zRot = Wiimote.RawForceZ + var.zOffset

midi2.Control31= (EnsureMapRange(var.xRot, 26,-26, 0,1))
midi2.Control32= (EnsureMapRange(var.yRot, 26,-26, 0,1))
midi2.Control33= (EnsureMapRange(var.zRot, 26,-26, 0,1))

//Old code: (MapRange(var.xRot, 26,-26, 1,127)-1)/127
/********************************
Setup for acceleration values
********************************/
//Just a test for now, comment the line above to disable X acceleration to midi note
midi2.C3 = Wiimote.RelAccX >= 15

C) LAST STEP! Now it is time to check to make sure Glovepie is sending the appropriate data by installing and running MidiOX (a virtual MIDI port, which emulates a MIDI device to use like a “bridge” with your midi softwares).After you install MidiOX, you should see something like this. The blue numbers should be scrolling as you move the wiimote, and green numbers should appear when you click on some of the buttons. If this is not the case, hit “Select MIDI devices to open” (the round blue button that looks like a midi input). I just opened and closed midi ins and outs untill it worked. For more info, check out the MidiOX site.If the numbers are scrolling rapidly, YOU ARE GOOD TO GO! Your wiimote is going into GlovePie, and then into MIDIOX, and from there you can send it to another MIDI device, or use it to control video mixing software.I used the Wiimote as a motion sensor for an interactive golf game in a freight container (http://www.angelsgateart.org/events/fr8.html)My next step it to apply this MIDI input into Resolume and Isadora, and do more conventional video mixing using the motion controls as opacity toggles. Wish me Luck!

Why did you decide to make your own interface?

This all started as a research paper conducted by me and Joao Pitte for a final project for our post-graduate studies, we were supposed to do research on interface design and decided to explore realtime video controllers. I had been vjing for some time and was one of the many who had always felt unhappy with the standard interfaces in live visuals performances. Except for kicking and triggering some clips i never liked much the keyboard and controlling effects with a mouse seemed always too limiting. Midi controllers allowed an infinite number of possibilities nevertheless their format was standardized by/for audio applications.

Another strong point of the project was its resources limits, the technology used had to be available, cheap, durable and responsive to our actions. After researching technology and the vj’ing activity, specifically in controlling video loops, there was a proposed design and a bunch of sketches. I couldn’t keep on going without testing the physical tool, so a prototype had to be made and loop-R was born.

How long have you been working on this project?

The project started in May 2004, the first fully functional prototype was built during the summer of 2004 and presented in december at ShowSkills Festival (uk). Then the project went through some minor hardware revisions, new paintings, some software tweaks and a lot of performances! Though the prototype was finished there was still a lot of tweaks going on and new ideas, too. After moving to New York I started studying again and some other projects took hold of my focus, but in one way or the other Loop-R was always present and I’m now finishing a new model.

Where did you get the parts?

I got most of the parts second-hand, through online sources like eBay. The first version was built from an industrial 3m touchscreen and its metal enclosure and a Doepfer Pocket Dial midi controller. The software system was made using Isadora.

Did the hardware lead the software design, or the other way around?

I guess none did.. from the beginning i saw them as pretty much the same thing. One of my concerns was the barrier between our main point of focus (the screen) and what we’re doing with our hands, there’s a lot of focus-shifting going on and there was a need to end that. Also, natural mappings between the user actions, the controllers and the system were studied according to the media we wanted to control. The elements had to be blended as one, controllers melting over the GUI, no distinction between software and hardware.
All of this contributed to the conceptual model and from that it was just a matter of picking the right hardware and software. Of course, all of this was constrained by not having much knowledge of electronics, little money and even less time.

Are you interested in mass-producing this or selling the idea?

I don’t think i’d say “no” to mass production if i have the chance, nevertheless there’s much involved to do that. Maybe producing it in a small scale… we’ll see. The heart of the project has always been a DIY approach so i’d be really happy if it showed the people that they don’t need one just like this, but they can build their own controllers, their own tools customized precisely for their needs and ideas. There’s a lot of people doing this and there has never been a better time to do it than now. the amount of tools and information we have available to us is amazing, easily accessible and contains huge creative power. The possibilities are infinite!

How easy is it to travel with the loop-r?

Well, lets just say it’s not a laptop…6kg (13 lbs) in an aluminum tool case with all cables, power adapter. plus computer (says my roadie). I’ve travelled over Europe with few problems besides some strange looks from the customs officers.

Who or what inspires you?

This is going to sound geeky but in this project i couldn’t forget Bill Buxton, Fitzmaurice and D.Norman’s researches and writings and give thanks for sharing them. They were extremely influential in my interface design approach. Zachary Lieberman and Golan Levin are amazing creators of expressive interfaces.

Maybe the whole history of artists and creators throughout the visual performance history and synaesthesia pursuit who always looked for more. Currently David Dessens (SanchTV) and his VVVV custom high-resolution multi-projection systems is one of the shows I most want to see. And, of course, a lot of inspiration comes from music!

What do you do for a living?

Interaction-media-creative-researcher-artist-vj-student-boy, currently enrolled in the Interactive Telecommunications Program at Tisch-NYU.

How did you get started with Video Mixing?

I always had a strong passion for music and performance. Though I could never play any musical instrument, I was always making video clips in my mind. By ’97 I got a hold of the VJamm and X-Pose demos and, naturally, got crazy and thought i just had found my tool. By 2000 I had moved to VJing outside my bedroom and never stopped.

Where are you from? What is ‘the scene’ for video mixing like where you are?

I’m from Portugal but i have been living in NYC for almost one year now. In Portugal the scene is growing and becoming more mature. There are lots of new VJs every year with really nice ideas and work going on. Also Music, Cinema and other Arts festivals are acknowledging live visual performance and investing in it, promoting workshops, competitions and performances.

Where do you see video performance heading?

Vjs are building their own software and controllers bringing a lot of innovation to the arena and multi-projection has been the buzzword for the past years. Also, generative graphics and coded visuals are becoming more popular than ever.The interaction with sound has been directing live visuals, not disregarding it, more collaborative interactions with the audience and space are being explored and enriching the experience.

The Playstation 2 Game Controller is a wonderful interface. I debuted this installation at the 2007 Maker Faire, and found that kids and adults alike (but especially kids and 20-something gamers) were able to understand and control this interactive Audio Visual System very quickly.

This system builds upon the work I started here: A/V Synth Controlled by Game Pad, and the graphics were inspired by the Interface Design of Battlestar Galactica. Watch the video above and come back for Nitty-Gritty technical details.

First up is Junxion – software that converts USB to MIDI messages. First, a disclaimer – I bought this back when it was $29, and I think it’s a bit overpriced now (€25 for the basic version and €75 for the deluxe). You can try arj.MultiControl – the last version I used was buggy, but they may have fixed things up since then. In any case – here’s what my Junxion setup looks like:

I have most of the buttons sending CC messages, with the exception of the four circular buttons on the right side of the controller (this would be Triangle, Circle, X, Square on a PS2), which send notes, since they drive the synthesizer. This midi gets sent to my Ableton Live setup:
Track 1 are my two drum clips – I trigger the Play Clip with L1 and R1 and the Track Stop with either L2 or R2. There’s a ‘Stereo Beat Cutter’ Effect – I control the ‘Chance’ parameter of this with my Directional Pad, defaulting to ’0′ when I’m not pressing anything, and jumping to four possible ranges when I press Up, Down, Left or Right.

Track 2 is my Synth. I pitch the notes down 2 octaves and run them through an Arpeggiator to the Joystick preset. The Analog Sticks on the gamepad control ’2nd Pitch’ and ‘Filter’ in the Joystick Synth.

Track 3 is my MIDI-OUT track. Here I take the Arpeggiated MIDI generated by Track 2, and send it out to my IAC Port (the built-in MIDI routing device in the Mac OS). This brings us to:

The Quartz Composer patch to run the visuals. Basically, we have a lot of sprites controlled by LFOs, and the controller (for which you can see my handy Macro Patch on the left here) modifies the LFOs, or changes the Alpha of the sprites to make them appear and disappear. One of the tricks I use is this patch to the right, called ‘MIDI Keys 12′. This was a simple enough idea, pointed out to me by Electronic Music Producer Chiral when I was brainstorming for ways to improve a synthesizer-based patch. Inside is a giant mess, of which you can see a tidbit below.

Basically, I take every possible octave for each note, and run it through a Javascript patch which averages the data from the 10 octaves coming in, and outputs one signal. The end result is a signal that responds to C# no matter which octave you’re playing.

Let me know what you think of this project so far – this is only the first version, and I’ve much more work to do. Ideas I have for the future:

• Left Analog stick should control the tunnel flight, rather than the Directional Pad
• Something should happen when you press Start or Select.
• Third Drum loop? (I could use L2 for a loop, and have R2 be the only ‘Stop’ button)
• The analog sticks can be pressed in for an additional two button clicks – I should use these
• Add multiple controllers – one for drums, one for lead, one for Vox, etc. that all control the same graphic system.
• Your Idea Here.

Downloads

Disclaimer: This is not ‘ready for the public’ at all yet. I’m providing the source code for those who tinker with MIDI routing and interactive Audio/Video. One of my goals is to build a system that will be easy for everyone to download and use, but we’re nowhere near that point.

Ableton Live Project: playsonic_ableton.zip
Junxion Preset: logitech_gamepad.jxn
Quartz Composer Project: playsonicv1.qtz
All 3 of the above in 1 file: playsonicv1.zip

YouTube version: Playsonic:  AV Fun with a Game Controller

THE CHALLENGE

Arm yourself with your favorite video clips – from video sites, email forwards, film compilations and dusty old VHS tapes. Pit your choices against a friend’s in a head-to-head battle of Top This! played in realtime on our custom-built Rumbletron. If you’re playing with a friend, you can decide on a category beforehand – perhaps music videos, or the 80′s, or videos with the color red – and challenge each other to put together a better collection by the time the Maker Faire rolls around.

WHAT TO BRING

Bring at least five of your best Internet Video gems on any sort of removable media – CD, DVD, External HD, USB Flash Drive, etc. Prep them using the tech specs at the bottom of this document.

HOW TO PLAY

Take a look at the video above to get an idea of how it will work. Here are the steps for clarification:

• Each player loads their videos into the interface.
• Player 1 begins by playing a clip.
• Player 2 observes the video and queues an appropriate follow-up video by pressing the corresponding button on the NuVJ.
• When Player 1′s video is done, s/he flicks the crossfader all the way over to Player 2, which automatically triggers the clip that Player 2 has queued earlier.
• Now Player 1 watches Player 2′s clip and picks a follow-up, queuing it via button-press.
• When Player 2′s video is done, s/he flicks the crossfader back to Player 1, triggering the queued movie, and watching so s/he can pick a follow-up
• Play continues in this matter, crossfade and queue, crossfade and queue, until each player has shown 5 videos.
• After the last video, players should perform a victory dance.

CONTENT GUIDELINES

There will be innocent young eyes roaming the fairgrounds, taking in all the amazing sights. Let your conscience be your guide, yeah? In case of warped conscience, avoid the following: realistic blood/gore, p0rn, swearing, hate speech, Dark Crystal (scared the crap out of me as a kid).

More details coming – Sign Up as a VJ Kung Fu Contributor to stay in the know.

TECH INFO

Your clips should be at between 10 and 60 seconds long, and compressed using the following settings:

640×480 Pixels
Motion-JPEG B Codec
29.97 FPS
PCM Audio

You cannot edit start points once they have been loaded, so do that beforehand with your favorite editing program.

BANNERS

Embeddable YouTube Video:

<span class=”mceItemObject” width=”425″ height=”350″><span name=”movie” value=”http://www.youtube.com/v/KBVG7o1Ta-s” class=”mceItemParam”></span></param> <span class=”mceItemEmbed” xsrc=”http://www.youtube.com/v/KBVG7o1Ta-s” mce_src=”http://www.youtube.com/v/KBVG7o1Ta-s” type=”application/x-shockwave-flash” width=”425″ height=”350″></span></span>