Watch in HD on Vimeo

I was invited by Francis Tseng of Golden Ages to create audio-reactive visuals for the band’s concerts.

The visuals revolve around a rotating spiral thing made of balls and lines, inspired by ctenophora and zooplankton. There are 7 “scenes”, one for each song in their set list. With each scene, the visuals becomes more distorted and more agitated. It was built with Processing and reacted to the sound of the band fed to the computer during the concert. The post-processing effects (blurring, glowing, displacement) were done with GLSL shaders. Above is a screen capture of this audio visualization fed with the MP3 of Everything Will Be Alright (official video).

This was a fun project and I really like how it turned out. Looking forward to work on more projects of this kind.

Watch the HD video on Vimeo.

Or click here for the realtime version (Java applet). A good graphics card is required.

Realtime graphics made with Processing, toxiclibs for general vector math and Minim for audio playback. The following links were very helpful:

• Algorithm for even distribution of points on a sphere from CGA FAQ wiki, used for uniformly distributing tentacles over a spherical body’s surface.
• Introduction to Catmull Rom Splines by Robert Dunlop, used for interpolating camera’s position.
• Per-pixel lighting GLSL shaders from Lighthouse3D.
• Billboard particles code from this OpenFrameworks forums thread.
• Solution for finding the normal of a point in a sine wave from this Gamedev forums thread. I had to solve this problem to calculate the normals of the moving tentacles, because each tentacle is a cylinder deformed by a vertex shader with a sine wave. I need to learn calculus. :-)

I feel that I haven’t explored 3D graphics in my projects so far, so that was one thing I wanted to do this time. But 3D graphics implies lighting, texturing, shadows, which are all difficult problems to solve. In the end, I only used one directional light and wasn’t able to implement shadows nor ambient occlusion.

Another important thing was sequencing. One person on Vimeo commented that untitled 246 lacked some sort of ending, and I agreed. Even abstract visual graphics can benefit from some sense of storytelling. I think this time there are too many things happening in the ending scene, but not much changes between the first and the second sections of song, so there’s an imbalance.

Conclusion: I think I tried to build something too big (or too complex) for my current OpenGL skills. Have to keep studying. Next time I’ll probably try something a bit simpler, though. :-)

PS: 2010 has been a busy year, I’ve worked on some interesting Flash development projects, but none of them are public yet. Hope to be able to show them soon!

A video for a song. Code and music by me.
(Note: music is not realtime generated, only the graphics).

Video made with Processing, toxiclibs, penner.easing lib, Robert Hodgin’s perlin noise movement, and Paul Bourke’s 2D line intersection algorithm.

Music made with FL and MDA JX10.

Instead of running a “real” audio analysis with SoundMixer.computeSpectrum, I just measured the sound’s intensity level with SoundChannel.leftPeak and SoundChannel.rightPeak.

The SWF file was compiled with Flex SDK 3.3 and weighs 3.985 bytes (PBJ data included).

Music: “A” by Turzi.

Concept and development by Fabiano Onça and colmeia.

Made with OpenFrameworks.

This is a quick experiment made with vvvv.

Just spinning spheres and moving textures.

Continuing my GLSL studies — now with dynamic lighting (vertex shader) and texturing (fragment shader).

This application runs at 35fps at 1280×720 pixel resolution on my ATI HD 4850 card. There are 1024 tentacles, drawn with gluCylinder — I guess this can still be optimized.

Made with OpenFrameworks v0.06 and GLFW.
Music: “M” by Telefon Tel-Aviv.

Replaced the small spheres by cones distributed over the big sphere’s surface. Blur amount is controlled with the mouse.
Made with OpenFrameworks.
Music: “My Two Nads (Dad Reprise)” by Out Hud.

Lightning the scene with the vertex shader.
Made with OpenFrameworks.
Music: “Trash Scapes” by Ellen Allien.

My first GLSL -based animation. The blob is made of 3 glutWireSpheres deformed by applying a vertex shader. Deformation is affected by the music and mouse coordinates. In the code snippet below, fft is a value taken from ofSoundGetSpectrum and mult is the pair of mouse coordinates (x, y).

#version 120
uniform float fft;
uniform vec2 mult;
uniform vec3 col;
varying vec4 v;
void main()
{
    v = vec4(gl_Vertex);
    vec4 v2 = v;
    v.x += cos( v2.z * mult.x ) * fft;
    v.y += sin( v2.x * mult.y ) * fft;
    v.z += cos( v2.y * mult.x ) * fft;
    gl_Position = gl_ModelViewProjectionMatrix * v;
}

Check the high-res images at Flickr .
Made with OpenFrameworks .
Music: “Geek Down” by J Dilla.

I haven’t released any typefaces since 2006 but I still draw letters all the time — I always like to have a piece of paper and a pen or pencil at hand. Today I scanned and uploaded some sketches to my Flickr account.

P.S.: Just finished doing some blog maintenance:

• deleted “Playground” category
• created Processing and OpenFrameworks categories
• deleted trackback spam
• added links to my Blip.fm, FriendFeed and Twitter profiles
• added Facebook blog network widget.

—

After making some tests with Java OpenGL (JOGL) in Eclipse, I decided I should try OpenFrameworks again to delve deeper into OpenGL. The main problem with JOGL is that it’s a “layer” on top of the original OpenGL API; some things are different and often more complex in JOGL (for example the need to pass ByteBuffers instead of arrays).

This audio visualization is a simple exercise with FBOs, VBOs and shaders. An audio-reactive particle system (512 quads) is rendered to a FBO and applied as texture to a cube. This cube is then rendered to a second FBO, which is applied as texture to a second cube. There is a slight blur effect made with GLSL shaders. The stripes result from the way the cube was UV-mapped, an unexpected effect that turned out interesting.

At a 640×480 pixels resolution, framerate varies from 30 to 60 FPS in my computer, equipped with a GeForce 6200 (I need to upgrade my PC).

Happy 2009 to all! :-)

Just finished a simple Processing sketch for the Processing Monsters project by Lukas Vojir. He invited Processing developers to create and share (source code included) little black-and-white, mouse reactive monsters — there are some cool entries already, go check them out.

—

After watching Moscow Olympics’ Second Trace video, Errol from KittyWu Records invited me to make a video for Singaporean band I Am David Sparkle.

Jaded Afghan is one of the best tracks on their 2007 album This Is The New and also a song well suited to my beat detection algorithm (based on this gamedev.net article — look for “frequency selected sound energy algorithm #2”).

Inspired by this thread (especially comments from Chris O’Shea and Dave Bollinger), this Processing sketch has two modes: capture and render. In capture mode, there’s audio playback, FFT analysis and beat detection — the resulting data is saved to a text file. In render mode, there’s no audio: all data is retrieved from the text file and PNG still frames are rendered. It took approximately 1h30 to render the 6779 frames for this video, at 800×450 pixels, 24FPS. The movie file with audio was assembled in Adobe Premiere.

It’s highly recommended to download the original 170MB Quicktime file — the link is at the bottom of the page, below “Statistics”. A Vimeo account is required, but it’s free.

—

Libraries and code snippets used in this project:

• geomutils by Karsten Schmidt
• _gl.pde from Robert Hodgin
• Minim by Damien Di Fede
• mouseWheel.pde by Marius Watz
• OCD by Kristian Damkjer
• Traer Physics by Jeffrey Traer

Game designer Fabiano Onça conceived the game, in which people must fill geometric shapes with their own silhouettes (as captured by webcams hanging from the ceiling):

Tantalus Quest at FILE 2008 from eduardo omine on Vimeo.

Software was built with OpenFrameworks, which is to C++ what Processing is to Java. A prototype was built with Flash (AS3), but it was slow — reading pixel values (BitmapData.getPixel) can be processor-heavy. Thanks to OpenFrameworks, porting the AS3 code to C++ was quite easy.

The application is very simple: the images captured by the cameras are brightened, blurred and thresholded, resulting in black blobs. The amount of blob pixels inside the geometric shape count as positive points and the pixels outside the geometric shape count as negative points.

This was my first project with computer art in a physical space — it’s something that I should explore further in my personal projects.

More pictures at Flickr.

More Context-Free images at Flickr

—

This is my first music video; made with Processing + Ess library.

The song is Second Trace by Filipino band Moscow Olympics, from their recently released debut album Cut The World. Although the band’s musical references are clear (post-punk, shoegaze), their music has that ineffable quality that makes it stand out.

—

It seems that computing a Voronoi diagram with mathematical accuracy for a real-time animation is not really feasible. I found this Processing hack that takes advantage of graphic acceleration to draw an approximate Voronoi diagram. It works by drawing 3D cones at the points of interest of the diagram and rendering a top view — the regions are drawn automatically as the Voronoi edges sit on the intersections between the cones.

Using this hack, it is possible to make a real-time Voronoi diagram animation, albeit with a limited number of points and a restricted applet area. This video was rendered with 364 particles (= sum of powers of 3 from 0 to 5).

—

Voronoi diagrams display areas of influence of a set of points in a plane. And they can also be used to create beautiful, organic images.

But even though constructing a Voronoi diagram geometrically seems easy, computing it efficiently can be quite hard. I’m still trying to figure out both Fortune’s and Bowyer-Watson algorithms, but as a first and quick exercise I implemented a naive algorithm with Processing and generated a short animation.

The following demo is a simple implementation of Cover Flow that takes advantage of the 2.5D API — all DisplayObjects now have x, y, and z properties (position, rotation, scale).

Use the arrow keys to navigate:

I developed a Cover Flow clone with Papervision3D 1.5 for the Quatro Arquitetura website a few months ago, and adapting it to the new FP10 API was really easy. Download the source code. You’ll also need Tweener.

Sound analysis was made with SoundMixer.computeSpectrum(), FFTMode set to true (a frequency spectrum results in a more interesting visualization) and stretchFactor set to 4. Increasing the stretchFactor value means analyzing a smaller frequency range of the sound —thus ignoring non-audible high frequencies. The 16×16 vertex grid represents the average values of left and right channels.

Music Is Music As Devices Are Kisses Is Everything (5:20) by 65daysofstatic (from the 2007 album The Destruction Of Small Ideas) is the song; a nice fit for this visualization because of the contrasting quiet and loud parts —the visualization of the piano notes in the quiet sections is very clear.

View SWF (5MB, move mouse on X direction to change rotation, music plays 10 times)

Update: comments closed because of spam.

Update #2: this experiment was featured on PV3D blog — 9 Cool Experiments in 3D.

The final SWF runs smoothly at 30fps on my computer, and initially uses about 5MB of memory as measured by System.totalMemory. The biggest challenge in this project was fixing a memory leak that occurred whenever a ribbon was removed from the scene. If you run into this problem when removing a DisplayObject3D, remember to:

• reset the geometry.faces and geometry.vertices arrays;
• unregister the object from its material.

public function destroy():void
{
    geometry.faces = [];
    geometry.vertices = [];
    material.unregisterObject(this);
}

Each ribbon has 3 life phases:

1. Growth: the ribbon grows until it reaches 120 vertices;
2. Stability: the ribbon keeps moving but older faces and vertices are removed, keeping vertex count equal to 120;
3. Removal: the ribbon stops moving and 2 vertices and 2 faces are removed frame by frame until it reaches zero vertices and zero faces.

The applied material is a simple FlatShadeMaterial with a moving light source. Finally, a BitmapLayerEffect with BlurFilter was applied and the viewport.blendMode was set to BlendMode.ADD.

View animation (move mouse to change camera rotation)

The pseudo swarm behaviour of the particles is obtained by combining the movement of the visible particles with the movement of a non-visible, reference particle.

Interestingly, the animation performed much better when I unified all Event.ENTER_FRAME handlers in one single place. Originally I made each particle and each trail listen to Event.ENTER_FRAME, removing the event listeners when destroying the particle. Now I only listen to Event.ENTER_FRAME in the Main class, and then loop through all instances to call each instance’s update() method.

The background color changes with time; one cycle should take 1 minute to complete. You can also click the background to toggle full screen mode.

View SWF