(the teeming void)

Generative Heritage - notes on Succession

noreply@blogger.com (Mitchell) — Thu, 04 Dec 2014 02:57:00 +0000

I spent three months this year on sabbatical at Culture Lab, Newcastle University (UK). It was a privelege to spend time in such a vibrant research lab, as well as to get to know the city of Newcastle. One of the projects to come out of my visit is Succession, an experiment in generative digital heritage that uses Newcastle and its history to think about industrialisation, global capital, our shared pasts and potential futures. Personally, it brings together two strands of my work that have been separate until now - on generative systems and digital cultural collections. Hence you'll also find this cross-posted over on The Visible Archive. Here, some notes and documentation on the work, some musings on generative and computational heritage.

Much of my recent work with digital cultural collections has worked to create rich representations of these ever-expanding datasets. A key thread has been an interest in the complexity of these collections; the multitudes they contain, their wealth of potential meaning as complex, interrelated wholes, rather than simply respositories of individual resources. Visualisation can provide a macroscopic view of this complexity, but it can be just as vivid when sampled at a micro scale. Tim Sherratt's Trove News Bot tweets digitised newspaper articles in response to the day's news headlines, creating little juxapositions, timely sparks of meaning that can be pithy, funny, or provocative. Trove News Bot appropriates the twitter bot - the joking-but-deadly-serious computational voice of our age - and adapts it to work with the digital archive. We could call this generative heritage; using computational processes to create new artefacts (and meanings) from historical material.

Succession applies this generative approach to the digital heritage of Newcastle Upon Tyne. Newcastle has a rich industrial heritage; it played a major role in the Industrial Revolution that began in Britain and went on to remake global civilisation. Today Newcastle is a post-industrial or de-industrialised city: coal, steel and shipbuilding have given way to service industries: education, retail, entertainment and tourism. As an outsider exploring the city I was struck by the mixture of pre-modern, industrial and post-industrial eras in the fabric of the city. Different (often inconsistent) patterns of life, work and economy are accreted in layers as the city continues the everyday process of adaptation, experimentation with the possible; working out what comes next.

The city, like the digital archive, is a multitude; an unthinkably complex matrix of people, things, systems, narratives. Newcastle - more than many other cities - also speaks to the expansive dynamics of industrialisation, globalisation, extractive industry, fossil fuels; the whole modern trajectory that has brought us to our current predicament. This seems to be both urgent and unthinkable - or perhaps, unsayable. How can we speak back to this complexity; how can we make in a way that responds to this tangled, expansive mess? Here generative techniques offer a way to synthesise complexity and create multitudes, formations that might portray the city as it was, or hint at what it could be. Automatic juxtaposition and remix create nonsense but also, occasionally, glimmers of a new sense, or at least a texture or sensation that emerges from a random constellation of images, sources and contexts. Succession requires us to piece together fragments of history; and this is a work of imagination, as Ross Gibson writes, framing his own work of generative heritage (with Kate Richards, Life After Wartime

Our parlous states need imagination. We need to propose “what if” scenarios that help us account for what has happened in our habitat so that we can then better envisage what might happen. We need to apprehend the past. Otherwise, we won’t be able to align ourselves to historical momentum. Without doing this we won’t be able to divine the continuous tendencies that are making us as they persist out of the past into the present.

In practical terms, the work is based on a corpus of around two thousand images sourced from the Flickr Commons. Most come from the (wonderful) Tyne and Wear Archives and Museums collection; many more from the Internet Archive Books collection, with a smattering of others from UK and international institutions. Succession uses these ingredients to generate new digital "fossils"; composite images assembled in the browser using HTML Canvas. This generative process is extremely simple: pick five sources at random, and place them in the frame using some semi-random rules for positioning, compositing and repetition. Opacity is kept low, so that the sources blend and merge. The visual process often obscures the source images - they end up buried, cropped or indistinguishable, squashed like fossil strata. But at the same time the source items are preserved and presented in context, so each composite retains references to its sources and their attendant contexts. Composites can be saved, acquiring an ID and permalink; the images in this post show some of my favourites, but there are over a hundred to sift through already.

As a generative system this is, in formal terms, incredibly simple. It's essentially a combinatorial process, in that each composite consists of five elements from a set of around two thousand. Yet already this adds up to 2.5 x 10^15 unique combinations - it would take eight million years to see them all, at one per second. Compositing and layout parameters are random within constraints - so this simple machine can produce an immense variety of unique results; I'm still surprised and delighted by the fossils people discover (or generate). But this computational variety is also strongly shaped by the human creative choices involved in making the work. This is what Bill Seaman (combinatorial media artist par excellence) calls "authored space" - a domain of potential that is expansive but never arbitrary. The corpus reflects a handful of coherent themes, seasoned with generous sprinklings of the lateral and miscellaneous; the aim is, in Seaman's words, a kind of "resonant unfixity." Also the corpus and the compositing process work in tandem; for example compositor treats the largely monochrome line-art and engravings of the Internet Archive material differently to other (largely photographic) sources. The generative machine is programmed in part by the textures and qualities of its material.

The Internet Archive book images are interesting on several fronts; for one, they are an amazing demonstration of the power of computational processes for generating and describing large collections (like 2.6 million items large). Given the right kind of source material, this computational leverage changes the logic of collections completely. When adding and describing items is expensive, it makes sense to be selective, and publish only what is most "significant". Automation makes it possible to simply publish everything - for who's to say (really) what is significant, or how it might one day be significant? In Succession the Internet Archive material plays a crucial role. The line art and diagrams - many from obscure publications like the Transactions of the North of England Institute of Mining and Mechanical Engineers - offer evocative fragments of the machinery of mid-nineteenth century industrialisation.

As for generative digital heritage, it's a fairly open-ended proposal. What happens when we turn algorithms loose on our digital culture with makerly, synthetic, speculative or poetic intention? There are some pretty solid precedents in the digital humanities for these approaches; Schnapp and Presner call for a "generative" DH in their 2009 manifesto. Before that Drucker and Nowviskie outlined a "speculative computing" with a strongly generative flavour. Gibson and Richards' Life After Wartime is an early exemplar of generative heritage in the digital arts. More recently we've seen the rise of massive online collections, web-scale computing, and a proliferation of cultural, critical and creative bots, not to mention projects like #NaNoGenMo. If there is such a thing as generative digital heritage, then now's the time.

Proto-Computing - an Interview with Ralf Baecker

noreply@blogger.com (Mitchell) — Sat, 08 Jun 2013 01:24:00 +0000

At CODE2012 I presented a paper on "programmable matter" and the proto-computational work of Ralf Baecker and Martin Howse - part of a long-running project on digital materiality. My sources included interviews with the artists, which I will be publishing here. Ralf Baecker's 2009 The Conversation is a complex physical network, woven from solenoids - electro-mechanical "bits" or binary switches. It was one of the works that started me thinking about this notion of the proto-computational - where artists seem to be stripping digital computing down to its raw materials, only to rebuild it as something weirder. Irrational Computing (2012) - which crafts a "computer" more like a modular synth made from crystals and wires - takes this approach further. Here Baecker begins by responding to this notion of proto-computing.

MW: In your work, especially Irrational Computing, we seem to see some of the primal, material elements of digital computing. But this "proto" computing is also quite unfamiliar - it is chaotic, complex and emergent, we can't control or "program" it, and it is hard to identify familiar elements such as memory vs processor. So it seems that your work is not only deconstructing computing - revealing its components - but also reconstructing it in a strange new form. Would you agree?

RB: It took me a long time to adopt the term "proto-computing". I don't mean proto in a historical or chronological sense; it is more about its state of development. I imagine a device that refers to the raw material dimension of our everyday digital machinery. Something that suddenly appears due to the interaction of matter. What I had in mind was for instance the natural nuclear fission reactor in Oklo, Gabon that was discovered in 1972. A conglomerate of minerals in a rock formation formed the conditions for a functioning nuclear reactor, all by chance.

Computation is a cultural and not a natural phenomenon; it includes several hundred years of knowledge and cultural technics, these days all compressed into a microscopic form (the CPU). In the 18th century the mechanical tradition of automata and symbolic/mathematical thinking merged into the first calculating and astronomical devices. Also the combinatoric/hermeneutic tradition (e.g. Athanasius Kircher and Ramon Llull) is very influential to me. These automatons/concepts were philosophical and epistemological. They were dialogic devices that let us think further, much against our current utilitarian use of technology. Generative utopia.

Schematic of Irrational Computing courtesy of the artist - click for PDF

MW: Your work stages a fusion of sound, light and material. In Irrational Computing for example we both see and hear the activity of the crystals in the SiC module. Similarly in The Conversation, the solenoids act as both mechanical / symbolic components and sound generators. So there is a strong sense of the unity of the audible and the visual - their shared material origins. (This is unlike conventional audiovisual media for example where the relation between sound and image is highly constructed). It seems that there is a sense of a kind of material continuum or spectrum here, binding electricity, light, sound, and matter together?

RB: My first contact with art or media art came through net art, software art and generative art. I was totally fascinated by it. I started programming generative systems for installations and audiovisual performances. I like a lot of the early screen based computer graphics/animation stuff. The pure reduction to wireframes, simple geometric shapes. I had the feeling that in this case concept and representation almost touch each other. But I got lost working with universial machines (Turing machines). With Rechnender Raum I started to do some kind of subjective reappropriation of the digital. So I started to build my very own non-universal devices. Rechnender Raum could also be read as a kinetic interpretation of a cellular automaton algorithm. Even if the Turing machine is a theoretical machine it feels very plastic to me. It a metaphorical machine that shows the conceptual relation of space and time. Computers are basically transposers between space and time, even without seeing the actual outcome of a simulation. I like to expose the hidden structures. They are more appealing to me than the image on the screen.

MW: There is a theme of complex but insular networks in your work. In The Conversation this is very clear - a network of internal relationships, seeking a dynamic equilibrium. Similarly in Irrational Computing, modules like the phase locked loop have this insular complexity. Can you discuss this a little bit? This tendency reminds me of notions of self-referentiality, for example in the writing of Hofstadter, where recursion and self-reference are both logical paradoxes (as in Godel's theorem) and key attributes of consciousness. Your introverted networks have a strong generative character - where complex dynamics emerge from a tightly constrained set of elements and relationships.

RB: Sure, I'm fascinated by this kind of emergent processes, and how they appear on different scales. But I find it always difficult to use the attribute consciousness. I think these kind of chaotic attractors have a beauty on their own. Regardless how closed these systems look, they are always influenced by its environment. The perfect example for me is the flame of a candle. A very dynamic complex process communicating with its environment, that generates the dynamics.

MW: You describe The Conversation as "pataphysical", and mention the "mystic" and "magic" aspects of Irrational Computing. Can you say some more about this a aspect of your work? Is there a sort of romantic or poetic idea here, about what is beyond the rational, or is this about a more systematic alternative to how we understand the world?

RB: Yes, it refers to an other kind of thinking. A thinking that is anti "cause and reaction". A thinking of hidden relations, connections and uncertainty. I like Claude Lévi-Strauss' term "The Savage Mind".

Figuring Data (Datascape Catalog Essay)

noreply@blogger.com (Mitchell) — Tue, 04 Jun 2013 06:21:00 +0000

This essay was commissioned for the exhibition Datascape, at the Cube Gallery, QUT in April 2013. I should mention that since writing it I've discovered that Jer Thorp was way ahead of me on to the new oil thing.

“Data is the new oil” - Ann Hummer, Hummer-Winblad Venture Partners (source)

In the swirling chaos of twenty-first century capitalism, everybody wants to know what’s next. “Data is the new oil” is a pithy little announcement. It reminds us how we got here, powered by the long energetic boom of fossil fuels, now entering its closing stages. it announces a successor, a new wealth (and just in time). But in drawing the analogy, it also constructs data in a certain way; as a sort of amorphous but precious stuff, a resource for exploitation, and a sort of promising abundance. Similarly The Economist trumpeted the “Data Deluge” on their February 2010 cover: a businessman catches falling data in an upside-down umbrella, funnelling it to water a growing flower whose leaves are hundred dollar bills.

We need not (and should not) accept this analogy; but it demonstrates how data is figured, or constructed, in our culture. Our everyday life and culture is traced, tangled and enabled by digital flows. We produce and consume data as never before. But what exactly is this data? What can it do, and what can we do with it? Who owns or controls it? How can we understand, appreciate, or even sense it? The construction of data as a cultural actor is vital because data itself is so abstract, so hard to pin down. We ought not leave it to the captains of industry, and their upside-down umbrellas. In Datascape we see artists working with data, applying and diverting it for their own ends, as well as offering their own figurations of its potentials and limits. In a culture increasingly built on data, these works provide moments of cultural introspection, reflections on this abstract stuff that is our new social medium.

Google, Facebook, Twitter and the rest make us - their users - into data. This makes us anxious about privacy and surveillance, but perhaps a more interesting question is what it’s like to be data. If we are all data subjects now, then what is data subjectivity? Jordan Lane’s Digital Native Archive imagines a new bureaucratic archive for the data subject, and immediately comes to the question of mortality. If we are data, and data can be faithfully preserved, are we now immortal? Or are we, instead, dead forever, entombed in a rationalised hierarchy of metadata, request protocols and archival record formats? Christopher Baker’s My Map (below) shows us what it might be to take charge of a personal archive, with a tool that reveals the patterns and relationships in email correspondence. This self-portrait suggests that one of the challenges of data subjectivity is simply knowing oneself: the scale of our personal data exceeds our grasp.

In two of the most prominent data art works from the mid 2000s, we mine these personal archives en masse. Golan Levin’s The Dumpster and Sep Kamvar and Jonathan Harris’ We Feel Fine scour the internet for “feelings” that are compiled into datasets, and in turn staged as dynamic visualisations. In turning our digital selves into swarming dots and bouncing balls, the artists animate us as members of a teeming throng. Data here is in part a new form of social realism, a way to represent the complex texture of life in the crowd; but these works also ask us to reflect on the limits of data-subjectivity. Can the intensity of our inner lives really be represented in cool, abstract data? Are we all so much alike? Aaron Koblin’s Sheep Market answers both yes and no; for we can see here both the comical diversity of the crowd (and its sheep avatars), and the uniformity that digital systems encourage.

The pathos of this contrast, between the coolness of the digital and the warm, messy intensity of humankind, emerges again in Luke du Bois’ Hard Data, where the tolls of war unfold as stark lists and map references. Du Bois’ soundtrack, generated from the same source data, acts as an emotional mediator, trying to return some of the tragic importance that the data fails to convey. Du Bois’ work pivots between the data-subject and what we might call the data-world. For if the world, too, is now data, then what might that feel like? How do we approach such a world?

In many works here the weather - a complex (and increasingly uncooperative) material flux - is a sort of proxy for the data-world: a field that is both easy to measure, and difficult to grasp. In Miebach’s Weather Scores, Viegas and Wattenberg’s Wind Map (above), and my own Measuring Cup, weather data is a source of aesthetic richness, as well as a pointer to the world beyond, the world that data traces. The weather - so much part of our everyday sensations - is abstracted here into numbers and symbols, only to be remade in new sensual forms. What if we could see the wind across an entire continent? Or hold a hundred years of temperature? Or hear the tides as music?

Here we get a glimpse of an alternative figuration of data itself. Rather than some kind of precious (but immaterial) stuff, or fuel for market speculation, data here is a relationship, a link between one part of the world with another, and a trace that can be endlessly reshaped. Of course, that trace is imperfect; a mediated pointer, not a pure reproduction. So Viegas and Wattenberg issue a disclaimer for their Wind Map: this is just an “art project”, they say; we "can't make any guarantees about the correctness of the data or our software.” Yet that connection remains; and art here plays the role that it always has. It transforms our understanding of the world, by representing it anew.

Local Colour: Smaller World Network

noreply@blogger.com (Mitchell) — Tue, 07 Feb 2012 02:47:00 +0000

Back in September I showed a little work called Local Colour at ISEA 2011. This project continues my thinking about generative systems, materiality and fabrication. It's a work in two parts: the first is a group of laser-cut cardboard bowls, made from reclaimed produce boxes - you can see more on Flickr, and read the theoretical back-story in the ISEA paper. Here I want to briefly document the second element, a sort of network diagram realised as a vinyl-cut transfer. The diagram was created using a simple generative system, initially coded Processing - it's embedded below in Processing.js form (reload the page to generate a new diagram).

Network diagrams are one of the most powerful visual tropes in contemporary digital culture. Drawing on the credibility of network science they promise a paradigm that can be used to visualise everything from social networks to transport and biological systems. I love how they oscillate between expansive significance and diagrammatic emptiness. In this work I was curious to play with some of the conventions of small world or scale-free networks. A leading theory about how these networks forms involves preferential attachment: put simply it states that nodes entering a network will prefer to connect to those nodes that already have the most connections. In visualising the resulting networks, graph layout processes (such as force direction) use the connectivity between nodes to reposition the nodes themselves; location is determined by the network topology.

This process takes the standard small-world-network model and changes a few basic things. First, it assigns nodes a fixed position in space. Second, it uses that position to shape the connection process: here, as in the standard model, nodes prefer to connect to those with lots of existing connections. But distance also matters: connecting to a close node is "cheaper" than connecting to a distant one. And nodes have a "budget" - an upper limit on how far their connection can reach. These hacks result in a network which has some small world attributes - "hubs" and "clusters" of high connectivity - but where connectivity is moderated by proximity. Finally, this diagram visualises a change in one parameter of the model, as the distance budget decreases steadily from left to right. It could be a utopian progression towards a relocalised future, or the breakdown or dissolution of the networks we inhabit (networks in which distance remains, for the time being, cheap enough to neglect).

The process running here generates the diagram through a gradual process of optimisation. Beginning with 600 nodes placed randomly (but not too close to any other), each node is initially assigned a random partner to link to. Then they begin randomly choosing new partners, looking for one with a lower cost - and cost is a factor of both distance and connectivity. The Processing source code is here.

An Interview with Paul Prudence (for Neural 40)

noreply@blogger.com (Mitchell) — Mon, 09 Jan 2012 01:24:00 +0000

Yet another blog "paste" to provide some semblance of life around here. Paul Prudence recently interviewed me for Neural 40: The Generative Unexpected, ranging over generative art (utopian and otherwise), cross modal AV and data visualisation among other things. Thanks to Paul for some thoughtful questions.

PP: It might be argued that some of the main themes infused in generative art are those to do with a kind of techno-utopianism and futurism. Have you come across any generative artworks that deal with dystopian themes or have a sense of anachronism about them? More importantly are the technologies and software used in creating these artworks inherently defining their aesthetics?

It's true that there's a flavour of the techno-utopian to a lot of digital generative art, especially in the online digital scene. The founding principle of generative art is, inescapably, the generative capacity of its own system, so perhaps it is optimistic by definition? Online culture - or the realtime social media flow of projects, memes and links that we tend to bathe in - is also techno-utopian at its core, still strongly influenced by the West-Coast startup culture of the companies involved. But with a bit of digging some more diversity emerges; the work of my friend Jon McCormack for example, is highly reflective about the nature / technology relationship - though it sometimes conceals its ambivalence under a very beautiful surface. Another Australian artist - Murray McKeich - makes work that is both anachronistic and dystopian, like his pZombies, gruesome avatars for generative agency composited from scanned rubbish.

On the other hand the flipside of techno-utopia is real richness and generative excess - the ability of formal systems to reveal terrains of sublime complexity. At best this "maximalist" strand of generative practice can induce a state of wonder, little chinks of access to the unthinkable complexity of the real material world.

Do the technologies define aesthetics? They certainly shape the aesthetics powerfully - but at least now the field of technology is more open and malleable for artists than ever before. It might be that the most important new works in this field are coding platforms or communities, rather than art or design projects. Processing won a Golden Nica, after all. But in this field monolithic "technologies" are increasingly breaking down - Processing for example is very influential, and there is certainly a Processing "look", but with a new framework or library appearing every other week, we can't blame technology for limited diversity in the field.

PP: Much generative art is concerned with certain kinds of abstraction and systematised multiplicity of form without a framework of proposition, resolution and conclusion. Do you think there is any room for a sense of narrative in generative art? Could you give me examples of generative artworks that deal with narrative successfully?

I would argue that every generative artwork involves a framework of proposition, resolution and conclusion. It is the formal and procedural structure of the generative system that creates the work: a set of entities, attributes, relationships, processs, rules, constraints, and visualisations (more here). The problem, for the way generative art is both made and received, is that that system is often hard to get at - it's an abstract thing, which the artist may or may not describe or publish. A lot of work in the digital generative scene operates in an image culture where "look" is valued over process or concept. So although it's sometimes hard to access, I would argue that there is often a narrative inside even the most "retinal" generative art - it's the narrative of the system. Sometimes it's fairly clear - for example Brandon Morse's wonderful procedural animations of collapsing structures (also another dystopian work!). For me Morse's work is wonderfully poignant because it works by resemblance - it reminds us of real things collapsing - but it also works by metonymy, referring to the idealised world of computer graphics and simulation; so it seems like the simulation itself is collapsing as well (below: Achilles (2009) - photo by Paul Prudence).

PP: Each year we see different algorithms come into fashion as tools for the generative artist. Perlin Noise, Circle Packing, Voronoi, Reaction-Diffusion and Sub-divisioning algorithms are good examples. How important is it for an artwork to hide traces of the software and algorithm that was used to generate it it? Can you predict what the next big algorithm might be? Or do you see any new potential in an old or overlooked algorithm?

If you need to hide the traces of your algorithm, change your algorithm. I too am fascinated by the algo-memetic fashion parade that moves through digital design and generative art. This relates to the question of look vs system; these systems seem to reproduce using their appearance as a sort of lure - it's a bit like sexual selection in a memetic ecology, survival of the prettiest. As a result people seem to apply them without any understanding or interest in the system or process. I wrote last year about the Voronoi algorithm along these lines. So algo-fashions will come and go, but for me the most rewarding work is always a result of deep engagement with the generative system - taking a system and hacking it into something else entirely, or deriving new systems. Erwin Driessens and Maria Verstappen for example have a long track record of inventing algorithms that you can't just grab off the shelf - their Breed and Ima Traveller works are sort of mutant cellular automata - but really they don't fit any clear template. Nervous System also implement new systems: they go to the scientific literature in biology, or even run their own physical trials, and implement models from scratch. There aren't many designers currently with the ability to do that. Jonathan McCabe is another good example of this; his multi-scale Turing patterns (below) are a genius hack of a very old algorithm. Jonathan's Origami Butterfly process is completely new (and equally distinctive).

So there isn't a Platonic shelf somewhere stocked with generative algorithms for designers to select from. The space of potential generative systems is unimaginably massive. Make one up, or at least hack an existing one into something else. Even very simple changes to existing systems can be very productive. For years I have been playing with systems based on Murray Eden's growth model - perhaps the simplest (and first) ever model of biological growth. There's much more to explore.

PP: What is the role of serendipity and non-determinism in the formulation of a successful generative artwork?

When teaching generative art my colleague Tim Brook initially bans his students from using randomness. I don't do the same, but I can see the logic of it: randomness adds meaningless variation. Used directly, it's just that - meaningless variation that can give a false impression of richness. But it can be very handy - for example when exploring the range of outcomes of a complex system, randomising its parameters can throw up useful samples of the generative space of that system. Again it's about understanding the system. Serendipity is another thing; I think most generative artists work hard to cultivate serendipity, to entice systems into a state where pleasant surprises emerge. Many artists hand-pick "candidates" from large populations of generated works - seeking out those serendipitous moments. Although variation is fundamental to generative work, it's interesting to observe reactions to Written Images, where each volume is a unique variant of the collected works, with no opportunity for artists to pick favourites. Not having final control over each artefact is still a bit scary (for me at least).

PP: In your Watching The Sky piece there is almost a tendency to study the image in a forensic manner, to try and decode the work, and to find environmental patterns in relation to patterns in the work. This method of analysis is in almost direct contrast to the usual manner in which a data visualisation might be constructed, where an artist decides on a specific representational system beforehand to create clarity and make a point. Perhaps you could comment a bit more on how data visualisation might move forward in this respect.

I am drawing on other work here - especially the early work of Lisa Jevbratt, like her classic 1:1. Jevbratt outlines a sort of data-mysticism, a view of data as a reservoir of unknown potential, and shows fine-grained patterns without concern for "readability". In Watching the Sky (and related work) I just use images as a data source; this is a simple ploy to introduce richness by working with rich, unstructured data - and data with a complex (but legible) relationship to the world. That work has certainly shaped my thinking on visualisation. Maintaining the "unstructured" complexity of the image as a data source - rather than reducing it to statistical features - is a great way to provide contextual cues. The commonsExplorer project I did with Sam Hinton - a visual explorer for Flickr Commons streams - uses tiny cropped "core samples" that offer telltale clues about the source images.

The other idea at work here (and in Jevbratt's work) is a sense of data as (a) material; as something with texture or grain that can be felt as much as analysed. I have experimented with making these ideas literal in data-form projects like Weather Bracelet and Measuring Cup.

PP: In one of your papers you discuss synaesthesia and cross-modality in contemporary audio visuals. It seems that an important criteria for a successful synaesthetic artworks is in meaningful, metaphorical or conceptual cross-wiring of sound and video - and not just a mechanical translation between the two. What other criteria are important in a successful cross-modal artwork?

Cross-modal or "coupled" audiovisuals exemplify one of the key questions of digital media - we could call it the mapping problem. If the basic materials of the work are digital - that is, abstract patterns that can travel through any number of different substrates - then how do we make them perceivable? Or, how do we choose a mapping, a way of making data available to perception? Manovich calls this the "built-in existential angst" of digital media. So of course there are an infinity of possible ways to connect sound and image - either mapping one into the other, or generating both from some common data source. I actually like mechanical or automatic mappings. Because they are stable and consistent they let us soak in the relationship, the map itself; and these automatic maps are often quite subtle and fine-grained, compared to more composed or intentional relationships. In Robin Fox's work for example a simple (polar) oscilloscope display creates images from audio signals - but Fox explores the mapping in depth, working out how to "play" it, reverse-engineering the audio signal to create images and revealing surprising correspondences (above: image via Not-Quite-Critics). Of course automatic mappings can be incredibly boring - how many modified graphic equaliser visualisations do we need to see - but I think this is often because the mapping is filtered through too many abstractions and interventions; it becomes a set of parameters.

PP: There has been a huge influence of generative art in recent years on traditional drawing techniques such as painting and sculpture. In reverse direction, what ways, if any, can generative artists learn from traditional plastic arts?

The link there for me is a sense of "procedurality" or "processuality". In Casey Reas' work we can see a strong relationship between computational and non-computational procedures such as those of Sol Le Witt. In teaching programming to designers, I have students write and execute a Le Witt style procedure, with pencil and paper. Digital generative systems are just formal procedures, executed by machines. Treating processes as human-executable helps unpack the black boxes of generative systems mentioned earlier, and hopefully reveal them as contingent and hackable. Otherwise: the joy of materiality. Generative art and design covets the lush tangibility of traditional media; and with the wave of interest in fabrication we are seeing ever more generative work realised in "off-screen" forms. The challenge then, for pasty code-artist types, is to match the craft skills of hands-on makers in realising the work.

PP: What early interests did you have that might have lead you to your current path as an artist and academic in this field?

Music - which I don't do much of any more, but it was a big part of my world for a long time. Music (or Western music anyway) is systematised and symbolic, but also immediate and affective. That combination has always interested me. Reading Gödel, Escher, Bach - as well as lots of popular science stuff on complex systems - was influential. I was playing around with computers from around the time of the Apple II; later I convinced my father to buy an Amiga 1000, ostensibly to be used in his architecture business. It didn't ever do much architecture but I used it to make lots of bad graphics and music. Also I grew up in an outer suburb, surrounded by wild bushland; I'm a romantic nature boy at heart.

PP: Can you tell me a bit about how the dual role of essayist/writer and artist works in your situation. The dialectical relationship must create a certain amount of self-reflexivity on both sides?

Writing is fundamentally another kind of making - when it works, text and ideas are a pretty heady medium. So to some extent it's all practice, or at least speculation, experimentation, thinking of various sorts. When it works best, the practical work can trial or extend the writing, and the writing can contextualise, interpret and unpack the art work. "Practice led research" works for me as an approach - especially if you don't split art-making and writing along neat practice / theory lines.

PP: Can you tell me about any projects you have planned for the future, any new books in the pipeline or art projects in progress?

Since 2008 I've been researching and developing interactive visualisations of cultural collections datasets, working with partners including the National Archives of Australia and most recently the National Gallery of Australia. The work is challenging and rewarding; I enjoy the way data vis can span the poetic and the prosaic, and the immersive richness of large data sets. That line of work has been pulling me away from "art", which is fine with me - I generally find the edges and interfaces around creative digital culture and practice more interesting than the portion of it inside gallery walls. But the writing is also ticking over, mostly on digital materiality (or transmateriality) and the aesthetics of computational art and design. There's a new book in there somewhere, I hope.

After the Screen: Array Aesthetics and Transmateriality

noreply@blogger.com (Mitchell) — Sun, 10 Apr 2011 03:31:00 +0000

At the risk of some sort of blog-will-eat-itself situation, I'm posting this paper, presented at TIIC last November, which includes several threads developed here previously - arrays, transmateriality, and the work of HC Gilje. There are some new bits too however, on screens, projection mapping, and lots of tasty examples of a putative "post-screen" practice.

1. Glowing Rectangles

For all the diversity of the contemporary media ecology - network, broadcast, games, mobile - one technical form is entirely dominant. Screens are everywhere, at every scale, in every context. As well as the archetypal "big" and "small" screens of cinema and television we are now familiar with pocket- and book-sized screens, public screens as advertising or signage, urban screens at architectural scales. As satirical news site The Onion observes, we "spend the vast majority of each day staring at, interacting with, and deriving satisfaction from glowing rectangles."

Formally and technically these screens vary - in size and aspect ratio, display technology, spatiotemporal limits, and so on. They are united however in two basic attributes, which are something like the contract of the screen. First, the screen operates as a mediating substrate for its content - the screen itself recedes in favor of its hosted image. The screen is self-effacing (though never of course absent or invisible). This tendency is clearly evident in screen design and technology; we prize screens that are slight and bright - those that best make themselves disappear. Apple's "Retina" display technology claims to have passed an important perceptual threshhold of self-effacement, attaining a spatial density so high that individual pixels are indistinguishable to the naked eye (below - image Bryan Jones).

The second key attribute of contemporary digital screens is their tendency to generality. The self-effacing substrate of the screen is increasingly a general-purpose substrate - unlinked to any specific content type; equally capable of displaying anything - text, image, web site, video, or word-processor. This attribute is coupled of course to the generality of networked computing; since the era of multimedia the computer screen has led the way in modeling itself as a container for anything (just as the computer models itself a "machine for anything"). The past decade has simply seen this general-purpose container proliferate across scales and contexts, ushering us into the era of glowing rectangles.

However over the past decade in design and the media arts, a wave of practice has appeared which as this paper will argue, resists the dominance of the glowing rectangle. Given the near-total cultural saturation of the screen, this is unsurprising, given the ongoing cultural dance of fringe and mainstream in which this practice participates. This is not simply a story of resistance however. In proposing and describing two particular strains of "post-screen" practice, this paper aims firstly to outline the shared terms of their relationship with the screen, and in the process develop a more detailed sense of these conceptual devices of generality, outlined above, and its opposite, specificity. Secondly, and more briefly, it outlines a theorisation of this practice, invoking transmateriality, an account of the paradoxical materiality of (especially digital) media, and Gumbrecht's notion of presence.

2. Arrays

During the opening ceremony of the 2008 Beijing Olympics, a huge grid of drummers assembled in the stadium, each standing before a large square fou drum, a traditional Chinese instrument. Each drum was augmented with white LEDs mounted on its surface, triggered with each drum stroke. The drummers formed a vast array of discrete audiovisual elements, precisely choreographed in the style of these spectaculars. Human pixels, but coarse and resolutely human; at one point the drummers desynchronised entirely, forming a thunderous grid of flickering light. In a ceremony created for the (broadcast) screen - to the infamous extent of splicing computer-animated fireworks into its telecast in place of real ones - the drummers were a moment of involution. Their array echoed all the other, more conventionally self-effacing screens threaded through the event; but it also inverted some of their key attributes. Firstly its substrate, instead of receding behind "content", came forward; if anything substrate and content were one and the same. Secondly, while this array nods towards the generality of the screen in its choreographed patterns - which like the patterns on a screen could be "anything at all" - it veers strongly in the opposite direction, towards the here and now, what I will call specificity. As I argued at the time, the poetics of this array rely on the specificity of its elements - the drummers, drums, and their solid-state illumination - rather than the patterns that play across it.

The drummers are one popular example of a formal trope we can find throughout media arts and design practice over the past decade. Daniel Rozin's 1999 Wooden Mirror is one of the earlier examples. Wooden Mirror is an array of square wooden tiles embedded in a large octagonal frame, along with a bundle of custom electronics. The tiles are fitted with servomotors, so that each one can tilt up and down on its horizontal axis. As its angle to the light changes, each tile appears brighter or darker. Rozin wires up the array to a videocamera, to complete the mirror circuit: the brightness of pixels in the incoming image drives the angle of the tiles. Given the overtly visual logic of the work, it's interesting that its sound is equally striking: the wooden tiles clatter like mechanical rainfall, sonifying the rate of change of the image; as the image becomes still, the clatter dies off to a low twitching. Again, this array emphasises the material presence of its substrate. The tonal "generality" of the wooden mirror is functional enough to be familiar, but the coarse mechanical clattering of these pixels makes them inescapably specific.

Rozin has made many similar mirrors; notable is Trash Mirror (2001) where the individual elements - irregularly shaped pieces of rubbish - are packed into a freeform mosaic. This array moves one more step away from the homogeneous generality of the digital screen. Here the elements are irregular in size and shape, but also carry their own specific textures and colours. In Mirrors Mirror(2008) the regular grid returns, but the array elements are themselved replaced by mirrors; as these tilt they reflect different parts of the environment. Here the location of the tonal "content" in the array is, like the image source, deferred to the environment. In a familiar digital screen, image elements are luminous modules whose colour value is independent and absolute. In Rozin's Wooden Mirror that value becomes relative - tonality is based on self-shading, which depends on the lighting of the work. In Mirrors Mirror this relativity is multiplied; each element will reflect a different portion of the environment, depending on both its angle and the viewpoint of the observer.

In many cases these media art arrays depart from the two-dimensional grid entirely. Robert Henke and Christopher Bauder's ATOM (2007-8) (above) is an eight-by-eight grid of white helium balloons, each one fitted with LED illumination and tethered to a computer-controlled winch. The grid becomes a mobile, configurable light-form, tightly coupled with Henke's electronic soundtrack in live performance. This array lowers its resolution drastically, and limits its generality in one dimension (monochrome elements), but extends its reach (literally) into a third axis. ART+COM's 2008 kinetic sculpture at the BMW museum uses a similar configuration, but a higher "resolution" - in this case 714 metal spheres are suspended from motorised cables, forming a smoothly undulating matrix - a sort of programmed corporate ballet. Cloud (2008), a sculpture in Heathrow airport by London art and design firm Troika, illustrates another permutation: here a 2d array forms the skin of a large three-dimensional sculptural form. In this case the elements are electromagnetic flip-dots - components often used in airport signage before it was overtaken by glowing rectangles. As in Rozin's Mirrors, Troika consciously exploit the materiality, gestural character and the sound of these retro-pixels. rAndom International's 2010 Swarm Light demonstrates a "saturated" 3d array. The work consists of three cubic arrays of white LED lights, each ten elements per side; these cubic volumes host a flowing, flickering "swarm" of sound-responsive agents which traverse the space, brightening or dimming the array as they move.

The work of British designers United Visual Artists offers a useful longitudinal study in post-screen imaging; in particular their work addresses one of the central technical players in this field, LED lighting. UVA's first project involved a huge LED array that formed the stage set of Massive Attack's 100th Window tour. Unlike more screenful video backdrops, this low-res grid had an inescapable presence, hung directly behind the band and looming over the stage. Rather than an image machine, UVA treat the grid as a luminous dot-matrix for the twitching alphanumeric characters of real-time data. In subsequent work UVA develop this approach in a number of directions, but digitally articulated light - enabled by the LED - is a recurring theme. In Monolith (2006) UVA use a pair of large, full-colour LED screens, but treat them as a dynamic light source rather than a substrate for images; subtle gradients and washes of colour spill over the audience and into the installation environment, coupled with generated sound. In Volume (2006), another installation piece, the array elements are long vertical LED strips, again treated as generators of pattern, colour and sound; the work forms an interactive field as each element responds to nearby activity. In the context of this steady dismemberment of the screen, UVA's later work The Speed of Light is notable in that it leaves LED arrays aside entirely. Instead it uses installed lasers manipulated into dynamic, walk-in calligraphy, as if light had been finally prised away from its digital substrate, and turned loose in the environment.

Beyond their formal similarities, these arrays share some core approaches and contexts which provide a coherent portrait of a sort of post-screen practice. These works adopt one key feature of the screen - the "generality" of an articulated substrate - but trade it off to varying extents for more "specificity" - exploiting the local, particular materiality of the work and its environment. This specificity is also technological, reflecting a practice that crafts hard- and software into idiosyncratic configurations, rather than using off-the-shelf infrastructure. Light is a strong theme, in particular the solid-state, digitally addressable light of the LED (essentially a free-floating pixel). However the optical in these arrays is always tightly coupled with other modalities, especially sound, which is either a cherished byproduct of the array mechanism (as in Rozin's Mirrors and Troika's Cloud) or generated by the array elements themselves (as in the drummers and UVA's Volume). A quality of liveness is linked with the turn to specificity and being-in-the-environment; from the "live data" of UVA's Massive Attack show, to the live interaction and generation of their later installations, to the live video driving Rozin's Mirrors. Performance and temporary installation are the dominant forms here - emphasising the intensified moment, rather than the any-time of static content.

3. Projection Mapping and Extruded Light

In one sense these arrays present a disintegration of the screen - they pull its elements apart and embed them in the environment. In another strain of media arts practice, something like the converse occurs, though with what I will argue are similar interests and agendas. In this approach screen-like technologies are used intact, rather than decomposed; but their function and their relationship to the environment is transformed. These works reverse-engineer the digital image, exploiting its digital (general) malleability in order to fit it to a specific environment.

The work of Norwegian artist HC Gilje illustrates one trajectory of this second post-screen approach. Gilje's work from the late 90s was in live digital video, with his ensemble 242.pilots. This practice was linked to the burgeoning activity in experimental electronic music at the time; here again, performance, improvisation and the intensified moment - what Gilje calls an "extended now" - are central concerns, though the work is strongly screen-focused in its results . In Gilje's work over the following decade, he demonstrates another path towards the post-screen. Gilje's nodio (2005-) is a custom software system for distributing video content across collections of linked "nodes". In drifter (2006) these nodes are manifest as a ring of twelve screens which form a linked audiovisual interspace. With dense (2007) these nodes take on a more sculptural presence - hanging strips of fabric illuminated from both sides with a tailored video-projection. Here Gilje adapts the screen technology of the video projector to a sculptural environment, pushing it one step away from image and towards illumination. The work also depends on a specific material surface - the translucent weave of the fabric enables the double-sided layering of pattern.

shift (2008) (above) develops this approach: a technique known as projection mapping, in which the projected image is reverse-engineered to fit a specific surface. In shift Gilje's nodes are simple rectangular boxes, constructed from plywood. Using more custom software, the artist illuminates a cluster of these boxes with precisely mapped projected images. The coupled sound emanates from speakers housed in each box, so the objects are again audiovisual (and acoustically distinct) nodes; Gilje composes material for this environment in search of what he terms "audiovisual powerchords" - moments of intense juxtaposition and interplay. In blink (2009) Gilje dispenses with the boxes, instead treating the bare installation space. Simple, geometric elements - angular lines and bands of tone and colour - are reflected and modulated by the space itself, diffusing from irregular polished floorboards and painted walls. The work plays the room with articulated light, carefully matched to its geometry in way that heightens our awareness of the interplay of space, light and materials.

Projection mapping has recently flourished in "visualist" practice across art, design and performance contexts; trompe-l'oeil architectural facades are one popular genre, manipulating the built environment by rendering it with a tailored skin of articulated light (see for example Urbanscreen's Kubik 555). German designers Grosse 8 and Lichtfront demonstrate a logical extension of the technique, using multiple projectors to create an "augmented sculpture" in the round.

Another notable example is Scintillation (2009) (above) by Xavier Chassaing, a digital stop-motion film in which projection mapping is used to layer a domestic environment with luminous swirls of particles, igniting the petals of an orchid and tracing the curves of a moulded plaster cornice [24]. As in Gilje's blink, Scintillation emphasises the ambience of the projected light - reflections and diffusions are heightened by hand-held macro cinematography, artfully producing an impression of material texture. But in the process it raises some interesting problems for our analytical premise - a shift from the screenful image to something more live and specific. For Scintillation is absolutely a work of filmmaking; here projection mapping - the tailored materialisation of the image - is deployed as a technique for producing generalisable, substrate-independent image content.

The final example in this survey addresses the same tension. In their recent short film Making Future Magic (above), London design agency Berg give an ingenious demonstration of both the material turn of post-screen imaging, and its recuperation as image content. Berg developed an animation technique combining multiple-exposure stop-motion with a hand-held source of articulated light - specifically the glowing rectangle of the moment, Apple's iPad. 3d forms are digitally modelled and animated, then decomposed into sequences of 2d slices. These slices are then replayed into the environment, and thus recomposed into 3d forms, by moving an iPad screen over successive still frame exposures. As Berg term it, this is "extruded light" - as in UVA's latest work, it's as if light itself has been unpinned from its substrate. The results are a beguiling combination of loose, organic light painting with simple 3d geometry and DSLR imaging. As Berg frame the work, it fits entirely within the post-screen turn proposed here. Responding to a brief around "a magical version of future media", Berg are "exploring how surfaces and screens look and work in the world ... finding playful uses for the increasingly ubiquitous ‘glowing rectangles’ ...". Again the material embeddedness of this articulated light is emphasised - the way it reflects from puddles and diffuses through foliage. Screen as object in the world, rather than window to somewhere else. As in Scintillation however the inescapable irony is that the outcomes of this work are entirely bound up with screenful images - with the generalising infrastructures and distribution pipelines of social image sharing, print-on-demand and networked video.

4. Transmateriality and Presence Culture

To recap briefly: the ubiquitous digital screen is characterised by both generality - an ability to display any content at all - and self-effacing slightness - it tries to make itself disappear as a neutral substrate for content. In contrast to these tendencies this paper describes two distinct but parallel strains of "post-screen" practice in the media arts and design. Arrays mimic the grid configuration of the screen, but lower its resolution and emphasise the material presence of the array elements - their local and individual specificity is balanced with their malleable generality (their ability to carry anything-at-all). Projection mapping and "extruded light" practices also emphasise specificity, materiality and a local, performative being-in-the-world, but they do so by different means - exploiting the malleability of the digital screen (and the computational representations it hosts) in order to make it intensely site-specific. To the extent that they both adapt and resist the attributes of our familiar glowing rectangles, we could describe these practices as post-screen, but this "post" is nothing like a conscious critique, let alone a revolutionary break. However hard they may pull towards specificity and local materiality, they are readily - by design or necessity - recaptured as screen fodder.

Both these post-screen tendencies and their screenful recuperation can be usefully framed through the notion of transmateriality, a concept that attempts to capture a fundamental duality in digital (and other) media: they are everywhere and always material, yet often function as if they are immaterial. In a transmaterial view media always operate as local material instances (this is their aspect of specificity) yet retain the ability to hold specificity at bay - resisting the contingencies of flux - to create a functional generalisation in which this pixel is the same as that one, the email I send is the same as the one you receive, and one node on the network is much the same as any other.

In the glowing rectangle paradigm functional generality is entirely dominant. The work considered here, on the other hand, revels more in the pleasures and practices of specificity - the clatter of servo-actuated wood or the play of light on this particular wall. In their push towards liveness (of interaction or data), performativity, their integration of sound, and their emphasis on evanescent materiality, these works evoke what Hans Ulrich Gumbrecht would call "presence culture" - that mode of apprehending the world which is characterised by fleeting but intense moments of being, and a sense of being part of the world of things, rather than outside it, looking in. Gumbrecht constructs presence in opposition to a dominant "meaning culture", in which the essence of material things can be obtained only through interpretation. Gumbrecht describes the relationship between these poles as one of dynamic oscillation. "Presence phenomena" become "effects of" presence, "because we can only encounter them within a culture that is predominantly a meaning culture. ... [T]hey are necessarily surrounded by, wrapped into, and perhaps even mediated by clouds and cushions of meaning".

In exactly the same way we find an inevitable oscillation here between screen and post-screen. We can align the screen with generality and meaning culture, and the post-screen with specificity and presence culture; but here too the post-screen is evanescent and elusive, instead existing largely within the dominant screen culture. However this is not to discount the utopian aspirations of a post-screen practice, which might instead be located through the perspective of transmateriality. For in echoing the screen, or in literally bending it to the local, present and specific, these works operate as reminders of the ubiquitous and everyday materiality of our media, of the fact that depite appearances, every glowing rectangle is already local and specific. If that specificity is latent, then these works demonstrate practical strategies for making it explicit; from hardware hacking to modular LEDs and custom software, they participate in what might be called "expanded computing", using the malleability of digital media to reactivate its presence - and thus our presence, too - in the world of things.

Dynamic Design - Three Systems

noreply@blogger.com (Mitchell) — Sat, 12 Mar 2011 05:37:00 +0000

After far too long, some vaguely formed thoughts on dynamic design, after some converging links and conversations in the last few days. One of these is the new MIT Media Lab identity from The Green Eyl. It's nice work, but also seems like a new high-water mark for generative or dynamic graphic design.

In this approach graphic design goes "meta": from controlling a set of visual relationships, to controlling a system for generating visual relationships. As in other generative forms, there's a payoff in the multiplicity of the results - one logo? try 40,000 variants! But more interesting I think is a change in the locus of design, where design happens. To see one of these new logos is to appreciate its colour, form and typography; to see a dozen is to begin to appreciate the variety and coherence of relationships the designers have created. But to engage with the work fully - for example, if you're a Media Lab person, to generate your own personal variant - is to understand that it's not a logo, or even a family of logos, but a dynamic "identity system". And because this is a logo, any instance of it comes to signify not only the client, but the dynamic system, or to be more specific, a quality of "dynamic systemness." What better brand value for the Media Lab?

There is also an aspect of something like performance here. Instead of an imprint or copy, the logo becomes a performance of its system (signifying that system in the process). In discussing this with my friend Geoff Hinchcliffe the other day, he pointed out that this is really nothing new for graphic design. Any book jacket design is inevitably a performance of the genre (or system) that is "book jacket". Graphic forms like book covers are often highly constrained and rule-driven, just like this new-fangled dynamic design. Geoff's own Twitter Modern Classics demonstrates this beautifully, rendering tweets through the design templates of Penguin's iconic paperbacks. If cover design is a set of rules, it's no surprise a computer can execute them so effectively. Here dynamic design is a poetic strategy, a way to strike sparks of joy and surprise from the collision of form and content.

The final example comes by way of Daniel Neville, another designer with an interest in dynamic identity systems (or relational design). In fact the Melbourne Restaurant Name Generator is not really design at all. If anything it's something like generative satire, in the same genre that can turn out band names or even whole computer science papers. The Melbourne Restaurant thing works for me because it is such acute satire: from the recycled decor to the uber-limited menu and the obsession with bicycles, it just nails a whole urban scene. As a piece of generative satire it works by both portraying its target as formulaic - as nothing but a system - while also milking the absurd juxtapositions that its own system generates. It seems to cleave a complex thing at its joints, revealing underlying elements and relationships. Maybe there's something here for dynamic graphic design?

Uniform Diversity: Space-Filling and the Voronoi diagram

noreply@blogger.com (Mitchell) — Fri, 13 Aug 2010 01:04:00 +0000

This post is a short excerpt from a paper recently published in Architectural Theory Review 15(2) - a special issue on architecture and geometry with lots of good (Australian) stuff. My paper (pdf) is a critical look at space-filling geometry in generative design. It touches on several things already blogged - the Water Cube and ideal foams, and some generative projects that use self-limiting growth. This excerpt looks at the Voronoi diagram as a space-filling process.

The Voronoi diagram has become a ubiquitous motif in recent generative architecture and design. It, too, can be usefully read as a space-filling model. In formal terms, a Voronoi diagram is a way of dividing up space into regions so that, for a given set of sites within that space, each region contains all points in the space that are closer to one site than any other. The result is also foam-like, but as a model the Voronoi diagram has attributes quite different to the ideal Kelvin or Weaire Phelan foams.

Firstly, while the formal model is again based on a strict set of conditions (in this case proximity) it works with an arbitrary input — the given sites —rather than defining a regular structure. The Voronoi is thus a procedural geometric structure in a way that the ideal foams are not: its structure emerges through the application of a specific process or algorithm to a given set of inputs. In this way, the specific spatial relations between neighbouring cells depend on, and emerge locally from, the given spatial relations of the specified sites. This trait also gives the Voronoi model a kind of malleability; sites can be added, removed, or moved, and the spatial structure readily adapts

Again we can read off the attributes of the Voronoi as a model in this way. It is multiplicitous, but in a different way to the grid-like uniformity of the foam models. In this case, the multiplicity can, in fact, be irregular: the sites can be positioned anywhere within a given space. However, this does not amount to much, in terms of heterogeneity: while the sites can be positioned arbitrarily, the procedure, and the relation between sites that it encodes, is entirely uniform. Each site, taken as a formal entity, is identical to every other; this is a kind of uniform diversity. Like the foam models, the Voronoi diagram treats space as indefinite and extensive: it can go on forever; its only practical limit being the computational resources required to calculate the diagram. The model itself has no way of defining an edge or bound. Finally, the variability of the Voronoi can be phrased another way, as arbitrariness; in other words, that there is no inherent reason for a given site to be where it is. There is nothing internal to the model that can generate that differentiation.

In Marc Newson's Voronoi Shelf, for example (above), we see a characteristically organic variety: a range of cell sizes and shapes, different wall thicknesses, all in an agreeable state of harmony. The form gives an impression of inherent logic. It is as if the harmony of the relationships between the cell sites assures us that there must be a reason for them to be as they are. This is unsurprising, given our familiarity with, and aesthetic attunement to, naturally occurring structures that resemble these cells. The visual signature carries an association of organic logic: but in formal fact the cell sites are arbitrary, that is to say, designed. There is no necessary relation of one to another, only (we can but assume) a designer's choice, which is concealed by an appearance, much as the surface of the Water Cube conceals the regularity of its foam model.

Conversely, some designers directly address the arbitrary input to the Voronoi diagram, treating it as an opportunity and exploiting the malleability of the model. As Dimitris Gourdoukis writes, "the problem of deciding on the initial set of points is, I think, one of the most interesting in relation to voronoi diagrams." In Gourdoukis' Algorithmic Body project (above), the locations of the Voronoi sites are specified by a second generative system, a cellular automaton; here the Voronoi acts as a geometric filter, interpreting and interpolating one set of spatial data into another. In Marc Fornes' POLYTOP, the designer proposes a mass-customised product in which customers can design the point cloud that drives the Voronoi geometry; here a problem of arbitrary choice is turned into a feature, towards uniqueness and specificity.

Measuring Cup

noreply@blogger.com (Mitchell) — Sun, 06 Jun 2010 07:31:00 +0000

Measuring Cup is a little dataform project I've been working on this year. It's currently showing in Inside Out, an exhibition of rapid-prototyped miniatures at Object gallery, Sydney.

This form presents 150 years of Sydney temperature data in a little cup-shaped object about 6cm high. The data comes from the UK Met Office's HadCRUT subset, released earlier this year; for Sydney it contains monthly average temperatures back to 1859.

The structure of the form is pretty straightforward. Each horizontal layer of the form is a single year of data; these layers are stacked chronologically bottom to top - so 1859 is at the base, 2009 at the lip. The profile of each layer is basically a radial line graph of the monthly data for that year. Months are ordered clockwise around a full circle, and the data controls the radius of the form at each month. The result is a sort of squashed ovoid, with a flat spot where winter is (July, here in the South).

The data is smoothed using a moving average - each data point is the average of the past five years data for that month. I did this mainly for aesthetic reasons, because the raw year-to-year variations made the form angular and jittery. While I was reluctant to do anything to the raw values, moving average smoothing is often applied to this sort of data (though as always the devil is in the detail).

The punchline really only works when you hold it in your hand. The cup has a lip - like any good cup, it expands slightly towards the rim. It fits nicely in the hand. But this lip is, of course, the product of the warming trend of recent decades. So there's a moment of haptic tension there, between ergonomic (human centred) pleasure and the evidence of how our human-centredness is playing out for the planet as a whole.

The form was generated using Processing, exported to STL via superCAD, then cleaned up in Meshlab. The render above was done in Blender - it shows the shallow tick marks on the inside surface that mark out 25-year intervals. Overall the process was pretty similar to that for the Weather Bracelet. One interesting difference in this case is that consistently formatted global data is readily available, so it should be relatively easy to make a configurator that will let you print a Cup from your local data.

This is Data? Arguing with Data Baby

noreply@blogger.com (Mitchell) — Wed, 19 May 2010 01:20:00 +0000

These IBM commercials are gorgeous, lavish examples of modern motion graphics from Motion Theory. Like some of the agency's earlier work, and a handful of other examples noted here, these ads show how code-literate design (could we call it the P factor?) is transforming this field. For all those reasons, I love this work; but it also really bothers me. I'll try to explain.

IBM - Data Baby from John Stanch on Vimeo.

The opening line of this voiceover says it all, really. This is data. Making that call - defining what data is - is a powerful cultural gesture right now, because as I've argued before data as an idea or a figure is both highly charged and strangely abstract. It makes a lot of sense for a corporation like IBM to stake a claim on data; this stuff is somehow both blessing and curse, precious and ubiquitous, immaterial and material. IBM promises here to help with the wrangling, but also, most powerfully, to show us what data is.

So, what is data here? In these commercials data is first and foremost material. It is a physical stuff. In Data Baby it wraps a little infant like some kind of luminescent placenta, drifting away into the air, thrown off in shimmering waves as the child breathes. In Data Energy it trails like a cloud behind a tram, and spins with the blades of a wind turbine. A lot of the (beautiful) animation work here has been devoted to simulating behaviour, making this colorful, abstract stuff seem to be tightly embedded in the world with us. What that means is both coupling it tightly to real objects, and supplying it with immanent dynamics - making it drift, disperse or twirl.

IBM Data Energy from Michael Suarez on Vimeo.

The second interesting property of data here - related to the first - is that it just exists. Look again at Data Baby, and note that there is no visible sign of this data being gathered (or rather, made). No oxygen saturation meter, no wires, no tubes, no electrodes. Not a transducer in sight. Not until the closing wide shot do we even see a computer. (This is fascinating in itself; IBM (or their ad agency) gets it that the computer is no longer the right image, or metaphor, for "information technology". Neither is the network; now it's immanent, abundant data.) In other words data here is not gathered, measured, stored or transmitted - or not that we can see. It just is, and it seems to be inherent in the objects it refers to; Data Baby is "generating" data as easily as breathing.

Completing this visual data-portrait are some other related themes: data is multiplicitous and plentiful, it's diverse (many colours and shapes) but ultimately harmonious and beautiful - in Data Transportation it looks like an urban-scale 3d Kandinsky painting.

Several things bother me about this portrayal. The first is the same is the reason I love it: it's powerfully, seductively beautiful, and this amplifies all my other reservations. The vision of data as material, in the world, is also incredibly seductive; my concern is that we get such pleasure from seeing these rich dynamics play out - that the motes wafting from Data Baby's skin seem so right - that we overlook the gaps in the narrative. This vision of material data is also frustrating because it has all the ingredients of a far more interesting idea: data is material, or at least it depends on material substrates, but the relationship between data and matter is just that, a relationship, not an identity. Data depends on stuff; always in it, and moving transmaterially through it, but it is precisely not stuff in itself.

You could say that I'm quibbling about metaphors here, and you'd be right, but metaphors are crucially important because they shape what we think data is, and what it does. Related to data as stuff is this second attribute; data that just is, in the same way that matter is neither created or destroyed, but just exists. This is crucially, maybe dangerously wrong. Data does not just happen; it is created in specific and deliberate ways. It is generated by sensors, not babies; and those sensors are designed to measure specific parameters for specific reasons, at certain rates, with certain resolutions. Or more correctly: it is gathered by people, for specific reasons, with a certain view of the world in mind, a certain concept of what the problem or the subject is. The people use the sensors, to gather the data, to measure a certain chosen aspect of the world.

If we come to accept that data just is, it's too easy to forget that it reflects a specific set of contexts, contingencies and choices, and that crucially, these could be (and maybe should be) different. Accepting data shaped by someone else's choices is a tacit acceptance of their view of the world, their notion of what is interesting or important or valid. Data is not inherent or intrinsic in anything: it is constructed, and if we are going to work intelligently with data we must remember that it can always be constructed some other way.

Collapsing the real, complex, human / social / technological processes around data into a cloud of wafting particles is a brilliant piece of visual rhetoric; it's a powerful and beautiful story, but it's full of holes. If IBM is right - and I think they probably are - about the dawning age of data everywhere, then we need more than a sort of corporate-sponsored data mythology. We need real, broad-based, practical and critical data skills and literacies, an understanding of how to make data and do things with it.

commonsExplorer

noreply@blogger.com (Mitchell) — Fri, 26 Mar 2010 21:26:00 +0000

A quick bit of cross-promotion. The commonsExplorer is an experimental "big picture" browser for Flickr Commons collections - Sam Hinton and I started working on it for MashupAustralia months ago, and it's finally ready. Read some background over on the Visible Archive blog, or download the app and try it out.

Data Walks - a #climatedata proposal

noreply@blogger.com (Mitchell) — Fri, 11 Dec 2009 22:51:00 +0000

In response to the UK Met Office's recent data release and Manuel Lima's call for visualisations, there's been a flurry of #climatedata activity in the last couple of days, including some revealing visualisations. Though I'm looking forward to playing with the data myself, this isn't a post about visualisation. It's a simpler proposal for a way to make the data tangible.

Global warming is ultimately a question about change in a single measurement - temperature - over time. One way or another, it can be boiled down to a line graph. How best to make that line tangible? Visualisation is great, but how else could we feel those changes, especially over time? One way would be to walk the data. We could make a kind of giant line graph, in the form of a path or road, then walk from 1850 to 2009. According to the Met Office's graph - remixed above with a picture of my local landscape - this would be a fairly undulating journey, but the last half especially would be a distinct and noticeable climb. Building this path at a walkable scale seems like hard work though. It would be much easier to use the paths we already have. So, here's a recipe for a #climatedata walk:

Make a graph. There are all kinds of options here. The Met Office graph shows global difference from a long-term (1961-1990) average. You could for example use local data only, or use raw average temperatures rather than difference from average. You would also need to select a year range from the data - want to walk the whole century or just post-WW2? All the data choices should be made clear to any walkers.
Fit to landscape. This is the tricky part. The idea would be to find a walkable route with changes in elevation that fit your line graph well. Finding a perfect fit will be very difficult, but finding an OK fit should be possible. This will involve some scaling questions: how long will the walk be, and how much elevation will it cover? Accessibility, ergonomics, experience design, affect - lots of juicy design decisions here. One crude but easy fitting procedure would be to begin with a route, find its elevation profile, then scale the graph to fit the start and end points of the graph to the route start and end, then note the points where the path and the graph intersect. Maybe some GIS / maps people could help with software tools here for route finding and fitting?
Tick marks. Walk the route and mark it out in order to make the whole thing legible. Mark out years or decades, as well as temperature variation (elevation). One option for paths with an imperfect fit, would be to notate the difference between the path and the graph at certain points, as well as points where the path and the graph intersect.
Walk. Again you can imagine many ways to do this, ranging from big organised public walks, to smaller private ones. Of course walking often leads to talking - and in a different way to, say, looking at a graph.

I should emphasise that I haven't even tried this, yet, but I hope to - Canberrans, if you're interested in helping organise a walk here, let me know. Wherever you are, if you do try it, let me know - also feel free to adapt / refine / repurpose the procedure. Could be fun, even informative - at the very least, you'll walk up a hill.

Readings in Digital Design

noreply@blogger.com (Mitchell) — Sat, 05 Dec 2009 05:28:00 +0000

The Master of Digital Design launched this year with an introductory unit which featured UC alumni Supermanoeuvre, and turned out some great work. Next year it ramps up, with more units and more students - very exciting. I'm currently preparing "Readings in Digital Design", a history and theory unit that presents some key concepts in this nascent, multidisciplinary field (or meta-field). While developing the unit I've also been thinking about how to make the whole course "open" in the broadest sense - accessible, transparent, connective, collaborative. There's a tangle of technical and institutional issues here which I have no single solution to, so in the meantime I'll take a "small pieces loosely joined" approach - this post is the first of those small pieces - the draft reading list at the core of the new unit.

The list attempts to sample the breadth of digital design practices and approaches - so it spans cyberculture, architecture, product design, interaction design, and media art. It also mixes historical sources, academic articles, blog posts and web video, for the same reason, to give a sense of the range of contexts and discourses at work here. With the exception of a couple of firewalled papers (thanks Wiley and ACM), all the sources are freely available online.

Feedback very welcome, as well as additions or gap-plugging - especially on open source in digital design, and tangible / physical computing. Please reuse / remix also, and let me know if you do - call it Creative Commons by-nc-sa.

Readings in Digital Design - Master of Digital Design 2010

Being Digital

Horswill, Ian. “What is Computation?,” 2007.
Palfreman, Jon. “Giant Brains.” The Machine that Changed the World. WGBH Boston, 1992.
Rheingold, Howard. “The First Programmer Was a Lady.” In Tools for thought. Cambridge, Mass.: MIT Press, 2000.

Pre/Histories of Digital Design

Kay, Alan. “Personal Dynamic Media.” In The New Media Reader, edited by Noah Wardrip-Fruin and Nick Montfort. Cambridge, Mass.: MIT Press, 2003.
Ivan Sutherland : Sketchpad Demo (1/2), 2007.
Mark, Earl, Mark Gross, and Gabriela Goldschmidt. “A Perspective on Computer Aided Design after Four Decades.” In eCAADe 2008: education in computer aided architectural design in europe annual conference, 2008.

Networks

Rheingold, Howard. “Xanadu, Network Culture, and Beyond.” In Tools for thought. Cambridge, Mass.: MIT Press, 2000.
O'Reilly, Tim. “What Is Web 2.0: Design Patterns and Business Models for the Next Generation of Software.” O'Reilly Media, September 30, 2005.
Burke, Anthony. “Network Paradigms.” In Network Practices, edited by Anthony Burke and Therese Tierney. Princeton Architectural Press, 2007.

Open Source

Raymond, Eric Steven. “The Cathedral and the Bazaar,” 2000.
Lessig, Lawrence. “On Free, and the Differences between Culture and Code” presented at the 23rd Chaos Communication Congress, Berlin, 2006.
The Open Art Network

Designing with Data

“The Petabyte Age: Because More Isn't Just More - More is Different.” Wired, June 23, 2008.
Jones, Matt. “Data as Seductive Material” presented at the Umeå Institute of Design Spring Summit, March 2009.
Armitage, Tom. “Toiling in the data-mines: what data exploration feels like.” BERG Blog, October 23, 2009.
Whitelaw, Mitchell. “Art Against Information: Case Studies in Data Practice.” Fibreculture 11 (2009).

Fab!

Gershenfeld, Neil. "The beckoning promise of personal fabrication," presented at TED, 2007.
Menges, Achim. “Manufacturing diversity.” Architectural Design 76, no. 2 (2006): 70-77.
Smith, Greg J. “Means of Production: Fabbing and Digital Art.” Rhizome, March 4, 2009.

Ubiquitous Computing and Urban Informatics

Weiser, Mark. “The computer for the 21st century.” Scientific American 256, no. 3 (1991): 66–75. Reprinted in IEEE Pervasive Computing, January 2002.
Hill, Dan. “The street as platform.” City of Sound, February 11, 2008.
Galloway, Anne. “Resonances and Everyday Life: Ubiquitous Computing and the City,” 2003.
Greenfield, Adam. “All watched over by machines of loving grace: Some ethical guidelines for user experience in ubiquitous-computing settings.” Boxes and Arrows, December 1, 2004.
Haque, Usman. “Pachube, Patching the Planet: Interview with Usman Haque.” Interview by Tish Shute, January 28, 2009.

Parametricism and its Discontents

Schumacher, Patrik. “Parametricism - A New Global Style for Architecture and Urban Design,” 2008.
Jacob, Sam. “The Ruins of the Future.” Strange Harvest, December 5, 2008.
Love, Tim. “Between Mission Statement and Parametric Model.” Design Observer, May 11, 2009.

Tangible and Physical Computing

Moggridge, Bill. “Multisensory and Multimedia.” In Designing Interactions. The MIT Press, 2007.
Igoe, Tom. “Physical Computing’s Greatest Hits (and misses).” hello., July 27, 2008.

Biomimicry, Complexity and Self-Organisation

Weinstock, Michael. “Self-organisation and material constructions.” Architectural Design 76, no. 2 (2006): 34-41.
Bentley, Peter. “Climbing Through Complexity Ceilings.” In Network Practices, edited by Anthony Burke and Therese Tierney. Princeton Architectural Press, 2007.
Kaplinsky, J. “Biomimicry versus humanism.” Architectural Design 76, no. 1 (2006): 66-71.

Redesigning Design

Sanders, Elizabeth, and Pieter Jan Stappers. “Co-creation and the new landscapes of design.” CoDesign 4 (March 2008): 5-18.
Howe, Jeff P. “Is Crowdsourcing Evil? The Design Community Weighs In.” Epicenter | Wired.com, March 10, 2009.

Sustainable Digital?

Bonanni, Leonardo, Amanda Parkes, and Hiroshi Ishii. “Future craft: how digital media is transforming product design.” In CHI '08 extended abstracts on Human factors in computing systems, 2553-2564. Florence, Italy: ACM, 2008.
DiSalvo, Carl, Kirsten Boehner, Nicholas A. Knouf, and Phoebe Sengers. “Nourishing the ground for sustainable HCI: considerations from ecologically engaged art.” In Proceedings of the 27th international conference on Human factors in computing systems, 385-394. Boston, MA, USA: ACM, 2009.
Karsten Schmidt. “Sustainablity and generative design.” toxi.in.process 22 Jul 2007.

Right Here, Right Now - HC Gilje's Networks of Specificity

noreply@blogger.com (Mitchell) — Tue, 27 Oct 2009 05:09:00 +0000

This essay was commissioned by Hordaland Kunstsenter in Bergen, Norway, to coincide with HC Gilje's solo exhibition blink (video below). It looks at Gilje's recent work - which spans audiovisual installation, performance, hardware, and networked forms - through the notion of specificity (developed earlier here).

The digital network, where we all spend ever more of our time, is a vast infrastructure of generality. It deploys a system which is standardised, formally defined, highly structured, and internally consistent. If I send you an email, I do it trusting that the interlinked systems of hard- and software, the protocols for data encoding and transmission, the network switches and servers, will all hold together so that the email you receive is the same as the one I sent. Perhaps I'm in Australia, and you are in Norway: we could say that the network generalises our two points in space - for the network, they are the same. As I draft my email it exists as a pattern of voltages and magnetic flux inside my computer. To transmit that pattern effectively, the digital network must erase or resist any local errors or inconsistencies that it might encounter along the way, so that it does not matter if the pattern travels by optical fibre or copper, or in radio waves, or if a boat anchor cut through a cable near Indonesia. It does not matter that your computer is made of different atoms to mine. Those are specificities - local, material events and instances. Digital culture, and networked space, absorbs specificities, compensates for them, rectifies them into generality. Wireless broadband and mobile computing make us into human nodes, bathing in shared connective protocols.

The aesthetics of digital media flow from a related generality, where sound and image are encoded as fields of data. If a pixel is a number, an image is a grid of pixels, video a stream of images, and each of these numbers can take any value at all, then formally, an aesthetics of digital video is only a matter of finding the right values - fishing around in a space containing all possible digital video. If digital media creates this generalised space, anything at all, the media arts are faced with unavoidable questions: not only what to make - which values to choose, but how to choose them, and why?

HC Gilje's work arises from a moment when the anything-at-all of digital video was just opening up, thanks to a combination of new real-time tools, cheap computing power, and some key interdisciplinary influences. Drawing on experimental sound and music, improvisation and performance became important solutions; working live in a specific situation, artists would gather, process, generate, and recombine material. In work from the late 1990s and early 2000s, from Gilje and his collaborators in 242.pilots, as well as video ensembles such as Granular Synthesis and Skot, the result is abstract and intense, a flow of layered digital texture. In performance it saturates the body and senses; big screens, big speakers. Instead of the narrative transport of cinema, which takes us somewhere else, this work creates - and is created in - an intensified sense of presence, what Gilje calls an "extended now". This methodology is vital; it focuses the open-ended generality of digital media in to a point: on this, rather than anything-at-all.

This moment relies on a circuit, a close coupling between artist and media; data flows become experienced events - sounds and images - which in turn inform new data flows, and so on. Audience and performers share a digital-material situation. The specificity of digital media comes forward; for of course these media are always specific, always local, always embodied; but that specificity is usually suppressed by the functional logic of generality. At the same time though, the processes underway here depend on exactly that generality, on the machine's ability to rapidly transform data and shift it between instantiations - from the voltages in video memory to the patterns of projected light.

In nodio (2005-) Gilje creates a system of networked audiovisual nodes that process and share image material. Each node generates sound derived from its image, in a process of automatic translation. On one hand this translation is another demonstration of the abstract pliability of the digital - its ability to transform anything into anything (generality); on the other, its tight audiovisual correpondences generate sparks of material intensity - real events, rather than digital effects (specificity). With these distributed nodes Gilje deploys audiovisual materials in space, creating flows and juxtapositions that function as dynamic sculpture. Of course the formal model of nodio echoes our most ubiquitous generalising paradigm: the network. Once again, the artist applies this digital tendency for generalisation in order to cultivate instances of specificity - the texture and sensation of the here and now.

From drifter (2006) (above) to dense (2006) and shift (2008) (below), Gilje's audiovisual nodes map out a developing exploration of specificity. drifter deploys standard computer hardware, formed into sculptural modules; in passing material between nodes Gilje begins to break the frame of the screen, creating an implicit inter-space. In dense, the hardware moves out of the sculptural field, and the screen is further deconstructed. Instead of the frontal configuration of the cinema / computer, these suspended fabric strips are illuminated from both sides with a video "weave". The familiar architecture of the screen as a blank (general-purpose) substrate containing or supporting image content, is reconfigured here; the specific materialities of screen and content overlap. Even more so in shift, where the nodes are now wooden boxes, illuminated with precisely controlled video projections. As in earlier nodio works sound and image are directly related. Here Gilje extends this fusion to the sculptural objects; each node is also its own speaker-box, so that the digital articulation of sound and image is realised, and grounded materially, in the nodes themselves.

These works drive towards a spatial materialisation of audiovisuals: dynamic constellations of AV intensity, fields for what Gilje calls "audiovisual powerchords". The projectors, speakers and networks of the nodio works present one means to this end, deploying existing media technologies. Again we find an interplay of generality and specificity, as Gilje adapts generalising systems - projectors, computers, networks - to realise materialised instances. The Wind-up Birds (2008) (below) represent another angle of approach; Gilje sets video aside, and creates materialised, local, sculpturally autonomous nodes from electronic and mechanical materials. In these robotic woodpeckers digital media and sculptural embodiment are further enmeshed. The birds communicate using digital radio, and their behaviour is programmed in a custom chip; but their sound is simply percussion - a mechanical switch, tapping on a specially constructed wooden slit-drum. Again this is specificity over generality: a loudspeaker is an acoustic shape-shifter, a technology which promises any sound, in the same way that the screen promises any image. By contrast the Birds produce only one sound, their sound, a specific conjunction of solenoid, timber and vibrating air.

The Birds will run for a month on their own batteries, strapped to trees, calling to each other and any other creatures nearby. These nodes are unplugged: they begin to come away from the technological support system of mains power and the shelter of the gallery or studio, and move out into the world. As in the artist's other work the engineering here is inseparable from the artistic agenda; the Birds are in that sense a realisation of Gilje's spatial and formal aims, an autonomous constellation of intensities. But they also literally expand from there; where the nodio works explore the composition of spaces within a network of intensities, the Birds move outwards, creating points of intensity in the wild, and evoking a spatial alertness - a way of being in and listening to the world - that extends beyond the well-marked edges of an artwork. The Birds are more like an experimental intervention, a digital-material overlay in a complex field of the living and non-living.

Similarly the Soundpockets works (both 2007) make small sonic interventions in urban spaces, pursuing local intensification and juxtaposition through directional soundbeams and micro-scale radio transmissions. Once again we find this interplay of the general - the anything-at-all of the digital - and the specific, the here and now. The "extremely local radio stations" of Soundpockets 2 form a sort of folded juxtaposition of three layers: globalised network infrastructures and protocols, the traced or mediated locations of field recordings, and the specific time and place of the transmissions. Just as Soundpockets 1 uses exotic soundbeam acoustics to perturb urban spaces, Soundpockets 2 shows how we can draw in technological infrastructures in order to reconfigure the real environment, creating flows and distributions that form intense moments of difference and specificity.

In this reading Gilje's work is partly critical. Pursuing specificity, and an intensified, material experience of the here and now, it pushes against the generalising tendencies of digital media. By the functional logic of the network, each node is formally identical, and must be effectively insulated from its environment. Ubiquitous computing promises us "everyware" - total connectivity, the complete interpenetration of the network and our lived environment [2]. But if the network is a generalising force, if it erases differences between places, what will life in "everyware" be like? Gilje's work suggests a utopian alternative: networks that are always local in time and space; nodes of right here, right now. Gilje's work strives for what Hans Gumbrecht calls "presence"; a way of knowing the world that is characterised by intense moments of encounter or revelation - aesthetic experiences that place us in the world, and of it, rather than observing from the intellectual distance of interpretation.

The beauty of Gilje's work though is that it not only suggests this prospect, but demonstrates it, makes it happen; and in that sense the work is constructive, rather than critical. In emphasising the specificity of media technologies, Gilje's work shows us a different way to frame those technologies; as always material, always in the world with us - a view I have called transmateriality. As Matthew Kirschenbaum writes, "computers ... are material machines dedicated to propagating a behavioral illusion, or call it a working model, of immateriality." Gilje shows us both sides of this statement, the functional illusion - generality - and its material foundation - specificity. It shows us a way to reframe the network, too; as always local, always specific; a tangle of real flows and propagating patterns; and endless possible ways of reconnecting the world with itself. Finally Gilje shows us one crucial role for the artist, in this context: seeking out configurations that intensify, rather than dilute, our sense of being in the world.

Weather Bracelet - 3D Printed Data-Jewelry

noreply@blogger.com (Mitchell) — Wed, 07 Oct 2009 02:00:00 +0000

Given my rantings about digital materiality and transduction, fabrication is a fairly obvious topic of interest. I posted earlier about an experiment with laser-cut generative forms and Ponoko - more recently I've been playing with 3d-printing via Shapeways, as well as trying out data-driven (or "transduced") forms. This post covers technical documentation as well as some more abstract reflections on this project - creating a wearable data-object, based on 365 days of local (Canberra) weather data.

Shapeways has good documentation on how to generate models using 3d-modelling software. Here I'll focus more on creating models using code-based approaches, and Processing specifically. The first challenge is simply building a 3d mesh. I began with this code from Marius Watz, which introduces a useful process: first, we create a set of 3d points which define the form; then we draw those points using beginShape() and vertex().

The radial form of the Weather Bracelet model shows how this works. The form consists of a single house-shaped slice, where the shape of each slice is based on temperature data from a single day. The width is static, the height of the peak is mapped to the daily maximum, and the height of the shoulder (or "eave") is mapped to the daily minimum. To create the radial form, we simply make one slice per day of data, rotating each slice around a central point. As the diagram below shows, this gets us a ring of slices, but not a 3d-printable form. As in Watz's sketch, I store each of the vertices in the mesh an array - in this case I use an array of PVectors, since each PVector conveniently stores x,y and z coordinates. The array has 365 rows (one per day, for each slice) and 5 columns (one for each point in the slice). To make a 3d surface, we just work our way through the array, using beginShape(QUADS) to draw rectangular faces between the corresponding points on each of the slices.

To save the geometry, I used Guillame laBelle's wonderful SuperCad library to write an .obj file. I then opened this in MeshLab, another excellent open source tool for mesh cleaning and analysis. Because of the way we draw the mesh, it contains lots of duplicate vertex information; in MeshLab we can easily remove duplicate vertices and cut the file size by 50%. MeshLab is also great for showing things like problems with normals - faces that are oriented the wrong way. When generating a mesh with Processing, the order in which vertices are drawn determines which way the face is ... er, facing... according to the right hand rule. Curl the fingers of your right hand, and stick up your thumb: if you order the vertices in the direction that your fingers are curling, the face normal will follow the direction of your thumb. Although Processing has a normal() function that is supposed to set the face normal, it doesn't seem to work with exported geometry. Anyhow, the right hand rule works, though it is guaranteed to make you look like a fool as you contort your arm to debug your mesh-building code.

The next step in this process was integrating rainfall into the form. I experimented with presenting rainfall day-by-day, but the results were difficult to read; I eventually decided to use negative spaces - holes - to present rainfall aggregated into weeks. Because Shapeways charges by printed volume, this had the added attraction of making the model cheaper to print! The process here was to first generate the holes in Processing as cylindrical forms. Unlike the base mesh, each data point (cylinder) is a separate, simple form: this meant I could take a simpler approach to drawing the geometry. I wrote a function that would just generate a single cylinder, then using rotate() and scale() transformations made instances of that cylinder at the appropriate spots. Because I wanted the volume of each cylinder to map to rainfall, the radius of each cylinder is proportional to the square root of the aggregated weekly rainfall. As you can see in the grab below, the base mesh and the cylinders are drawn separately, but overlayed; they were also saved out as separate .obj files. The final step in the process was to bring both cleaned-up .obj files into Blender (more open source goodness) and run a Boolean operation to literally subtract the cylinders from the mesh. This took a while - Blender was completely unresponsive for a good few minutes - but worked flawlessly.

Finally, after checking the dimensions, exporting an STL file from MeshLab, and uploading to Shapeways, the waiting; then, the printed form. I ordered two prints, one in Shapeways' White, Strong and Flexible material, and the other in Transparent Detail. You can clearly see the difference between the materials in these photos. The very small holes tested the printing process in both materials; in the SWF print the smallest holes are completely closed; in the TD material they are open, but sometimes gummed up with residue from the printing process (which comes out readily enough). Overall I think the TD print is much more successful - I like the detail and the translucency of the material, as well as the cross-hatched "grain" that the printing process generates.

So, a year of weather data, on your wrist - as a proof of concept the object works, but as a wearable and as a data-form it needs some refinement. As a bracelet it's just functional - the sizing is about right, but the sharp corners of the profile are scratchy against the skin. As a data-form, it could do with some simple reference points to make the data more readable - I'm thinking of small tick-marks on the inner edge to indicate months, and perhaps some embossed text indicating the year and location. More post-processing work in Blender, I think.

Another line of development is to do versions with other datasets - and hey, if you'd like one for your city, get in touch. But that also raises some tricky questions of scaling and comparability. The data scaling in this form has been adjusted for this dataset; with another year's data, the same scaling might break the form - rain holes might eat into the temperature peaks, or overlap each other, for example. A single one-size-fits-all scaling would allow comparisons between datasets, but might make for less satisfying individual objects - and, finding that scaling requires more research.

What has been most enjoyable with this project, though, is the immediate reaction the object evokes in people. The significance and scale of the data it embodies, and its scale, seem to give it a sense of value - even preciousness - that has nothing to do with the cost of its production or the human effort involved. The bracelet makes weather data tangible, but also invites an intimate, tactile familiarity. People interpret the form with their fingers, recalling as they do the wet Spring, or that cold snap after the extreme heat of February; it mediates between memory and experience, and between public and private - weather data becomes a sort of shared platform on which the personal is overlayed. The form also shows how the generalising infrastructures of computing and fabrication can be brought back to a highly specific, localised point. This for me is the most exciting aspect of digital fabrication and "mass customisation" - not more choice or user-driven design (which are all fine, but essentially more of the same, in terms of the consumer economy) - but the potential for objects that are intensely and specifically local.

Tiny Sketching

noreply@blogger.com (Mitchell) — Sat, 22 Aug 2009 21:55:00 +0000

As a kind of test pattern to fill the current break in transmission, here are my contributions to Tiny Sketch, an OpenProcessing / Rhizome competition (open until mid September) for Processing sketches under 200 characters.

In Bit Sunset I just load the pixels[] array, pick a random block of pixels, and add a large number to their value. This process throws up some surprising results as the colour values gradually increase, then start pushing into the alpha bits of the ARGB integer; eventually, as it fills the alpha bits, it settles into a pallette of pinks and greens that are gradually smashed into pixel-dust.

Albers Clock was an attempt to slow the pace of TinySketch even further; it visualises the current time in the form of an Albers square, with three colours, one each for hour, minute and second. I also like that it creates an image that is synchronous (within timezones, at least), unlike the asynchronous, individualised runtimes of most sketches.

There are dozens of amazing sketches in this collection - it's a fascinating microcosm (in every sense) of the current Processing / generative / code art scene. Given the tight constraints it's not surprising to see some demoscene virtuousity in the code - like Martin Schneider's Sandbox, a physical simulation painting app in 200 characters. There is also some classic software art conceptualism and reflexivity - like Jerome St Clair's Joy Division cover and Kyle MacDonald's Except. Great to see projects like this - and OpenProcessing itself - reviving applet culture in an open source, web2.0-flavoured way.

Landscape, Slow Data and Self-Revelation

noreply@blogger.com (Mitchell) — Fri, 15 May 2009 04:44:00 +0000

This text was an invited contribution to Kerb 17: Is Landscape Architecture Dead? This looks like a rich volume with a sharp critical edge, and a swathe of interesting material spanning architecture, urbanism, art and landscape. Unfortunately my contribution was edited fairly severely; so here's the unabridged version. Redundancy warning for regular readers: there's a slight rehash of Watching the Sky in here; but afterwards there's fresh material on landscape / data projects by Driessens and Verstappen and Usman Haque.

Data is, we imagine, an immaterial thing; or at least ethereal, made of light and electricity, processed at superhuman speed, transmitted in real time. The everyday world we move in seems dense and slow by comparison. The landscape is slower again; thick, heavy and persistent. At the moment however those two domains, the fast lightness of data and the heavy slowness of the landscape, are urgently linked. We are faced with the prospect of momentous change in the landscape that is somehow both slow and fast; too slow for our real-time culture to grasp, and too fast for the living systems of the landscape to adapt to. This paper presents a handful of works that dwell in that disjunction, between landscape and data; not solving it at all, but at least forming links, complicating assmptions, and recasting the relationship between two terms that seem to neatly encapsulate our future.

In Watching the Sky a camera looks out my office window, at the sky and the landscape. A banal view over a university campus to a bushy ridge in Belconnen. The camera takes an image every three minutes; four hundred and eighty images in twenty four hours. Tethered to a computer, the camera records for weeks at a time; the computer accumulates thousands of images. I think of the images as data, traces of change in the world outside the office window. I visualise, or re-visualise, this image data in the simplest possible way; an automated process "cuts" a narrow vertical slit from the same location in each image, and compiles all these slits together (this is a digital imitation of an analog photographic technique known as "slit-scan"). In the rectangular visualisations the slices are tiled from left to right. In the radial visualisations slices are gradually rotated so that a twenty-four-hour period spans one complete revolution (the "seam" is at midnight).

In the resulting images the patterns of change within and between days are immediately visible. As I imagined, day and night, cloud and sky are obvious. The brief, delicate colour shifts of dawn and dusk were more surprising. Below the horizon, though, patterns appeared that complicated the work's nominal focus on the sky. It became clear that some of the richest and most revealing data here came from the landscape. In one of the earliest sketches I found small but distinct variations in the horizon line over the course of a day, and recurring on successive days. I eventually realised that this was caused by the afternoon breeze, shifting foliage by a few pixels within the frame. In other words, subtle changes in the material field of the landscape carried through to the image data. Moreover in many ways the landscape visualises its own internal structure: the trees blowing in the breeze are partly instruments, revealing material changes around them (the breeze); but also data, traceable as pixels. In many images the passage of a shadow across the ground appears as a recurring pattern, an enfolded or multiplexed representation of another set of material interactions; the landscape measures and reveals itself, but not as an object, image or view. It is a connective, dynamic, material system; what is revealed are the specific interactions of that system with itself. The image data acts as a kind of core sample, drilling through multiple spatial and material systems, but each is connected outwards, beyond the frame. The wind in the trees doesn't belong to this image, but like the angle of the sun revealed in the shadow, is an index of a wider system.

It also became clear that the landscape is densely packed with human, social data which is equally apparent in image data. In the rectangular visualisations presented here stripes of colour are visible towards the bottom of the frame. These are caused by cars, parked illegally under the trees; they form another ad-hoc graph that reflects cultural, institutional calendars and cycles, though again they are intermingled with other scales and structures.

Landscape is also cast as a self-revealing instrument in Driessens and Verstappen's Tschumi Tulips project. This landscape installation occupied the Tschumi Pavilion, in Hereplein, Groningen, during the northern Spring in 2008. The pavilion is a rectilinear glass container, rising at an angle from the surrounding park. In this installation the artists filled the base of this box with soil and planted over ten thousand white tulips. A matching array of tulips was planted outside, extending the line of the pavilion. Like scientists, the artists set up two identical subjects, but vary their environment: ten thousand tulips inside, ten thousand outside. A webcam reveals how these variations in environment are slowly materialised in the life of the tulips. The tulips inside grow, bloom and then, wonderfully, decay more rapidly than their twins outside. As in Watching the Sky, long time spans are compressed into human-scale time and space; and here too digital imaging plays a pragmatic role in that revelation. Deployed in rectangular masses we can easily read the flowers as abstract, sculptural materials; organic pattern and variation enframed and aestheticised. But at the same time the work has a kind of deadpan resonance, a rendition of life, and death, inside a greenhouse.

The Huey-Dewey-Louie Climate Clock, by Usman Haque and Robert Davis, addresses the long timescales of environmental change head-on in a proposal that further develops this articulation of slow data and landscape. The clock is a multi-layered system of autonomous machines and material processes. The "Huey" agent slowly builds "accretion mounds" using material extracted from the atmosphere and formed into accumulating conical stacks over the course of a year; like tree rings or geological strata these embed environmental materials directly into a designed representation. The "Dewey" element is a circular array of one hundred transparent containers, in which air and biomass samples are preserved year by year. Like Driessens and Verstappen's Tulips, Haque and Davis propose a biological instrument of one hundred genetically identical daffodils, which are sown and harvested each year, then entombed in the plinths - again a simple grid, a layer of invariance is imposed that allows the landscape to essentially represent itself, materially. Finally Louie, an autonomous solar-powered robot, gathers soil samples and compresses them into cubes, one per day. The surface of each cube is imprinted with some current data point - chosen by daily popular vote; perhaps oil price, or rainfall. So here fast, real-time, socially selected data comes to rest directly on the slow, material medium of the soil.

At one stage, not long ago, it may have seemed that we were leaving the landscape behind, or drafting it in only in as a support or substrate for the flickering patterns of real-time culture. Even now, that seems possible: the monthly figure for new housing construction, a bellwether for economic growth, is imposed on the landscape by earthmovers and roadbuilders, underscored by raw mounds of earth. The works presented here suggest an alternative role, perhaps an alternative future for the landscape; as slow data and slow instrument, a complex material system that can be subtly designed into self-revelation.

Transduction, Transmateriality, and Expanded Computing

noreply@blogger.com (Mitchell) — Sun, 03 May 2009 06:30:00 +0000

In common usage a transducer is a device that converts one kind of energy to another. Wikipedia lists a fantastic variety of transducers, mapping out links between thermal, electrical, magnetic, electrochemical, kinetic, optical and acoustic energy. In this form transducers are everywhere: a light bulb transduces electrical energy into visible light (and some heat). A loudspeaker transduces fluctuations in voltage into physical vibrations that we perceive as sound.

In analog media, transduction is overt (put the needle on the record...). But digital media are riddled with it too. Inputs and output devices all contain transducers: the keyboard transduces motion into voltage; the screen transforms voltage into light; the hard drive mediates between voltage and electromagnetic fields. A printer takes in patterns of voltage and emits patterns of ink on a page. Strictly transduction only refers to transformations between different energy types; here I want to extend it to talk about all the propagating matter and energy within something like a computer, as well as those between that system and the rest of the world. From this transmaterial perspective a computer is a cluster of linked mechanisms and substrates; a machine for shifting patterns through time and space.

If this sounds unfamiliar, it's only by historical accident. Mechanical computers, where these patterns are physically perceptible, predate electrical (let alone digital) ones, by centuries (above: a replica of Konrad Zuse's Z1, a mechanical computer from 1936. Image by rreis). Materially, our current computers are more or less black box systems. Their transductions come as a sort of preconfigured bundle or network, a set of familiar relations constructed again by mixtures of hard- and software, protocols, standards: generalising frameworks. I press a key, a letter appears; this is all I need to know. Click "OK". No user-serviceable parts inside.

Except that currently, across the media arts and a whole slew of other fields, the computer is undergoing a rich and productive decomposition. It's composting, to borrow a Sterlingism. This goes under all kind of different names: hardware hacking, device art, homebrew electronics, physical computing. Such practices mount a direct assault on the computer as a material black box, literally and figuratively cracking it open, hooking it up to new inputs and outputs, extending and expanding its connections with the environment. Microcontrollers like the Arduino present us with nothing but a row of bare I/O pins. Finally we can tackle the question of what should go in, and what should come out: of transduction. A whole generation of artists, designers, nerds and tinkerers are taking up soldering irons and doing just that. Below: the Spoke-o-dometer from Rory Hyde and Scott Mitchell's Open Source Urbanism project.

One side-effect of this decomposition of computing is that the ontological status of the digital starts to break down with it. As Kirschenbaum shows brilliantly, the digital is just the analog operating within certain tolerances or threshholds. Thomas Traxler's The Idea of a Tree (below) is a solar-powered system that fabricates objects from epoxy, dye and string, by turning a spindle. Solar energy generates electrical energy, which drives the motor, which draws the string through the dye and onto the spindle: a chain of analog transductions produce an object that manifests specific changes in its local environment. The work is a beautiful demonstration that variability doesn't have to be worked up with generative code: if the system is open to it, it's already there in the flux of the material field.

This is not to dismiss computing, only to recast it: an incredibly dynamic, pliable set of techniques for manipulating the material environment. Paradoxically the very generalities of computing - the abstractions and protocols that insulate it from local, material conditions - make it a powerful tool for transduction, that is, the propagation of specificities. Usman Haque's Pachube is a generalised infrastructure, a set of protocols and standards that rest in turn on wider standards like XML, and which assume a whole stack of functional layers: IP, HTTP, and so on. All in order to propagate material patterns and flows from here to there: this is an architecture of transduction whose utopian aim is to "patch the planet" into a translocal ecology of linked environments.

Digital fabrication is part of the same shift: an expansion and extension of the computer's range of material transductions. Digital pattern, to lasercutter instructions, to physical form. Fabbing shows how material matters. It's unsurprising that a piece of laser-cut ply is aesthetically different to a luminous pattern of pixels; more interesting is the way computation reaches out into the substrate's material properties, and the range of potential applications and domains it opens up. Fabbing has often presented itself with a narrative of materialisation, making the virtual real, translating bits into atoms - Generator.x 2.0 was subtitled "Beyond the Screen." Not so: because of course, the "virtual" never was, and the screen is material too. Fabbing does get us beyond the screen, but only because its processes and materials have different properties, different specificities, and they hook us up to new contexts, as well as new sensations. (Below: Andreas Nicolas Fischer & Benjamin Maus: Reflection - from 5 Days Off: Frozen)

Transduction suggests a way to link practices like physical computing, fabrication, networked environments, and many more. Data visualisation - in the broadest sense, from poetic to fuctionalist - is about creating customised transductions, sourcing new inputs and/or manifesting new outputs (even if they don't reach "beyond the screen"). We could add tangible interfaces, augmented reality, and locative systems. What does all this amount to? In 1970 Gene Youngblood observed a similar moment as the dominant cultural form diversified into a networked, participatory, interdisciplinary field of practices. He called it expanded cinema. So perhaps we can call this expanded computing: digital media and computation as material flows, turned outwards, transducing anything to anything else.

Master of Digital Design / Grow Your Own Logotype

noreply@blogger.com (Mitchell) — Wed, 15 Apr 2009 02:49:00 +0000

Over the past year or so I've been working on a major new offering here at UC. So, I'm delighted to finally launch the new Master of Digital Design online. This course will offer something quite unique in the Australian context: a trans-disciplinary coursework Masters focused on digital practice for designers and creative practitioners of all sorts. The key practical approaches are generative techniques, data visualisation and design, and physical computing; and we'll be using these to address three core themes or questions: the urban, the public, and the sustainable.

As readers of this blog will know, these themes and approaches are right in line with my own research and creative interests; so frankly, I'm thrilled to be leading this course. Teaching with me will be a crew of talented designers, artists and researchers including Stephen Barrass, Sam Hinton and Geoff Hinchcliffe. Finally, we'll be drawing on the wisdom and experience of an international advisory panel whose work exemplifies what we mean by digital design - a practice that engages deeply, and critically, with digital processes, digital materials, and digital contexts: Karsten Schmidt, Rory Hyde, Nervous System, Anthony Burke and foAM.

The course launch has also provided a great excuse (er, opportunity) to play with some ideas around generative branding and marketing. I've been tinkering with this logotype for ages; it uses the same basic algorithm as Limits to Growth but artificially constrains the growth to a letterform (in the guise of a hidden bitmap image). Lately I've extended the logotype into a little generative marketing gadget; a Processing applet that lets you grow endless variations, and receive the results as a PDF file, attached to an email. The aim is to provide a little taste of the power - and pleasure - of generative design.

Behind the scenes this project was yet another demonstration of the brilliance of Processing and its community. The key technical challenge was the upload-and-email functionality. Seltar's "save to web" hack provided the template; upload image data over HTTP, and have a PHP script catch and save the file. From there it was relatively straightforward to have PHP generate the email, with the help of the Pear MailMime package. The final step was uploading a PDF, rather than a bitmap. This seemed impossible, because the built-in PDF library needs to write a local file, which means the extra annoyance of a signed applet. I posted a query on the Processing forums and within 24 hours PhiLho saved me with a solution that extends the PDF class to allow access to the PDF data as a Byte array, without first saving the file. Amazing: thank you! Add the super-useful ControlP5 for the UI sliders and buttons, and the whole thing is built on, in and with free, open-source software. Again, a demonstration of why digital design is such an exciting field of practice right now.

Watching the Street (Navigator) / citySCENE

noreply@blogger.com (Mitchell) — Sun, 15 Mar 2009 04:17:00 +0000

Vague Terrain 13: citySCENE has just launched. As editor Greg J. Smith writes:

This issue of Vague Terrain is founded on two notions - that the city is a stage set for intervention and an engine for representation.

The collection expands out from this premise in multiple directions: carto-mashups, projection-bombing, sound walks, psychogeographic imaging and ubicomp experiments. Early highlights for me included Crisis Fronts' Cognitive Maps and Database Urbanisms, which presents some impressive work on data visualisation and generative models as urban mapping strategies (below: Case Study: Los Angeles). Overall, on a first look, this collection is incredibly rich. It shows that a creative, wired-up, critical urbanism is not just a wisftul aspiration of the technorati, but a real practice.

Having said all that, it's a privelege to be a part of this collection. My contribution is Watching the Street (Navigator), a browsable visualisation of a single day of images from the Watching the Street dataset. It tests out the hunch that these time-lapse slit-scans can be used to read real patterns in the urban environment - that they are (or can be) more than just suggestive abstractions. It uses a simple interface to display both a single source frame, and a correlated slit-scan visualisation, with image-space and time-space sharing an axis, a bit like a slide rule. Greg Smith called it an "urban viewfinder", which sums the intention up nicely.

Playing with the navigator for a while seems to confirm that hunch. The composites reveal temporal patterns in the environment, but not the spatial context that allows us to identify their causes; the source frames show that spatial context, but not the change over time. Reading the two against each other involves chains and cycles of discovery, analysis and inference. These might be open-ended (spatiotemporal browsing) or more directed. What time do the sandwich-boards go out? How long does the delivery truck stay?

Building the navigator presented some interesting technical challenges: mainly, how to make a web-friendly interface to 1440 source frames (240 x 320) and 480 slit-scan composites (720 x 320). That adds up to about 75Mb of jpegs. Processing 1.0 came to the rescue, with its new built-in dynamic image loader. requestImage() pulls in an image from a given URL, on cue, without bringing the whole applet to a grinding halt; it provides some basic feedback on the state of that image - whether it's loading, loaded, or un-loadable. I also blundered into two other useful lessons: how to use the applet "base" parameter, and how to manage Java's local cache, which kept throwing up earlier versions of the applet during testing.

Having made a lean, mean, browser-friendly version, I'm now thinking of adapting the navigator into a full-screen, offline app, with the whole eight-day dataset, and perhaps some tools for annotation and intra-day comparison. Best of all would be a long term installation; a sort of urban space-time observatory, watching the street but also opening it up to ongoing interpretation. If you'd like it running in your foyer, let me know.

JCSMR Curls

noreply@blogger.com (Mitchell) — Thu, 15 Jan 2009 23:45:00 +0000

This post is (belated) documentation of a project I worked on in 2007-8, creating an audio-responsive generative system for a permanent installation for the Jackie Chan Science Centre (yes, that Jackie Chan) at the John Curtin School of Medical Research, on the ANU campus. Along with some Processing-related nitty gritty, you'll find some broader reflections on generative systems and the design process. For less process and more product, skip straight to the generative applets (and turn on your sound input).

In mid 2007 my colleague Stephen Barrass and I were approached by Thylacine, a Canberra company specialising in urban art, industrial and exhibition design. Caolan Mitchell and Alexandra Gillespie were designing a new permanent exhibition, the first stage of the new Jackie Chan Science Centre, housed in a new building - a razor-sharp piece of contemporary architecture (below) by Melbourne firm Lyons. Instead of just bolting a display case and a few plaques to the wall, Mitchell and Gillespie (wonderfully) proposed a design that hinged on a dynamic generative motif - a system that would ebb and flow with its own life cycles, and echo the spiral / helix DNA structures central to the School's work, and already embedded in the building's architecture.

My initial sketches (below) took the spiral motif fairly literally, drawing vertical helices and varying their width with a combination of mouse movement and a simple sin function - the results reminded me of the beautiful spiral egg cases of the Port Jackson Shark. At that stage we were talking about the possibility of projecting back onto the facade of the building, which has big vertical glass panels; this structure informed the vertical format. I made a quick video mockup of the form on the facade - which was incredibly easy, thanks to the robust, adaptable, extendable goodness of Processing (a recurring theme in the process to come).

These sketches meet the simplest criteria of the brief (spiral forms) but do nothing about the more interesting (and difficult) ones: cycles of birth, growth and death, and dynamics over multiple time scales. Over the next couple of months I developed two or three different approaches to this goal.

The phyllotaxis model blogged earlier was one attempt. Spurred on by the hardcore a-life skills of Jon McCormack and co. at CEMA, I built a system in which phyllotactic spirals self-organised spontaneously. Because in Jon's words, anyone can draw a spiral, what you really want is a system out of which spirals emerge! The model worked, but I had trouble figuring out how phyllotactic spiral forms might meaningfully die or reproduce. Also, by that stage I had two other systems that seemed more promising.

From the early stages I wanted to make the system respond to environmental audio. The installation would be in a public foyer with plenty of pedestrian traffic, so audio promised a way to tap in to the building's rhythms of activity at long time scales, as well as convey an instantaneous sense of live interaction. In the two most developed sketches audio plays a key role in the life cycle of the system.

One sketch moved into 2d, and started with a pre-existing model for growth, by way of the Eden growth algorithm (this system would later be adapted again into Limits to Growth). I had already been playing with an "off-lattice" Eden-like system where circular cells could grow at any angle to their parent (rather than the square grid of the original Eden model). This system also made it easy to vary the radius of those cells individually. The next step was to couple live audio to the system; following a physical metaphor, frequency is mapped to cell size, so that larger cells responded to low frequency bands, and smaller cells to high frequencies. Incoming sound adds to the cell's energy parameter; this energy gradually decays over time in the absence of sound. Cell reproduction, logically enough, is conditional on energy.

The result is that cells which are best "tuned" for the current audio spectrum will accumulate more energy, and so are more likely to reproduce, spawning a neighbour whose size (and thus tuning) is similar to, but not the same as, their own; so over time the system generates a range of different cell sizes, but only the well-tuned survive. The rest die, which in the best artificial life tradition, means they just go away - no mess, no fuss. In the image below cells are rendered with stroke thickness mapped to energy level. The curves and branches pop out of rules sprinkled lightly with random(), resulting in a loose take on the spiral motif, which is probably the weak point in this sketch. I still think it has potential - nightclub videowall, anyone? Try the live applet over here (adjust your audio input levels to control the growth / death balance).

The third model takes this approach to energy and reproduction - about the simplest possible a-life simulation - and folds it back into the helical structures of the first sketches. In this world an individual is a 3d helix, built from simple line segments. Again each individual is tuned to a frequency band, which supplies energy for growth; but here "growth" means adding segments to the helix, extending its length. Individuals can "reproduce", given enough energy, but here reproducing means spawning a whole new helix, with a slightly mutated frequency band. All the helixes grow from the same origin point - they form a colony, something like a clump of grass.

This sketch went through many variants and iterations over the next month or so; in retrospect the process of working to a brief, within a design team, pushed this system further than I ever would have taken it myself. At the same time I was testing the system against my own critical position; I've argued earlier that the generative model matters, not just for its generativity but the entities and relations it involves.

From that perspective this system was full of holes. Death was arbitrary: just a timer measuring a fixed life-span. "Growth" was a misnomer: the number of segments was simply a rolling average of the energy in the curl's frequency band, so the curls were really no more than slow-motion level meters. Taking the organic / metabolic analogy more seriously, I worked out a better solution. An organism needs a certain amount of energy just to function; and the bigger the organism, the more energy it needs. If it gets more than it needs, then it can grow; if it gets less than it needs, for long enough, it will die. So this is a simple metabolic logic that can link growth, energy and death. Translated into the world of the curls: for each time step, every curl has an energy threshhold, which is proportional to its size (in line segments); if the spectral energy in its band is far enough over that threshhold, it adds a segment - like adding a new cell to its body; if the energy is under that threshhold, it doesn't grow; and if it remains in stasis for too long, it dies. Funnily enough, the behaviour that results is only subtly different to the simple windowed average. Does the model really matter, in that case? It does for me at least; if and how it matters for others is another question.

Next, the curls developed a more complex life-cycle - credit to Alex Gillespie for urging me in this direction. In line with the grass analogy, curls grow a "seed" at their tip when they are in stasis; when they die, that seed is released into the world. Like real seeds, these can lie dormant indefinitely before being revived - here, by a burst of energy in their specific frequency band. After several iterations, the seed form settled on a circle that gradually grows spikes, all the while being blown back "down" the world (against the direction of growth) by audio energy (below). As well as adding graphic variety, seeds change the system's overall dynamics. Unlike spawned curls, seeds are genetically identical to their "parent" - attributes such as frequency band are passed on unaltered. Because each individual can make only one seed, that seed is a way for the curl to go dormant in lean times; if it gets another burst of energy, it can be reborn. The curls demo applet demonstrates this best (again, adjust your audio input and make some noise).

A few technical notes. One big lesson here was the power of transform-based geometry. Each curl is a sequence of line segments whose length relates to frequency band (lower tuned curls have longer segments); each segment is tilted (rotateZ), then translated along the x axis to the correct spot. A sine function is used to modulate the radius of each curl along its length; frequency band factors in here too; this radius is expressed as a y axis translation. Then the segment is rotated around the x axis, to give depth. I iterate this a few hundred times to get one curl, and repeat this process up to twenty times to draw the whole world - each curl has its own parameters for tilt, x rotation increment, and frequency band.

In the live applet audio energy ripples up the curls, from base to tip. This was added to reinforce the liveness of the system and add some rapid, moment-by-moment change. It was implemented very simply. I used a (Java) ArrayList to create a stack of audio level values; at each time step, the current audio level value is added at the head of the list, and the ArrayList politely shuffles all the other values along. So each segment's length is a combination of three values; the base segment length, a function to taper the curl towards the tip, and the buffered audio level.

The graphics are all drawn with OpenGL - following flight404 I dabbled with GL blend modes, specifically additive blending, to get that luminous quality. The other key visual device here is the smearing caused by redrawing with a translucent rect(); instead of erasing the previous frame completely this fades it before overlaying the new frame. It's an easy trick that I've used before. But as Tom Carden explains, in OpenGL it leaves traces of previous frames. I discovered this firsthand when Alex and Caolan asked whether we could lose the "ghosts." I was baffled: on my dim old Powerbook screen, I simply hadn't seen them. Eventually, juggling alpha values I could reduce the "ghosts" to almost black (1) against the completely black (0) background - but no lower. Finally I just set the initial background to (1) instead of (0), and the ghosts were gone.

The adaptability of Processing came through again when it came to realising the installation. The final spec was a single long custom-made display case, with three small, inset LCD panels. These screens would run slide shows expanding on the exhibition content, but also feature the generative graphics when idle; the case itself would also integrate the curls as a graphic motif. For the case graphics, I sent Thylacine an applet that output a PDF snapshot on a key press; they could generate the graphics as required, then import the files directly into their layout.

The screens posed some extra challenges. The initial idea was to have the screens switch between a Powerpoint slideshow, and the curls applet; but making this happen without window frames and other visual clutter was impossible. In the end it was easier to build a simple slide player into the applet: it reads in images from an external folder, allowing JCSMR to author and update the slideshow content independently.

So to wrap up the Processing rave: it provided a single integrated development and delivery tool for a project spanning print, screen, audio, interaction, animation and even content management. Being able to burrow straight through to Java is powerful. Development was seamlessly cross-platform; the whole thing was developed on a Mac, and now runs happily on a single Windows PC with three (modest) OpenGL video cards. The installation has run daily for over six months, without a hitch (touch wood).

Some installation shots below, though it's hard to photograph, being a glass fronted cabinet in a bright foyer - reflection city. I'll add some better shots when I can get them. If you're in Canberra, drop in to the JCSMR - worth it for the building alone - and see it in person.

And very finally, photographic proof of the Jackie Chan connection - image from The Age.

Fabricated Growth Forms (Processing to Ponoko)

noreply@blogger.com (Mitchell) — Fri, 19 Dec 2008 05:49:00 +0000

Like many others playing with generative techniques, I'm fascinated by the potential of digital fabrication. Getting beyond the screen and into the world of objects is a significant move for a field that has, until the last few years, reveled in its own immateriality. There's a lot to think about in this material turn, but that's for another post. Here, a quick report on my first experiment with generative fabrication.

I don't have a laser cutter handy at my workplace (though as William Turkel writes there are lots of good reasons why I should) so I decided to check out Ponoko; I wanted to see what was involved in generating, uploading and fabbing a small project. I started with the Processing sketch from Limits to Growth, and tweaked it to turn out much smaller forms (a few hundred nodes, rather than tens of thousands). I used the built-in PDF export, then opened the PDFs in Illustrator. (Illustrator is the only commercial/proprietary software step in this process, so I'd be interested to hear of any alternatives). The forms are drawn as linked line segments of varying stroke widths. Ponoko needs an EPS with only the outside edge of this form, so I used Illustrator to merge it into a composite path, then set the stroke colour and width as instructed (0,0,255 and 0.001mm).

The upload to Ponoko took a few tries - I was getting some strange errors as their system failed to "see" the cutting paths on the template - but after some swift and cheerful technical assistance it all worked. Pricing was also trial and error; the first design I uploaded was more complex than these, and of course these branching forms pack a long cutting path into a small surface area. I simplified the design, packed four forms onto a sheet, and opted for 4mm ply rather than acrylic. Final cost including (expensive) shipping to Australia was about $A60 (currently around $US40). Not what I'd call cheap, but not prohibitive either. There are intricate discussions of the economics of the business - shipping, exchange rates, local vs global, etc - on the Ponoko forums.

Eighteen days later, they arrived. Novelty counts for a lot here, but still, I'm totally charmed by these objects. A few surprises, but all good: they are smaller and finer than I imagined, and they smell very slightly of charred wood (excellent!). The cut edges are dark with a nice smooth, burnished surface, and the ply surface is clean. The scale and intricacy of the things seems to entice people to touch and handle them. I find them far more satisfying than the (much more detailed) laser prints I made with the same system.

Immediately it's clear how the fabbing process, and the materials, can reach back up through the production chain and influence the design and the generative system. One flaw in the design is a product of how I'm drawing the shapes: there are small rounded "shoulders" at the joints between line segments, caused by the overlap between one rounded line cap and the next segment - this is obvious in the physical forms. Better to draw the segments as tapered rectangles, and avoid the shoulders. Also, the branching topology is structurally risky; how to introduce more joins without breaking the generative model? This interplay, between computational process, manufacturing process, material and form, seems really promising. Ponoko seems to be an excellent, affordable way to try this out, and the built-in fab-on-demand shopfront is great, if you want to sell your wares. But it's still, ironically, working with a mass-production paradigm of one design, n copies. With hooks for a more dynamic, generative front end, it could get really interesting: designers like the wonderful Nervous System are doing this already. More documentation of the growth forms over on Flickr.

Watching the Street

noreply@blogger.com (Mitchell) — Thu, 27 Nov 2008 10:35:00 +0000

The recent Dorkbot show seemed to go off nicely - it was great to be part of such a strong show of local work (some documentation). I showed some prints from Limits to Growth, as well as a more experimental process piece, Watching the Street - a (sub)urban remake of Watching the Sky.

Credit to Nathan McGinness for the suggestion: use the same time-lapse / slit-scan technique to image change in an urban environment. Technically, the setup was fairly straightforward. Instead of a digital stills camera I used a webcam (in portrait orientation), and wrote a simple Processing script to save stills at one-minute intervals, while extracting and compiling one-pixel slices into 24-hour composites. The webcam was installed in a window box on the gallery street front, with a view across the road, under a street tree, to one of Manuka's low-rise shopping arcades (above). I also attached a printer to the installed rig, so that a new composite could be produced and pinned to the wall each day. So here, some of the resulting images, and a bit of commentary.

The image-gathering process got off to a rocky start. After a few hours, the webcam came unstuck from the side of the window-box, and lay forlornly on its side for the next 48 hours (here's what that looks like). I gaffed it back in place just before the opening, and restarted the capture in time to catch some gallery-goers loitering around out the front.

These two are the Frday the 7th and Saturday the 8th of November, the first two full day composites. Those striped rectangular chunks around mid-frame are cars, parked in the 30 minute loading zone accross the road. Some stay for a few minutes, a couple for what looks like an hour. Of course on the Saturday, the loading zone doesn't operate, and there's a single car parked in it from mid-morning to mid-afternoon. The single-pixel vertical shards give an indication of passing car and pedestrian traffic.

A quiet, sunny Sunday the 9th; the form hinted at on the 8th, reveals itself as the shadow of the big plane tree, creeping across the footpath. Then the following Friday the 14th. It's all happening; lots of car and pedestrian traffic, changes in sunlight, looks like an afternoon breeze in the foliage as well. The dominant, bluish horizontal stripe in all these images is the neon sign on the shopping centre - which runs all night. The orange rectangle that extends into the evening is the interior light of a shop - which you'll notice switches off at slightly different times each night.

So you'll notice that as in Watching the Sky, I'm persisting in reading these as visualisations of the environment, as well as digital images in themselves. I'm struck by how this simple, indiscriminate process reveals both expected and unexpected patterns, and continues to provoke new questions. This despite, or I would argue because of, its openness to multiple material / temporal systems. In an interesting bit of synchronicity, I was teaching in the UTS Street as Platform masterclass with Dan Hill (more on that soon) while this piece was running. Could a simple visualisation process like this function "informationally", as it were; to help answer real questions about a very specific slice of urban environment, in near-real time? More interesting for me, could it function in that way without prescribing the question in advance - that is, could it support an open-ended process of exploration and interpretation? I'm planning to build an interactive version of this piece, to try out these ideas. In these static visualisations there's a huge amount of data missing: I set the slice point more-or-less arbitrarily, so there are 479 other potentially interesting slices to browse. It would be nice to be able to change the slice point dynamically, as well as navigating through the source images. I notice that Processing 1.0 (yay!) now supports threaded loading of images: could come in handy. Meanwhile, the full set of composite images are up on Flickr.

Dorkbot CBR at Manuka CCAS

noreply@blogger.com (Mitchell) — Mon, 27 Oct 2008 03:36:00 +0000

Dorkbot Canberra's inaugural group show opens Thursday November 6th at Canberra Contemporary Artspace Manuka. It's a great, super diverse lineup, including wearables, data art, solar power, generative grunge, drawing machines and audiovisuals. I'll be showing a big crop of prints from Limits to Growth, as well as doing a kind of urban version of Watching the Sky, gathering images from the street. Here's the full press release.

Synesthesia and Cross-Modality in Contemporary Audiovisuals

noreply@blogger.com (Mitchell) — Wed, 01 Oct 2008 23:58:00 +0000

Though written about a year ago, this essay has just been published in Senses and Society. It's related to the Synchresis project posted earlier but makes a more rigorous investigation of synaesthesia, as it is (so often) applied to fused or algorithmic audiovisuals. After a quick tour through the history of synaesthesia in the arts, it uses some nifty perceptual neuroscience to argue for an alternative model, of contemporary audiovisuals as cross-modal objects that reveal the space of relation between modalities - the map. It takes work by Andrew Gadow (below) and Robin Fox as case studies, but also touches on Oskar Fischinger, Robert Hodgin, Norman McLaren and others. The version here has plenty of pics and vids; for a more paper-based experience grab the pdf (and please use the print version for any citations). Oh and pardon the American spellings here - journal style.

In the age of ubiquitous digital media, synesthesia is everywhere. In human, neurological form, it is rare: for perhaps three in a hundred people, a stimulus in one sensory modality automatically induces a sensation in another. Auditory-to-visual synesthesia, or “colored hearing” is much rarer still. Yet now this phenomenon is realised, apparently, inside every digital music player, on VJ screens in every club, in robot lightshows. On these screens sound is transformed into visual pattern and form instantly and automatically; an exotic perceptual phenomenon becomes a technically mediated commonplace.

In fact digital synesthesia is a trope that occurs in the production and use of mainstream digital media, as well as the media arts. Computer users find audio visualisers built in to their music players; as this software shows, audiovisual relations can now be reduced to an algorithm, a formal procedure that interprets (sound) and emits (image) data; though in this case the results are mostly mundane psychedelia. In some recent music video audio visualisations are integrated into the narrative and performative conventions of the genre; in Justin Timberlake’s Lovestoned video (2007) the singer’s image is constructed from the flickering bands of a visualised audio spectrum. Here a technically guaranteed unity of sound and image is literally reinscribed on to the performing artist. In contemporary media arts practice the same techniques – computational analysis of sound driving generated visual elements – are widespread, and its aesthetics more diverse. In custom-coded audio visualisations artists such as Marius Watz (2005) and Robert Hodgin (2007) (below) construct visualisations tuned to specific soundtracks; the automatism of digital synesthesia animates specific, constructed worlds of form and image. The algorithm becomes an endlessly variable and dynamic intermediary between sound and image.

In a distinct but related approach, some media artists have opted instead to simplify or reduce that audiovisual relation, ofen bypassing computation altogether. In Carsten Nicolai’s Telefunken works (2000,2004) the stereo output of an audio CD player is connected to the audio and video inputs of a television screen; what is heard as synthetic tones and noisy drones, is seen on the screen as patterns of monochrome form and line. In an approach I will refer to here as transcoding, sound and image are linked through a direct transfer of signal, a simple cross-wiring.

Australian artist Robin Fox plugs audio from a custom-built digital synthesiser into an oscilloscope; in the resulting hybrid instrument, Fox explores a territory in which signal is simultaneously heard and seen; every sound is a form in motion, every form a sound (above). The connection, the cross-wiring of sound to image, literally manifests the sensory cross-over of synesthesia; more, the work itself seems to somehow induce synesthetic experience. The correspondence between sound and image is immediate, agile and intense; the audiovisual relation is completely consistent, somehow self-evident, yet continually surprising. There’s a feeling of something like revelation; one reviewer describes Fox’s Backscatter DVD (2005) as “mesmerizing” and “overwhelming,” and hints at a sense of “greater significance or higher purpose” (Baker Fish 2005). Andrew Gadow’s work approaches the same relationship from the other side; working with an old video synthesiser, he transfers its image signal directly into audio. The scan-line structure of the video signal becomes audible as modulations of a 50 Hz hum; flickering, disintegrating visual textures become abrasive but intricately detailed buzzsaw audio. Again the subjective experience can be powerful, a visceral sense of force or encounter; the audiovisual coupling is so close that it seems to disappear, distinct modalities fuse into raw sensation.

Synesthesia is widely used as an analogy around this work. The analogy provides a mapping that aligns subjective sensation with audiovisual signals; it maps perceptual or even neurological structures onto technical structures. The analogy also plays another role, foregrounding sensation in the reception of the artworks; proposing to operate, for the subject, at the level of direct sensation. Finally, synesthesia also connects this contemporary work with a historical artistic tradition. The new automatic or transcoded fusion of sound and image seems to mark the culmination of a practice spanning music, painting, film and electronic media and aspiring, as Jeremy Strick writes, to the ideal of synesthesia as “the unity of the senses, and, by extension, the arts.” (Strick 2005: 15) The 2005 Visual Music exhibition, curated by Strick, documents this tradition in detail, as well as making a bid for its continuation into the present:

In digital media ... music and visual art are ... created out of the same stuff, bits of electronic information ... . [T]he aspiration to novel experience created by the compounding of sensation and association has never been more possible. (ibid.)

This paper’s main aim is to test this analogy, and the related historical drive that Strick suggests; to consider if, and how, such practice can be thought of as synesthetic, and examine structural parallels between synesthesia as a perceptual and neurological phenomenon, and the automatic or transcoded linking of audio and visual media? Following the tradition of artistic synesthesia that Strick invokes, the approach here is to provisionally ignore the glaring gap in this analogy, between subjective sensation and objective, technical artefact. Scientific work in perception and neuropsychology is drawn in for a more detailed account of synesthesia; but it also offers an alternative model for this practice, based on theories of cross-modal interactions in normal perception. Close correlations between sound and image are, after all, an everyday perceptual occurrence. From this perspective, tightly correlated audiovisuals direct us towards the abstract structures that are its generative materials: signal, as distinct from image or sound; and the map, the pattern of correlation between signals in different domains. Although artists such as Fox and Gadow use obsolete, analog technologies, their work is a sensory manifestation of these characteristically contemporary abstractions.

Synesthesia

Scientists studying synesthesia define it as occurring “when stimulation of one sensory modality automatically triggers perception in a second modality” (Harrison and Baron-Cohen 1996: 3). There are many documented forms of synesthesia; stimuli such as numbers, letters, words, days of the week, and musical tones may trigger perceived color and shape; taste, smell and pain can also trigger perceptions of shape. Estimates of the prevalence of synesthesia vary widely between 1 in 20 and 1 in 20 000; one recent study found a prevalence of around 3% and showed that calendar-color and letter- and number-color forms occur most often, while audiovisual synesthesia or “colored hearing” is comparatively rare (Simner et al 2006).

After long being debunked or treated as a curiosity, synesthesia has attracted increasing scientific attention, and validation, in recent years. Neuropsychologist Richard Cytowic undertook one of the first modern studies (Cytowic 1989). As well as a basic validation – finding that synesthesia is a real phenomenon with a neurological basis – Cytowic proposed a set of diagnostic and clinical features of the condition (Cytowic 1996: 23-31). He found that synesthesia is “involuntary but elicited,” an automatic perceptual experience that cannot be supressed or controlled. Synesthetic perceptions are “durable and generic,” meaning that an individual’s cross-sensory connections do not change over their lifetime, and that synesthetic perceptions are elementary and general, rather than “elaborated” – for example colors and simple shapes, rather than a detailed mental picture. Cytowic also points out that synesthetic perceptions are unusually memorable: some synesthetes use their triggered percepts as an index that aids their recall of the evoking stimulus. Moreover, Cytowic states, synesthesia is an emotional experience, “accompanied by a sense of certitude (the ‘this is it’ feeling)” that he links to William James’ description of religious ecstasy, and in particular the affect of noesis, “knowledge that is experienced directly, an illumination that is accompanied by a feeling of certitude.” It is in part this affective dimension that leads Cytowic to propose a linkage between synesthesia and the limbic brain – associated with emotion and a sense of “salience.”

More recent science has continued to investigate the neurology and psychology of synesthesia, using modern imaging techniques to show activation in the anatomy of the synesthetic brain, as well as behavioural experiments that seek out the parameters of synesthetic experience. Recent work has confirmed Cytowic’s finding that synesthesia is involuntary and perceptually real, though the results also suggest that there is significant variation between synesthetic individuals (see for example Hubbard and Ramachandran 2005). Discussion centers on where synesthesia occurs in the notional chain of perceptual processing; for a minority of synesthetes it seems to occur early in the chain, before cognitive processes such as attention; for the rest it seems to occur later; Ramachandran and Hubbard label these forms"lower” and “higher” (2001: 14). There is general consensus that synesthesia has a neural basis in the form of increased connectivity between normally separate neural regions or modules (contrary to Cytowic’s limbic model), though the connective mechanism and architecture is debated. Some, including Ramachandran and Hubbard, propose that this connectivity is a result of defective “pruning” of neural connections, suggesting that synesthetic cross-wiring is a normal early developmental stage. The notion of synesthesia as common, underlying or originary is supported by the correlations between synesthetic and “normal” cross-sensory associations; despite individual differences in color-tone mapping, non-synesthetes and those with “colored hearing” make similar mappings between pitch and lightness (Marks 1996: 72). Similarly Ramachandran and Hubbard (2001:19) cite the consistency of an (albeit simple) mapping between shape and sound to support the same point: asked to link two shapes, one round, one spiky, with two names, bouba and kiki, subjects overwhelmingly associate kiki with the angular form and bouba with the rounded one. The authors continue, in one of the more expansive examples of synesthesia science, to propose links between this “normal” synesthesia and the angular gyrus, an anatomical region associated with cross-modal association as well as numeracy, the neurology of metaphor, emotion, art and the evolution of language. In this formulation synesthesia is an extreme case that offers clues to the neurology of normal – and significant – human abilities to associate and synthesise disparate sensations and concepts.

Synesthesia in the Arts: Models and Maps

The history of synesthesia in the modern arts is well documented, and will not be recounted in detail here. More important for this argument is a sense of the major strains or variants of the concept, and their creative implications. This lineage has been traced to Romantic and Symbolist interests in the correspondence of the senses; poems of Rimbaud and Baudelaire correlated letters, colors, smells and sounds (see for example Cook 1998:25). In what Judith Zilczer (2005: 26) terms the mystical strain of artistic synesthesia, these sensory correspondences were held to refer to a higher, unitary reality, informed by Theosophy and Romantic philosophy. Kandinsky, reputedly a synesthete with “colored hearing,” is the best known of a group whose painterly abstraction was informed by musical and synesthetic analogies. In Kandinsky’s Concerning the Spiritual in Art (1977: 25) sensation, and especially color, can set the soul vibrating like a musical instrument: his aim was a form of absolute or “nonobjective” visual art, comparable to music. Cook (1998: 46) describes this model, informed by Goethe’s philosophy, as triangular. Both sound and color derive from the spiritual, or higher vibration, at the apex; thus sound and color have no inherent correspondence but “correspond to one another in so far as they embody the same ultimate meaning.” In a second wave of visual music between the wars, artists such as Paul Klee, Man Ray, Georgia O’Keefe and Arthur Dove adopted more concrete and structural models of correspondence, attempting to map harmony, counterpoint and rhythm into the abstract picture plane (see for example Zilczer 2005: 52-67). Synesthesia itself plays a shifting role in this context. Kandinsky and composer Alexander Scriabin seem to have experienced it, while many other artists were inspired by, or in some cases literally borrowed, synesthestic correspondences. Discussing the influence of Kandinsky’s note-color correpondences on Schoenberg, Cook (1998: 49) proposes a “cultural synesthesia” – where the idea of sensory correspondence can carry a cultural value independent of its actual experience.

In fact cultural synesthesia – evoked, suggested, implied or idealised synesthesia – dominates the visual music tradition; there are very few instances of actual, spontaneous, automatic audiovisual correspondences. In the work of Messiaen, Scriabin and perhaps Schoenberg (via Kandinsky), synesthetic experience formed the basis for a systematised set of pitch-color correspondences, though even these are not straightforward. The correspondences are different for each composer, as we would expect based on recent science. Moreover each is conditioned by what Cook (1998:46) argues is a mixture of subjective and cultural factors. Any correspondence between the continuous color spectrum and the discrete values of the Western twelve-tone scale is dubious – though these correspondences flourished in the early twentieth century in the “color organs” of Rimington and others (see for example Cook 1998:37 and Peel 2006). Later emblematic practitioners of the visual music tradition, John and James Whitney, used tightly composed but again ultimately arbitrary relations between sound and vision. If, as Strick argues, this creative tradition aspires towards synesthesia, when it comes to practically manifesting that sensory relation it founders on the problem of the map, the pattern of correspondences. Of all possible relations between sound and vision, what is it that makes one different, or preferable, to another? While recent science suggests some underlying perceptual commonalities, the devil, and the aesthetic, is in the detail.

Animator Oskar Fischinger provides a near-precedent for transcoded audiovisuals, and demonstrates one possible solution to the question of the map. Fischinger’s Ornament Sound experiments of 1932 (above) explored the double identity of the optical film soundtrack, printing regular visual patterns into the 3mm-wide sound strip at the edge of the frame, enabling them to be automatically rendered as sound. Fischinger (1932) emphasised the potential of this technique for composers: “control of every fine gradation and nuance is granted to the music-painting artist.” This form also promised a newfound “definitive” control over performance: “his creation, his work, can speak for itself directly through the film projector.” Fischinger also recognises the visual interest in recorded sound waves; and although he anticipated their use in conjunction with animation, he did not envisage the “sounding ornaments” as visual content in themselves. Nonetheless, Fischinger had found a space of audiovisual correspondence that was preexisting and “definitive,” yet seemingly had limitless creative potential. The later work of animator Norman McLaren developed Fischinger’s techniques, synchronising hand-drawn optical soundtracks with animation in Dots (1940), and finally using the synthesised optical soundtrack as synchronised visual source material in Synchromy (1971) (below).

Contemporary transcoded audiovisuals realise Fischinger’s experiments by similarly bypassing, or rather abdicating, the question of the map. This is not to say that the map disappears in an unmediated or inherent audiovisual connection. Instead, for Robin Fox and Andrew Gadow as well as Fischinger, the map is found, rather than constructed; it is embedded in the medium. Fox plugs his laptop into an oscilloscope, which maps the left and right channels of the audio signal into the x and y axes of its display. This audiovisual relation is in a sense a readymade, an existing cultural/technical artefact. Its process is literally hardwired, embodied in the analog electronics of the scope, just as Fischinger’s was in the optical technology of film.

Fused AV and Synesthesia

Do transcoded audiovisuals then realise the synesthetic ideal, or literalise the analogy? We can draw some correlations. To recap, the current scientific consensus is that synesthetic perceptions are real, automatic and involuntary, and caused by neural cross-connections at some level of the perceptual system. The cross-mappings of synesthetic perceptions are highly variable from one individual to another, but highly consistent for the individual. The transcoding approach of artists like Fox and Gadow seems fairly close: in Fox’s oscilloscope work for example, images are created “automatically” as Fox feeds audio to the oscilloscope, cross-wiring audio to vision; Fox uses the oscilloscope’s hardwired audiovisual map, which is fixed and consistent; but that mapping is different to, for example, Gadow’s equally automatic sound-to-image mapping, based on the interchange of analog video and audio signals. Even the visual aesthetics of these works could be likened to reports of colored hearing: in Cytowic’s terms these are not “elaborated” percepts, but simple, abstract elements.

The synesthetic affect that Cytowic’s study identifies is also suggestive. Fused audiovisuals can evoke (for some at least) a similar sense of revelation or noesis. Fox’s oscilloscope works show us something that feels both self-evidently “right” and surprising or unimaginable; the primal phosphorescent dot shows us its universe, a set of relations that are manifestly coherent and consistent, but whose implications are unforeseeable. Fox’s compositional structure emphasises the process of revelation at times: Photosynthesis (AOR) (above), the opening track on his Backscatter release, offers an initially gentle introduction, as the single point of the trace is buffeted by rhythmic subsonic clicks before slowly unfurling into harmonic pattern; but by the end of that track wave after wave of complex, nested forms have emerged and co-modulated; each point on the path is another noetic moment yet each is consistent and coextensive with the others. Mandala I, following, demonstrates almost the opposite approach, as Fox’s micro-switched digital twitches call up flickering variants on the circular carrier wave; to push the cosmological analogy, this is some kind of faster than light travel – we traverse many places at once – but again there is a revelatory quality as we witness accumulating relations, both momentary – between each sound, its corresponding form and movement – and sequential, between each sound/form/movement and the next, and the next.

How far can this line go, though, before it falls into the yawning gap in the analogy? Audiovisual works are artefacts; objects of perception, not perceptions. To put it bluntly, synesthesia, by definition, occurs in the perceptual system of a synesthete, not in the crossed connections of a video synth. Once again, we can use the gap as a provocation, rather than an obstacle. One response is to think of these works not as replicating human neurology, but rather something else. “Artificial synesthesia” is the term used by Dutch neuroscientist Peter Meijer (n.d.) to describe his work on sensory substitution; his vOICe system transcodes video from a small camera, into synthetic audio, in an attempt to use sound to provide visual information to those with little or no vision. In Meijer’s words, “we are interested in forms of learned synesthesia (acquired synesthesia) that might result from machine-generated crossmodal mappings.” Among other things Meijer’s work suggests that perception is not a fixed set of channels, but a reconfigurable network; over time, blind users of the vOICe seem to integrate image transcoded into sound, as functional vision. A recent paper shows that the lateral-occipital tactile-visual area of the brain, normally associated with the tactile and visual perception of shape, is activated by expert vOICe users (Amedi et al 2007). Other work in the field of sensory substitution suggests that different forms of synesthesia can also be acquired: Peter König’s feelSpace belt conveys orientation through vibrating touch, providing a augmented sense that some volunteers were able to integrate over time (see Bains 2007).

Are transcoded audiovisuals some form of sensory substitution or artificial synesthesia? There are two important differences. Sensory substitution operates by mapping an otherwise absent modality into an existing one; absent vision into existing hearing, in the case of the vOICe, and absolute orientation into touch, in the feelSpace belt. However for most, audiovisual transcoding links two existing modalities, “channels” already in perceptual use. Secondly, sensory substitution involves long-term integration and interaction with the environment; we can learn new “channels” but only by feeling out and (literally) incorporating their correlations with our existing sensory matrix. There are some striking parallels, and transcoded AV certainly hints at artificial synesthesia and a rewired sensorium, but as bounded aesthetic objects these works cannot realise that perceptual transformation.

The Pleasures of Binding

Correlation, key to artificial synesthesia, offers an alternative approach to the perceptual aesthetics of fused audiovisuals. At the core of transcoded and other tightly linked audiovisual forms is an experience based on a correlation between auditory and visual elements. While synesthesia offers a neurological analogy for the generation (poetics) of fused AV, this correlated quality leads into the neuroscience of perception, and thus offers a way to frame these works from the other side, the side of reception (or aesthetics).

The detection of correlations in the perceptual field is a normal, and crucial, perceptual task. From an ecological perspective, correlations underpin the recognition of objects in an organism’s environment. Our perceptual systems “bind” correlated elements into groups that often correspond to objects in our physical environment. A cat hiding in the garden might initially appear as an unrelated set of visual elements – a light grey splodge here, a dark shape there. When we “see” the cat, we detect correlations between those elements that enable us to interpret them as part of an underlying object. The image of a hidden Dalmatian dog (below) is often used to illustrate this phenomenon.

As Ramachandran and Hirstein (1999: 21-23) point out, this process of binding has some interesting features. Binding is “sticky” – we seem to hold on to bound perceptual elements. Once seen, the Dalmatian cannot be un-seen a without a conscious effort. Moreover, the act of making a binding is pleasurable in itself: “the discovery of the dog and the linking of the dog-relevant splotches generates a pleasant ‘aha’ sensation.” The authors offer an evolutionary rationale for this payoff: “The very process of discovering correlations and of ‘binding’ correlated features to create unitary objects or events must be reinforcing for the organism – in order to provide incentive for discovering such correlations.” Our limbic system apparently rewards us for detecting sensory correlations in our environment, even in advance of the final “recognition” of an object: “at every stage in processing there is generated a ‘Look, here is a clue to something potentially object-like’ signal that produces limbic activation and draws your atttention to that region or feature.” These incremental rewards “bootstrap” the final moment of recognition. Ramachandran and Hirstein work this perceptual pleasure principle into a neurological theory of aesthetic experience, suggesting that artists and designers seek out and intensify the pleasures of sensory binding, creating artefacts that “tease the system with as many of these ‘potential object’ clues as possible.”

Ramachandran and Hirstein also go further in proposing that the discovery of more abstract correlations is also reinforced by a limbic reward (1999: 31). They relate this to the ecological imperitive for classification – our evolved need to establish correlations that group and distinguish objects in our environment: say, edible versus inedible plants. This version of binding operates diachronically, rather than the synchronous binding of visual elements into a recognised form. “Being able to see the hidden similarities between successive distinct episodes allows you to link or bind these episodes to create a single super-ordinate category… Consequently the discovery of similarities and the linking of superficially dissimilar events would lead to a limbic activation – in order to ensure that the process is rewarding.”

Cross-Modal Perception

How might processes of binding – the discovery of correlations – operate in fused AV, where the characteristic correlations are between, rather than within, sensory modalities? While studies of perception have traditionally focused on the senses in isolation, as independent neurological “modules", recent work has begun to explore the relations between sensory modalities. Media-based metaphors for perception encourage us to think of the senses as functionally distinct input channels. If sensory substitution shows that these channels can be re-wired, studies of cross-modal perception show that they are barely even distinct. The senses are involved in what Shimojo and Shams (2001: 506) describe as “vigorous interaction and integration,” mirroring Michel Chion’s description of the “mutual contamination” that characterises the audiovisual relationship in film sound ([1990] 1994: 9). Shimojo and Shams review experiments showing the range of these mutual influences: how vision can alter the content and spatial location of perceived sound; and how sound can alter the perceived intensity and timing of visual stimulus. We hear what we see, and see what we hear.

The perceptual trickery of these experiments is less interesting than what they suggest about normal perception. Just as the binding of visual percepts into a whole enables us to recognise objects in our environment, correlations in different sensory modalities cause us to bind those stimuli into a unified perception. This is illustrated with another trick, an experiment by Sekuler et al (1997), in which subjects were presented with two moving dots on intersecting paths. Two perceptual interpretations of this animation are possible: that the dots pass each other without touching, or that they collide and bounce off each other. Without sound, the former interpretation was dominant; however adding a brief sound at the crossing point biased perception strongly towards collision. This is an instance of cross-modal binding, where correlated stimuli in different modalities become fused into a coherent whole. It also suggests the ecological basis of cross-modal binding; that we interpret correlated events as cues to objects in the environment. The interpretation of sensory data seems to be shaped by pre-conscious processes that bind percepts into wholes; wholes that map onto ecologically plausible events. In the crossing dots experiment, sound binds with vision to alter our interpretation of the event. The correlated stimuli point to a common cause, a model that explains their coherence.

Fused audiovisuals are aesthetic objects founded on cross-modal binding. Ramachandran and Hirstein’s notion of the pleasures of binding applies here; in the transcoded AV of artists such as Fox and Gadow we experience sensory fields that are somehow entirely bound: completely self-consistent, devoid of extraneous elements. The affect that Ramachandran and Hirstein attribute to the moment of binding, the discovery of the Dalmatian – the ‘aha’ of recognition – seems to be intensified and prolonged here. It also suggests a connection between cross-modal binding and the noetic affect Cytowic identifies in synesthetic experience. If we accept the limbic payoff theory of aesthetics, then perhaps fused AV is a manifestation of this pleasure principle in the media arts.

Audiovisuals as Cross-Modal Objects

Cross-modal binding is not limited to experimental audiovisuals, however; in fact the opposite is true. Cinema and television constantly rely on our predilection for binding sound and image; this is the basis of Chion’s synchresis, a “spontaneous and irresistable mental fusion” caused by close synchronisation ([1990] 1994: 63). Lip sync is the archetypal example, where audiovisual correlation breathes life into the image of a body. Recall the ecological function of binding: to identify a common cause – an object in the environment. In most audiovisual media the objects are (all too) readily apparent. So if audiovisual correlations refer us to a shared cause, what is that cause in fused or transcoded AV? What is the underlying object, the cat hiding in the garden?

In a sound-to-image mapping, for example, it seems logical to propose that the cause is the source modality – sound. This involves a kind of reflexive redundancy; in Fox’s oscilloscope work, it would mean that the image is simply a pointer to the soundtrack, that it doubles or duplicates the sound. Subjectively at least, the relation seems richer and more complex than that; and it seems at odds with an ecological model of perception. Perhaps the common object is not the sound, but something more abstract: the signal. Signal here refers to a pattern of differences or fluctuations, a flux that, like data, must always be embodied but which, again like data, can be readily transduced between one embodied form and another. Fox’s laptop does not send sound to the oscilloscope, or in fact to the audio amp; it sends signal, a pattern of fluctuating voltage. That pattern is manifest on the scope as phosphorescent image, and when it leaves the speakers, as sound: but their common origin is the flux itself.

In transcoded audiovisuals sound and image perceptually triangulate a third point, the signal, that is imperceptible in and of itself. Signal maps to perception through the contingencies of both media technologies and sensory boundaries, but in itself it traverses these limits. This is apparent in Fox’s work, where subsonic fluctuations modulate the audible frequencies to create movements that are easily seen, but felt only as sharp thumps; the speakers struggle to transduce the signal into mechanical energy. Many of the complex, pointillist visual patterns are created by square-edged signal forms that again are acoustically impossible; the scope, more agile, is better able to trace them out. Similar trans-sensory signatures occur in Gadow’s work and that of other transcoders; Gadow’s Techne (2005) opens with a still blue screen and a raw, buzzsaw hum. The hum has no movement or form; as becomes clear as the piece develops, it corresponds to the blue video background. It is the sound of the 50 Hz scan-line structure of the video signal itself; so it looks like almost nothing. This is not to say that transcoded audiovisuals are reducible to the signal, an abstract or perhaps “higher” ideal, as in Goethe’s triangular model of color and sound. Here sensation and experience are foremost; these experiments feel out the ramifications of signal in specific circuits and transductions.

As Ramachandran and Hirstein suggest (1999:31), perceptual binding is both synchronic and diachronic, instantaneous and sequential. If the moment-by-moment audiovisual binding in these works refers us to their shared cause – the signal – how do these works operate in the diachronic axis? They often share a simple formal structure of establishment, development and elaboration, a successive playing-out of potential. Ramachandran and Hirstein state that perceiving “hidden similarities between successive distinct episodes allows you to link or bind these episodes to create a single super-ordinate category.” We can think of the sequential similarities here as products of the constant, underlying structure that shapes all the outputs of the system. That structure is the map, the specific but abstract shape of the audiovisual correspondence. The map is an elusive entity; rather than an object we can think of it as a procedure, a verb or algorithm; a way of transforming between modalities and their shared signal. In Fox’s work the polar mapping of the oscilloscope is an algorithm that transforms phase – local relations in time – and amplitude into circular space. Considered as cross-modal objects, these works direct us to the underlying signal; and the signal is embodied audiovisually through the intermediary of the map. The map describes a space of potential, a range of possible correlations between domains; and it is that territory, I would argue, that these works reveal as they traverse it.

Inframedia and the Map

In earlier work on experimental sound I proposed the notion of inframedia, “a stratum below or within the mainlines of electronic media” (Whitelaw 2001:51). The noisy textures, resonant fuzz, glitches, crackles and pops of electronic music since the late 90s reveal “the sensory and affective textures of a media substrate, rather than media ‘content.’” That substrate is a critical domain; media infrastructures are more than technological artefacts; they are rapidly changing focii of power. Inframedia aesthetics reflect a consciousness of that domain, while in its processes such work often pursues local and particular manipulations, hacks or diversions of those media technologies.

Fused audiovisuals – a practice with close cultural links to experimental sound and music – can be approached along similar lines. Like hiss and hum, the audiovisual aesthetics of signal direct us to the abstraction and transduction occurring inside, or underneath, our media streams. Glitch-driven audio is founded on cracks in the surface – moments of interruption which allude to, and materialise, their own infrastructure. In a sense transcoded audiovisuals are a prolongation of those moments, leading to a flattening of the surface/depth dichotomy of glitch; the cross-modal coherence of this work is based on a sustained exploration of signal. Instead of mapping signal anthropomorphically onto perceptual “inputs”, these works show us where signal and affect meet or overlap, as well as where they diverge; they show us signal passing into, out of, and through perception.

These works also direct us to the map – the abstract space of possible transformations between signals. That domain of transformation is also inframedial, a key structure in digital media forms and cultures. Lev Manovich (2002) has described this question as the “built-in existential angst” of digital media: “By allowing us to map anything into anything else ... computer media simultaneously makes all these choices appear arbitrary….” In almost all digital media, the map – the pattern of relations between input and output – is imperceptible, obscured or encoded. This is clearest in the work of artists working explicitly with data inputs. In the work of Alex Drauglescu for example, spam email is used as the input to an algorithm that creates complex three dimensional forms. The mapping – the process that transforms spam into form – is never revealed, and so a concrete, specific process becomes a blank spot filled in with an impression of magical transubstantiation. In some computational work the artist provides source code, an explicit specification of the map but one that is highly encoded and unavailable, in itself, to perception. In most digital media objects, the map is inextricable from the residue or artefact it shapes. We perceive only the output, the image, sound or form, in which the input and its transformations are collapsed.

The wider significance of transcoded audiovisuals is that they approach a perceptual manifestation of the map, that space of transformation. We sense it, in these works, interpolated between each instant and every other. It’s perhaps not surprising that this characteristically digital figure is manifest through largely analog means; as well as a critical distance from the digital, analog signals offer transformations that are rich and immediate. Crucially the maps themselves are simple and static – highly reduced, compared to their digital counterparts – and so more available to the aesthetic and affective explorations of transcoded audiovisuals.

The prospect of somehow apprehending the map is both esoteric and pragmatic. The map is the inescapable intermediary, the necessary condition of our data-experience; but what is the map, what is its shape, how does it transform this into that? What are its conditions, limits, bounds? These works literally feel out the map, and in the process begin to address these questions, offering a sense of the abstract transformations that underpin contemporary digital culture.

Synesthesia is a powerful and persistent trope in the audiovisual arts. As shown here it offers some enticing parallels with the techniques and affects of audiovisual practice, yet as a techno-sensory analogy it has inherent limits. As in the visual music tradition, synesthesia plays a largely figurative role, and it demands critical scrutiny as such. In this investigation however, the synesthetic analogy has opened a path towards its more everyday converse, cross-modal perception, which offers a useful framework for a neuro-aesthetics of fused audiovisuals. These two approaches converge in the figure of the map, the space of correlation; the feeling of noesis or revelation common to both synesthesia and cross-modal binding, could be described as the affect of the map. That affect is central to the aesthetics of fused audiovisuals; though I would argue it offers more than a neurological hit; it brings us into contact with the abstract but culturally crucial terrain of the map itself.

References

Amedi, Amir, William M Stern, Joan A Camprodon, Felix Bermpohl, Lotfi Merabet, Stephen Rotman, Christopher Hemond, Peter Meijer and Alvaro Pascual-Leone. 2007. “Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex.” Nature Neuroscience 10: pp. 687-689.

Baker Fish, Bob. 2005. Review of Robin Fox, Backscatter. Cyclic Defrost 10. http://www.cyclicdefrost.com/review.php?review=795

Bains, Sunny. 2007. “Mixed Feelings.” Wired 15.04. http://www.wired.com/wired/archive/15.04/esp_pr.html

Chion, Michel. [1990] 1994. Audio-Vision: Sound on Screen. New York: Columbia University Press.

Cook, Nicholas. 1998. Analysing Musical Multimedia. Oxford: Oxford University Press.

Cytowic, Richard. 1989. Synesthesia: a union of the senses. New York: Springer Verlag.

Cytowic, Richard. 1996. “Synesthesia, phenomenology and neuropsychology: a review of current knowledge.” In John E. Harrison and Simon Baron-Cohen (eds), Synesthesia: Classic and Contemporary Readings. London: Blackwell.

Fischinger, Oskar. 1932 “Sounding Ornaments.” Deutsche Allgemeine Zeitung (July 8). http://www.oskarfischinger.org/Sounding.htm

Harrison, John E., and Simon Baron-Cohen. 1996. “Synesthesia: an Introduction.” In John E. Harrison and Simon Baron-Cohen (eds), Synesthesia: Classic and Contemporary Readings. London: Blackwell.

Hubbard, E.M., and V.S. Ramachandran. 2005. “Neurocognitive mechanisms of synesthesia.” Neuron 48(3): pp. 509-520.

Kandinsky, Wassily. 1977. Concerning the Spiritual in Art. London: Dover.

Manovich, Lev. 2002. “The Anti-Sublime Ideal in Data Art.” http://www.manovich.net/DOCS/data_art.doc

Marks, Lawrence. 1996. “On colored-hearing synesthesia: cross-modal translations of sensory dimensions.” In John E. Harrison and Simon Baron-Cohen (eds), Synesthesia: Classic and Contemporary Readings. London: Blackwell.

Meijer, Peter. n.d. “Artificial Synesthesia for Synthetic Vision.” http://www.seeingwithsound.com/asynesth.htm

Peel, James. 2006. “The Scale and the Spectrum.” Cabinet 22. http://www.cabinetmagazine.org/issues/22/peel.php

Ramachandran , V.S., and E.M. Hubbard. 2001. “Synesthesia – A Window into Perception, Thought and Language,” Journal of Consciousness Studies 8(12): pp. 3-34.

Ramachandran, V.S., and William Hirstein. 1999. “The Science of Art: a Neurolgical Theory of Aesthetic Experience.” Journal of Consciousness Studies 6(6-7): pp. 15-51.

Sekuler, Robert, Allison B. Sekuler and Renee Lau. 1997. “Sound alters visual motion perception.” Nature 385: 308.

Shimojo, Shinsuke, and Ladan Shams. 2001. “Sensory modalities are not separate modalities: plasticity and interactions.” Current Opinion in Neurobiology 11: pp. 505-509.

Simner, Julia, Catherine Mulvenna, Noam Sagiv, Elias Tsakanikos, Sarah A Witherby, Christine Fraser, Kirsten Scott and Jamie Ward. 2006. “Synesthesia: the prevalence of atypical cross-modal experiences.” Perception 35(8): pp. 1024-1033.

Strick, Jeremy. 2005. “Visual Music.” In Visual Music: Synesthesia in Art and Music Since 1900 (exhibition catalog) . New York: Thames & Hudson.

Whitelaw, Mitchell. 2001. “Inframedia Audio.” Artlink 21(3): pp. 49-52.

Zilczer, Judith. “Music for the Eyes: Abstract Painting and Light Art,” in Visual Music: Synesthesia in Art and Music Since 1900 (exhibition catalog) . New York: Thames & Hudson.

Filmography

Fox, Robin. 2005. Backscatter. Videorecording. Melbourne: Synesthesia Records SYN012 DVD.

Gadow, Andrew. 2005. Techne. DVD-R courtesy of the artist.

Hodgin, Robert. 2007. “Trentemøller and Me.” http://www.flight404.com/blog/?p=52

McLaren, Norman. 1940. Dots. Animation. Available: http://www.youtube.com/watch?v=E3-vsKwQ0Cg

Mclaren, Norman. 1971. Synchromy. Animation. Available: http://www.youtube.com/watch?v=Jqz_tx1-xd4

Timberlake, Justin. 2007. “LoveStoned / I Think She Knows.” Zomba Recording. Music Video. Available: http://www.youtube.com/watch?v=GIYXHLlxD8U

Watz, Marius. 2005. “Video for @c: int.14/37.” http://www.unlekker.net/proj/cronica021/