Alex McLean

Paper on bricolage programming

2010-06-13T21:24:48Z

I’m very happy to have a paper “Bricolage Programming in the Creative Arts” accepted to this year’s PPIG (the Psychology of Programming Interest Group). You’re probably better off reading the PDF version, but I’ve put it here too:

Bricolage Programming in the Creative Arts

Alex McLean Geraint Wiggins

Centre for Cognition, Computation and Culture
Department of Computing
Goldsmiths, University of London

Abstract:
In this paper we consider artists who create their work by writing algorithms, which when interpreted by a computer generates their plotted drawings, synthesised music, animated digital video, or whatever target medium they have chosen. We examine the demands that such artists place upon their environments, the relationships between concepts and algorithms, and of cognition and computation. We begin by considering an artist’s creative process, and situating it within the bricolage style of programming. An embodied view of bricolage programming is related, underpinned by theories of cognitive metaphor and computational creativity, and finally with consideration of the bricolage programmer’s relation to time.

1 Introduction

Over the last decade, computer programming has enjoyed a major resurgence as a medium for the arts. A wealth of new programming environments for the arts, such as Processing, SuperCollider, ChucK, VVVV and OpenFrameworks have joined more established environments such as PureData and Max which have themselves gained enthusiastic adoption outside their traditional academic base. These environments offer varied approaches to supporting artistic use, including alternative programming languages, interfaces and workflow.

The purpose of the present discussion is to examine psychological issues which the resurgence of artistic programming has brought to the fore. What is the relationship between an artist, their creative process, their program, and their artistic works? We will look for answers from perspectives of psychology, cognitive linguistics, computer science and computational creativity, but first from the perspective of the artist.

2 Creative Processes

The painter Paul Klee [1953, p. 33] describes a creative process as a feedback loop: “Already at the very beginning of the productive act, shortly after the initial motion to create, occurs the first counter motion, the initial movement of receptivity. This means: the creator controls whether what he has produced so far is good. The work as human action (genesis) is productive as well as receptive. It is continuity.” This is creativity without planning, a feedback loop of making a mark on canvas, perceiving the effect, and reacting with a further mark. Being engaged in a tight creative feedback loop places the artist close to their work, guiding an idea to unforeseeable conclusion through a flow of creative perception and action. Klee writes as a painter, working directly with his medium. Programmer-artists instead work using computer language as a textual representation of their medium, and it might seem that this extra level of abstraction could hinder creative feedback. We will see however that this is not necessarily the case, beginning with the account of Turkle and Papert [1992], describing a bricolage approach to programming by analogy with painting:

The bricoleur resembles the painter who stands back between brushstrokes, looks at the canvas, and only after this contemplation, decides what to do next. Bricoleurs use a mastery of associations and interactions. For planners, mistakes are missteps; bricoleurs use a navigation of midcourse corrections. For planners, a program is an instrument for premeditated control; bricoleurs have goals but set out to realize them in the spirit of a collaborative venture with the machine. For planners, getting a program to work is like “saying one’s piece”; for bricoleurs, it is more like a conversation than a monologue. [Turkle and Papert, 1990, p. 136]

Although Turkle and Papert address gender issues in education, this quote should not be misread as dividing all programmers into two types; while associating bricolage with feminine and planning with male traits, they are careful to note that these are extremes of a behavioural continuum. Indeed, programming style is clearly task specific: for example a project requiring a large team needs more planning than a short script written by the end user.

Bricolage programming seems particularly applicable to artistic tasks, such as writing software to generate music, video animation or still images. Imagine a visual artist, programming their work using Processing. They may begin with an urge to draw superimposed curved lines, become interested in a tree-like structure they perceive in the output of their first implementation, and change their program to explore this new theme further. The addition of the algorithmic step would appear to affect the creative process as a whole, and we seek to understand how in the following.

2.1 Creative Process of Bricolage

Figure 1: The process of action and reaction in bricolage programming

Figure 1 shows bricolage programming as a creative feedback loop encompassing the written algorithm, its interpretation, and the programmer’s perception and reaction to its output or behaviour. The addition of the algorithmic component in the creative feedback loop makes an additional inner loop explicit between the programmer and their text. At the beginning, the programmer may have a half-formed concept, which only reaches internal consistency through the process of being expressed as an algorithm. The inner loop is where the programmer elaborates upon their imagination of what might be, and the outer where this trajectory is grounded in the pragmatics of what they have actually made. Through this process both algorithm and concept are developed, until the programmer feels they accord with one another or otherwise judges the creative process to be finished.

Representations in the computer and the brain are evidently distinct from one another. Computer output evokes perception, but that percept will both exclude features that are explicit in the output and include features that are not, due to a host of effects including attention, knowledge and illusion. Equally, a human concept is distinct from a computer algorithm. Perhaps a program written in a declarative rather than imperative style is somewhat closer to a concept, being not an algorithm for how to carry out a task, but rather a description of what is to be done. But still, there is a clear line to be drawn between a string of discrete symbols in code and the morass of symbolic, spatial and relational representations we assume underlies cognition.

There is however something curious about how the programmer’s creative process spawns a second, computational one. The computational process is lacking in the cognitive abilities of its author, but is nonetheless both faster and more accurate at certain tasks by several orders of magnitude. It would seem that the programmer uses the programming language and its interpreter as a cognitive resource, augmenting their own abilities in line with the extended mind hypothesis [Clark, 2008]. We will revisit this issue within a formal framework in §5, after first looking more broadly at how we relate programming to human experience, and related issues of representation.

3 Anthropomorphism and Metaphor in Programming

Metaphor permeates our understanding of programming. Perhaps this is due to the abstract nature of computer programs, requiring metaphorical constructs to allow us to ground programming language in everyday reasoning. Petre and Blackwell [1999] gave subjects programming tasks, and asked them to introspect upon their imagination while they worked. These self reports are rich and varied, including exploration of a landscape of solutions, dealing with interacting creatures, transforming a dance of symbols, hearing missing code as auditory buzzing, combinatorial graph operations, munching machines, dynamic mapping and conversation. While we cannot rely on these introspective reports as authoritative on the inner workings of the mind, the diversity of response hints at highly personalised creative processes, related to physical operations in visual or sonic environments. It would seem that a programmer uses metaphorical constructs defined largely by themselves and not by the computer languages they use. However mechanisms for sharing metaphor within a culture do exist. Blackwell [2006a] used corpus linguistic techniques on programming language documentation in order to investigate the conceptual systems of programmers, identifying a number of conceptual metaphors listed in Figure 2. Rather than finding metaphors supporting a mechanical, mathematical or logical approach as you might expect, components were instead described as actors with beliefs and intentions, being social entities acting as proxies for their developers.

The above research suggests that programmers understand the operation of their programs by metaphorical relation to their experience as a human. Indeed the feedback loop described in §2 is by nature anthropomorphic; by embedding the development of an algorithm in a human creative process, the algorithm itself becomes a human expression. Dijkstra [1988] strongly opposed such approaches: “I have now encountered programs wanting things, knowing things, expecting things, believing things, etc., and each time that gave rise to avoidable confusions. The analogy that underlies this personification is so shallow that it is not only misleading but also paralyzing.” Dijkstra’s claim is that by focusing on the operation of algorithms, the programmer submits to a combinatorial explosion of possibilities for how a program might run; not every case can be covered, and so bugs result. Dijkstra argues for a strict, declarative approach to computer science and programming in general, which he views as so radical that we should not associate it with our daily existence, or else limit its development and produce bad software.

The alternative view presented here is that metaphors necessarily structure our understanding of computation. This view is sympathetic to a common assumption in the field of cognitive linguistics, that our concepts are organised in relation to each other and to our bodies, through conceptual systems of metaphor. Software now permeates Western society, and is required to function reliably according to human perception of time and environment. Metaphors of software as human activity are therefore becoming ever more relevant.

Components are agents of action in a causal universe.

Programs operate in historical time.

Program state can be measured in quantitative terms.

Components are members of a society.

Components own and trade data.

Components are subject to legal constraints.

Method calls are speech acts.

Components have communicative intent.

A component has beliefs and intentions.

Components observe and seek information in the execution environment.

Components are subject to moral and aesthetic judgement.

Programs operate in a spatial world with containment and extent.

Execution is a journey in some landscape.

Program logic is a physical structure, with material properties and

subject to decay.

Data is a substance that flows and is stored.

Technical relationships are violent encounters.

Programs can author texts.

Programs can construct displays.

Data is a genetic, metabolizing lifeform with body parts.

Software tasks and behaviour are delegated by automaticity.

Software exists in a cultural/historical context.

Software components are social proxies for their authors.

Figure 2: Conceptual metaphors derived from analysis of Java library documentation by Blackwell [2006a]. Program components are described metaphorically as actors with beliefs and intentions, rather than mechanical imperative or mathematical declarative models.

4 Symbols and Space

We now turn our attention to how the components of the bricolage programming process shown in Figure 1 are represented, in order to ground understanding of how they may interrelate. Building upon the anthropomorphic view taken above, we propose that in bricolage programming, the human cognitive representation of programs centres around perception. Perception is a low dimensional representation of sensory input, giving us a somewhat coherent, spatial view of our environment. By spatial we do not just mean in terms of physical objects, but also in terms of features in the spaces of all possible tastes, sounds, tactile textures and so on. This scene is built through a process of dimensional reduction from tens of thousands of chemo-, photo-, mechano- and thermoreceptor signals. Algorithms on the other hand are represented in discrete symbolic sequences, as is their output, which must go through some form of digital-to-analogue conversion before being presented to our sensory apparatus, for example as light from a monitor screen or sound pressure waves from speakers, a process we call observation. Recall the programmer from §2, who saw something not represented in the algorithm or even in its output, but only in their own perception of the output; observation may itself be a creative act.

The component from Figure 1 not yet mentioned in this section is that of programmers’ concepts. A concept is ‘a mental representation of a class of things’ [Murphy, 2002, p.5]. Figure 1 shows concepts mediating between spatial perception and symbolic algorithms, leading us to ask; are concepts represented more like spatial geometry, like percepts, or symbolic language, like algorithms? Our focus on metaphor leads us to take the former view, that conceptual representation is grounded in perception and the body. This view is taken from Conceptual Metaphor Theory (CMT) introduced by Lakoff and Johnson [1980], which proposes that concepts are primarily structured by metaphorical relations, the majority of which are orientational, understood relative to the human body in space or time. In other words, the conceptual system is grounded in the perceptual system. Gärdenfors [2000] builds upon this by further proposing that the semantic meanings of concepts and the metaphorical relationships between them are geometrical properties and relationships. Concepts themselves are represented by geometric regions of low dimensional spaces defined by quality dimensions, either mapped directly from, or structured by metaphorical relation to perceptual qualities. For example “red” and “blue” are regions in perceptual colour space, and the metaphoric semantics of concepts within the spaces of mood, temperature and importance may be defined relative to geometric relationships of such colours.

Gärdenforsian conceptual spaces are compelling when applied to concepts related to bodily perception, emotion and movement, and Forth et al. [2008] report early success in computational representations of conceptual spaces of musical rhythm and timbre, through reference to research in music perception. However, it is difficult to imagine taking a similar approach to computer programs. What would the quality dimensions of a geometrical space containing all computer programs be? There is no place to begin to answer this question; computer programs are symbolic in nature, and cannot be coherently mapped to a geometrical space grounded in perception.

For clarity we turn once again to Gärdenfors [2000], who points out that spatial representation is not in opposition to symbolic representation; they are distinct but support one another. This is clear in computing, hardware exists in our world of continuous space, but thanks to reliable electronics, conjures up a symbolic world of discrete computation. Our minds are able to do the same, for example by computing calculations in our head, or encoding concepts into phonetic movements of the vocal tract or alphabetic symbols on the page. We can think of ourselves as spatial beings able to simulate a symbolic environment to conduct abstract thought and open channels of communication. On the other hand, a piece of computer software is a symbolic being able to simulate spatial environments, perhaps to create a game world or guide robotic movements, both of which may include some kind of model of human perception.

Computer language operates in the domain of abstraction and communication but in general does not at base include spatial semantics. In some cases computer languages are described as ‘visual’ even when spatial arrangement is purely secondary notation, ignored by the interpreter, such as in Patcher languages [Puckette, 1988]. In fact spatial layout is a feature of secondary notation in mainstream ‘textual’ languages too, through use of whitespace with no syntactical meaning. That programmers need to use spatial layout as a crutch while composing symbolic sequences is telling; to the interpreter, a block may be a subsequence between braces, but to an experienced programmer it is a perceptual gestalt grouped by indentation. From this we can understand computation as separate from spatial reasoning, but supported by it, with secondary notation helping bridge the divide.

An important aspect of CMT is that a conceptual system of semantic meaning exists within an individual, not in the world. Through language, metaphors become established in a culture and shared by its participants, but this is an effect of individual conceptual systems interacting, and not individuals inferring and adopting external truths of the world (or of possible worlds). This would account for the varied range of metaphor in programming discussed in §3, as well as the general failure of attempts at designing metaphor into computer interfaces [Blackwell, 2006b]. Each programmer has a different set of worldly interests and experiences, and so establishes different metaphorical systems to support their programming activities.

5 Components of creativity

We now have sufficient grounds to fully characterise how the creative process operates in our case study of bricolage programming. For this we employ the Creative Systems Framework (CSF), a high-level formalisation of creativity introduced by Wiggins [2006a], Wiggins [2006b] and based upon the work of Boden [2003]. Creativity is characterised as a search in a space of concepts. Three sets of rules are employed in this search; R defining the search space itself, T defining traversal of the space and E defining evaluation of concepts found in the space. However, the CSF describes much more than a reactive process of traversal and evaluation. Creativity also requires introspection, self-modification and for boundaries to be broken. In other words, the rulesets R, T and E are examined and challenged by the creative agent following them.

Using the terms of Gärdenfors [2000], R is a concept defining a space of concept instances.¹ For example in a creative search for music within a genre, the genre would be the concept and a piece of music conforming to a genre would be a instance of that concept. Crucially, R is not a closed space, but rather defined as a subspace of the universe of all possible concepts. This means that a creative agent may creatively push beyond the boundaries of the search as we will see.

We are now in a position to clarify the bricolage programming process introduced in §2.1 within the CSF. As shown in Figure 3, the ruleset R defines the programmer’s concept, being their current artistic focus structured by learnt techniques and conventions, the traversal strategy T is the process of encoding and interpreting the algorithm, and the evaluation function E is the perceptual process of observation and reaction.

Figure 3: The process of action and reaction in bricolage programming from Figure 1, annotated with the R conceptual space, T traversal strategy and E evaluation components of the Creative Systems Framework.

In §2, we alluded to the extended mind hypothesis [Clark, 2008], claiming that bricolage programming takes part of the human creative process outside of the mind and into the computer. The above makes clear what we claim is being externalised: part of the traversal strategy T. The programmer’s concept R motivates a development of the strategy T to be encoded in a program, but the programmer does not necessarily have the cognitive ability to fully evaluate the program. That task is taken on by the interpreter running on a computer system, meaning that T encompasses both encoding by the human and interpretation by the computer.

The traversal strategy T is structured by the techniques and conventions employed to convert concepts into operational algorithms. These may include design patterns, a standardised set of ways of building that have become established around imperative programming languages. Each design pattern identifies a kind of problem, and describes a kind of structure as a kind of solution.²

Figure 4: Application of a traversal strategy T leading outside the concept R, triggering transformational creativity.

The creative process is constrained by R, being the programmer’s idea of what is a valid end result. This is shaped by the programmer’s current artistic focus, being the perceptual qualities they are currently interested in, perhaps congruent with a cultural theme such as a musical genre or artistic movement. Transformational creativity can be triggered in the CSF when application of T results in a concept instance that exists outside the constraining bounds of R, shown in Figure 4. If the instance is valued according to E, then R is changed to include it. If the instance is not valued, then T is changed to avoid that instance in the future. As a result of including external interpretation in T, the programmer is likely to be less successful in writing software that meets their preconceptions, but as a result more successful in being surprised by the results. In other words, the artist’s act of externalising part of T as a computer program makes the results less predictable, and transformational creativity more likely.

In artistic bricolage programming, then, we conclude that creativity is a process of imagining a concept R, encoding an operational algorithm as part of T to explore within and beyond R, and a perceptual process E to evaluate the output. Through this process both R and T are continually transformed in respect of one another, in creative feedback.

According to our embodied view, not only is perception crucial in evaluating output within bricolage programming, but also in structuring the space in which programs are conceptualised. Indeed if the embodied view of CMT holds in general, the same would apply to all creative endeavour. From this we find a message for the field of computational creativity: a prerequisite for an artificial creative agent is in acquiring computational models of perception sufficient to both evaluate its own works and structure its conceptual system. Only then will the agent have a basis for guiding changes to its own conceptual system and generative traversal strategy, able to modify itself to find artifacts that it was not programmed to find, and place value judgements on them. Such an agent would need to adapt to human culture in order to interact with shifting cultural norms, keeping its conceptual system and resultant creative process as coherent within that culture. For now however this is all wishful thinking, and we must be content with generative computer programs which extend human creativity, but are not creative agents in their own right.

6 Programming in Time

“She is not manipulating the machine by turning knobs or pressing buttons. She is writing messages to it by spelling out instructions letter by letter. Her painfully slow typing seems laborious to adults, but she carries on with an absorption that makes it clear that time has lost its meaning for her.” Sherry Turkle [2005, p. 92], on Robin, aged 4, programming a computer.

Having investigated the representation and operation of bricolage programming we now examine how the creative process operates in time. Considering computer programs as operating in time at all, rather than as logic abstract from the world, is itself a form of the anthropomorphism examined in §3. However from the above quotation it seems that Robin stepped out of any notion of physical time, and into the algorithm she was composing, entering a timeless state. Speaking anecdotally, programmers report losing hours as they get ‘in the flow’ when writing software. Perhaps a programmer is thinking in algorithmic time, attending to control flow as it replays over and over in their imagination, and not to the world around them. Or perhaps they are not attending to the passage of time at all, thinking entirely of declarative abstract logic, in a timeless state of building. In either case, it would seem that the human is entering time relationships of their software, rather than the opposite, anthropomorphic direction of software entering human time. However there are ways in which human and computational time may be united, which we will come to shortly.

Temporal relationships are generally not represented in source code. When a programmer needs to do so, for example as an experimental psychologist requiring accurate time measurements, or a musician needing accurate synchronisation between processes, they run into problems. With the wide proliferation of interacting embedded systems, this is becoming a broad concern [Lee, 2009]. In commodity systems time has been decentralised, abstracted away through layers of caching, where exact temporal dependencies and intervals between events are not deemed worthy of general interest. Programmers talk of ‘processing cycles’ as a valuable resource which their processes should conserve, but they generally no longer have programmatic access to the high frequency oscillations of the central processing units (now, frequently plural) in their computer. The allocation of time to processes is organised top-down by an overseeing scheduler, and programmers must work to achieve what timing guarantees are available. All is not lost however, realtime kernels are now available for commodity systems, allowing psychologists [Finney, 2001] and musicians (e.g. via http://jackaudio.org/) to get closer to physical time. Further, the representation of time semantics in programming is undergoing active research in a subfield of computer science known as reactive programming [Elliott, 2009], with applications emerging in music [McLean and Wiggins, 2010].

6.1 Interactive programming

Programmers who ‘think’ in algorithmic time, like Robin earlier, are well served by dynamically interpreted languages. These allow a programmer to examine an algorithm while it is interpreted, taking on live changes without restarts. This is known as interactive programming, and unites the time flow of a program with that of its development. Interactive programming makes a dynamic creative process of test-while-implement possible, rather than the conventional implement-compile-test cycle, so that arrows shown in Figures 1 and 3 show concurrent influences between components rather than time-ordered steps.

Interactive programming not only provides a more efficient creative feedback loop, but also allows a programmer to connect software development with time based art. Since 2003 an active group of practitioners and researchers have been developing new approaches to making computer music and video animation, collectively known as Live coding [Blackwell and Collins, 2005, Ward et al., 2004, Collins et al., 2003, Rohrhuber et al., 2005]. The archetypal live coding performance involves programmers writing code on stage, with their screens projected for an audience, while the code is dynamically interpreted to generate music or video. Here the process of development is the performance, with the work generated not by a finished program, but its journey of development from an empty text editor to complex algorithm, generating continuously changing musical or visual form along the way. This is bricolage programming perhaps taken to a logical and artistic conclusion.

7 Conclusion

What we have seen provides strong motivation for programming which address the concerns of artists. These include concerns of workflow, where any time elapsed between source code edit and interpreted output is slows the creative process. Concerns of interfaces are also important, where in certain situations greater emphasis is placed on presentation of short scripts in their entirety as per bricolage programming, rather than hierarchical views of larger codebases. Perhaps most importantly, we have seen motivated the development of programming languages to greater support artistic expression.

From the embodied view we have taken, it would seem useful to integrate time and space further into programming languages. In practice integrating time can mean on one hand including temporal representations in core language semantics, and on the other uniting development time with execution time, as we have seen with interactive programming. Temporal semantics and interactive programming both already feature strongly in some programming languages for the arts, as we saw in §6, but how about analogous developments in integrating space into the semantics and activity of programming? This is a less well understood area requiring further research, but it would seem that novel approaches to the integration of computational geometry and perceptual models such as computer vision into programming language could serve artists well. By harnessing and extending research into visual programming languages, this could extend to notation, taking for example the ReacTable as inspiration [Jordà et al., 2007].

We began with Paul Klee, a painter whose production was limited by his two hands. The artist-programmer is not so limited, but shares what Klee called his limitation of reception, by the “limitations of the perceiving eye”. This is perhaps a limitation to be expanded but not overcome, rather celebrated and fully explored using all we have, including our new computer languages. We have characterised a bricolage approach to artistic programming as an embodied, creative feedback loop. This places the programmer close to their work, grounding symbolic computation in orientational and temporal metaphors of their human experience. However the computer interpreter extends the programmer’s abilities beyond their own imagination, making unexpected results likely, leading the programmer to new creative possibilities.

References

Alexander et al. [1977]: Christopher Alexander, Sara Ishikawa, and Murray Silverstein. A Pattern Language: Towns, Buildings, Construction. Oxford University Press, first edition, August 1977. ISBN 0195019199.
Blackwell and Collins [2005]: Alan Blackwell and Nick Collins. The programming language as a musical instrument. In Proceedings of PPIG05. University of Sussex, 2005.
Blackwell [2006a]: Alan F. Blackwell. Metaphors we program by: Space, action and society in java. In Proceedings of the Psychology of Programming Interest Group 2006, 2006a.
Blackwell [2006b]: Alan F. Blackwell. The reification of metaphor as a design tool. ACM Trans. Comput.-Hum. Interact., 13 (4): 490–530, December 2006b. ISSN 1073-0516. doi: 10.1145/1188816.1188820.
Boden [2003]: Margaret A. Boden. The Creative Mind: Myths and Mechanisms. Routledge, 2 edition, November 2003. ISBN 0415314534.
Clark [2008]: Andy Clark. Supersizing the Mind: Embodiment, Action, and Cognitive Extension (Philosophy of Mind Series). OUP USA, November 2008. ISBN 0195333217.
Collins et al. [2003]: Nick Collins, Alex McLean, Julian Rohrhuber, and Adrian Ward. Live coding in laptop performance. Organised Sound, 8 (03): 321–330, 2003. doi: 10.1017/S135577180300030X.
Dijkstra [1988]: Edsger W. Dijkstra. On the cruelty of really teaching computing science. 1988.
Elliott [2009]: Conal Elliott. Push-pull functional reactive programming. In Haskell Symposium, 2009.
Finney [2001]: Steven A. Finney. Real-time data collection in linux: A case study. Behavior Research Methods, Instruments, & Computers, 33 (2): 167–173, May 2001.
Forth et al. [2008]: Jamie Forth, Alex McLean, and Geraint Wiggins. Musical creativity on the conceptual level. In IJWCC 2008, 2008.
Gärdenfors [2000]: Peter Gärdenfors. Conceptual Spaces: The Geometry of Thought. The MIT Press, March 2000. ISBN 0262071991.
Jordà et al. [2007]: S. Jordà, G. Geiger, M. Alonso, and M. Kaltenbrunner. The reactable: Exploring the synergy between live music performance and tabletop tangible interfaces. In Proc. Intl. Conf. Tangible and Embedded Interaction (TEI07), 2007.
Klee [1953]: Paul Klee. Pedagogical sketchbook. Faber and Faber, 1953.
Lakoff and Johnson [1980]: George Lakoff and Mark Johnson. Metaphors We Live By. University of Chicago Press, first edition edition, April 1980. ISBN 0226468011.
Lee [2009]: Edward A. Lee. Computing needs time. Commun. ACM, 52 (5): 70–79, 2009. ISSN 0001-0782. doi: 10.1145/1506409.1506426.
McLean and Wiggins [2010]: Alex McLean and Geraint Wiggins. Petrol: Reactive pattern language for improvised music. In Proceedings of the International Computer Music Conference, June 2010.
Murphy [2002]: Gregory L. Murphy. The Big Book of Concepts (Bradford Books). The MIT Press, August 2002. ISBN 0262632993.
Petre and Blackwell [1999]: Marian Petre and Alan F. Blackwell. Mental imagery in program design and visual programming. International Journal of Human-Computer Studies, 51: 7–30, 1999.
Puckette [1988]: M. Puckette. The patcher. In Proceedings of International Computer Music Conference, 1988.
Rohrhuber et al. [2005]: Julian Rohrhuber, Alberto de Campo, and Renate Wieser. Algorithms today: Notes on language design for just in time programming. In Proceedings of the 2005 International Computer Music Conference, 2005.
Turkle [2005]: Sherry Turkle. The Second Self: Computers and the Human Spirit, Twentieth Anniversary Edition. The MIT Press, 20 anv edition, July 2005. ISBN 0262701111.
Turkle and Papert [1990]: Sherry Turkle and Seymour Papert. Epistemological pluralism: Styles and voices within the computer culture. Signs, 16 (1): 128–157, 1990. ISSN 00979740. doi: 10.2307/3174610.
Turkle and Papert [1992]: Sherry Turkle and Seymour Papert. Epistemological pluralism and the revaluation of the concrete. Journal of Mathematical Behavior, 11 (1): 3–33, March 1992.
Ward et al. [2004]: Adrian Ward, Julian Rohrhuber, Fredrik Olofsson, Alex McLean, Dave Griffiths, Nick Collins, and Amy Alexander. Live algorithm programming and a temporary organisation for its promotion. In Olga Goriunova and Alexei Shulgin, editors, read_me — Software Art and Cultures, 2004.
Wiggins [2006a]: G. A. Wiggins. A preliminary framework for description, analysis and comparison of creative systems. Journal of Knowledge Based Systems, 2006a.
Wiggins [2006b]: G. A. Wiggins. Searching for computational creativity. New Generation Computing, 24 (3): 209–222, 2006b.

1: The terms used by Gärdenfors [2000] diverge from those used by Wiggins [2006a], Wiggins [2006b]. Wiggins uses the term conceptual space in the place of Gärdenfors’ concept, and concept in the place of concept instance. The meaning is however the same, particularly when the recursive hierarchy of Wiggins’ theory is taken into account.
2: Interestingly, this structural heuristic approach to problem solving originated in the field of urban design [Alexander et al., 1977].

Flash on eval

2010-06-05T07:24:16Z

Here’s a tech demo of my current Haskell live coding environment, an emacs mode adapted from Rohan Drape’s haskell supercollider mode, with flash on eval adapted from the syntax tail highlighting minor mode.

Source coming soon here…

haskell 2 minute acid tech demo from yaxu on Vimeo.

Visualisation of Live Code

2010-05-29T20:39:49Z

I wrote a paper with Dave Griffiths and Nick Collins on the visualisation of live code, exploring ideas around live coding interfaces, accepted for the EVA London 2010 conference in July. A HTML version is below, or see the PDF Preprint.

Alex McLean (Goldsmiths), Dave Griffiths (FoAM), Nick Collins (University of Sussex) and Geraint Wiggins (Goldsmiths)

Abstract

In this paper we outline the issues surrounding live coding which is projected for an audience, and in this context, approaches to code visualisation. This includes natural language parsing techniques, using geometrical properties of space in language semantics, representation of execution flow in live coding environments, code as visual data and computer games as live coding environments. We will also touch on the unifying perceptual basis behind symbols, graphics, movement and sound.

1. Introduction

Live coding, the improvisation of video and/or music using computer language, has developed into an active field of research and arts practice over the last decade (Wang and Cook; 2004; Ward et al.; 2004; Collins et al.; 2003). Live coding is made possible by dynamic language interpreters, which allow algorithms to run while they are being modified, taking on changes without any break in the audio or visual output generated by the code. The development of software becomes part of the art in a very real sense; at the beginning of a typical live coded performance there is no code and no audiovisual output, but the output grows in complexity with the code.

A frequent criticism of computer music is the lack of performance, where an artist hides behind their laptop screen, and the audience is unable to see any activity that might ground their experience of the music (Cascone; 2003). Solutions continue to be explored, with many researchers focussing on developing tangible interfaces which bring the computer closer to a traditional instrument. However, a live coding tradition has developed taking the straightforward approach of projecting whatever is on the artist’s screen: the code, moving cursors, the debugging output… The audience is then able to see the human movements and code structures behind an improvisation.

This tradition of projecting screens is itself open to criticism; the audience members may feel distracted, or perhaps even excluded by the projection of code written in language they do not necessarily understand. The alternative of showing nothing, hiding behind a laptop screen, is felt to be untenable, but perhaps more should be understood about the practice of projecting code. Watching the articulations of a live guitarist may enhance the experience of a listener who does not play a musical instrument themselves. Can a live coder elucidate the more abstract thinking gestures of their practice? The search is on for ways of visualising code development that allows non-programmers to enhance their enjoyment and understanding of a live coded piece.

2. Perceiving code

Generally, a programmer cannot work with their eyes closed; a programmer’s text editor is a visual interface¹. Text editors have gained many features over the last few decades, to the point where we no longer call them text editors but Interactive Development Environments (IDEs). The visual presentation of code has developed its own aesthetic; colour is used to highlight syntax, fonts have been designed for code (e.g. ProFont, proggy), and visual tools for navigating around tree-like code structures. Nonetheless computation is fundamentally about symbol manipulation, and the composition of symbols lies at the heart of every IDE. When our eyes saccade across code, the shapes on the screen are categorised into these symbols, and we perceive them as the tokens (words) and statements (sentences) making up our program. The computer interprets code as a one dimensional string of discrete symbols, but humans perceive it as symbols within a spatial scene. Expert programmers may be able to chunk larger blocks of code as meaningful entities; less experienced live code audiences may become stuck on small details, but an elaborate dance of spatial change to code is evident over time.

Our perception of source code is aided not only by spatial organisation, but also by colour highlighting, in-line documentation and the well chosen names given to abstractions and data structures. These features are collectively known as secondary syntax², being that ignored by the interpreter but of benefit to programmers in understanding and organising their code. A challenge to those pushing the boundaries of programming language design is to find ways of taking what is normally secondary syntax as primary. For example the ColorForth language uses colour as primary syntax, replacing the need for punctuation. Even more radically, the instruction set of the Piet language illustrated in Figure 2 is formed by first order colour relationships within a two dimensional grid; instructions include directional modifiers so that control flow travels in two dimensions. Piet, among many other esoteric languages, is inspired by the two dimensional syntax of Befunge shown in Fig. 1, a textual language where arrow-like characters change the direction of control flow. Some languages bordering on mainstream, such as Haskell and to a lesser extent Python have a syntax that takes two dimensional arrangement into account when grouping statements, although this is otherwise unusual.

Secondary syntax is of great importance to human understanding, despite being ignored by the computer interpreter. Without spatial layout and elements of natural language a program would be next to unreadable by humans. Humans live an embodied existence in a spatial environment, and while we are perfectly able to perform computation, our spatial ability still supports such thought processes (Gärdenfors; 2000). As a result source code, as Human Computer Interface, is a half-way mixture of geometrical relations and symbolic structures. This is true even of the ‘patcher’ dataflow languages in common use in the digital arts (Puckette; 1988), such as Max and PureData. Patcher languages are often described as ‘visual’, but in fact all the functions are defined textually, and the visual arrangement is purely secondary syntax ³.

Visualisation of code may either act as secondary syntax in order to enhance code comprehension for human viewers, or go further as primary syntax to enhance meaning for both humans and computers. The latter is of particular interest, as to some extent it requires making models of human perception the basis of computer language.

2.1. Morphology of Sound, Shape and Symbols

TurTan is a geometric visual live coding language introduced by Gallardo et al. (2008), using the technology of the Reactable (Jordà et al.; 2007). The functions of the language are manipulated as physical blocks that are placed on a tabletop interface, with nearest neighbours forming a sequence, and relative angle mapping to the function’s parameter. The functions describe turtle graphics operations, and the resulting recursive forms are continuously updated on the table surface display.

TurTan inspired a system by Alex McLean and introduced here, with the working title of Acid Sketching. In Acid Sketching, a sound is specified simply by drawing a shape, where morphological measurements are mapped to parameters of an acid bassline synthesiser. The area of a shape is mapped to pitch, its regularity (perimeter length vs area) mapped to envelope modulation, and relative angle of central axis mapped to resonance. Several such shapes are drawn in an arrangement, where a minimum spanning tree of their centroids is taken as a polyphonic sequence, where distance equals relative time. Feedback may be projected back on to the drawing surface, so shapes flash red as they are triggered. A static figure would not make this clearer, however illustrative video is available online at http://yaxu.org/acid-sketching/.

While Acid Sketching and TurTan are far from what is typically understood as live coding, both lead us to challenge understanding of the role of symbols, shape and geometry in computation. Investigating how such concrete forms of interaction could be married with the abstractions of general, Turing complete programming languages could be an interesting research topic itself.

Critically connected to live coding engagement with time-based media, is the time-based revelation of code itself. For electroacoustic music, Pierre Schaeffer’s theories of sound timbre have been further dynamised into the time-variant sonic gestures of Denis Smalley’s spectromorphology (Landy; 2007). For live coding, we might analogously dub ’codeomorphology’ as the changing shape of code over time. Examples include the accumulating code revisions referenced on the edge of ChucK language Audicle documents, or SuperCollider’s ‘History’ class to document a live code performance. More visual representations of change over time would include accessible visualisations of programmer activity. Metrics might be displayed to characterise changes per second, from coarse keystroke counts to the depth of parse tree disruption; this brings us to self-evaluating performances, and coder re-coding of their very visualisations…

3. Visual experiments in live code

This section serves to introduce four novel visual/geometric live coding systems by Dave Griffiths, namely Scheme Bricks, Betablocker, Al-Jazari and Daisy Chain, along with some of the systems which inspired them. All of these languages were constructed within Fluxus, a game engine designed for live coding performances and experiments and available under a free (GPL) license from http://www.pawfal.org/fluxus/.

3.1. Execution flow and operational events

vv  <      <
    2
    ^  v<
 v1<?>3v4
    ^   ^
>  >?>  ?>5^
    v   v
 v9<?>7v6
    v  v<
    8
 .  >  >   ^
^<

Figure 1: A pseudo-random number generator written in the two-dimensional language Befunge.

Figure 2: Source code written in the Piet language with two dimensional, colour syntax. Prints out the text “Hello, world!”. Image © Thomas Schoch 2006. Used under the Creative Commons BY-SA 2.5 license.

Figure 3: Core war runtime display, showing visualisation of process memory shared between the players

Figure 4: A live edit in the Betablocker environment, selecting an instruction from a wheel of possibilities.

Figure 5: The robots of Al-Jazari, each with a thought bubble containing a program, live coded with a gamepad.

Figure 6: SchemeBricks, a lisp environment using colour instead of parenthesis, and flashes as a cue for control flow.

Figure 7: A section of a Daisy Chain program.

Computation is a metaphorical movement, where algorithmic processes operate on data (which can include the algorithms themselves) in memory in the discrete time steps of the CPU. Ways of visualising memory as it is changed have been developed for conventional debuggers, particularly in microcontroller applications where memory is small enough to be viewed in its entirety. More novel visualisations also exist, such as Tierra, an artificial life simulation where code evolves in a Darwinian competition which can only be appreciated when viewed as such, or Core War (Fig. 3), a game where player/programmers write code which fight over memory address space.

Live coding has the unique opportunity to visualise the movement of an underlying process while it is being formed. This helps an audience appreciate a live coding performance in a more meaningful way – as it bridges the gap between an abstract description of a process (the code) and process itself (the generated pattern of movement through memory). Betablocker (Fig. 4) is a raw visualisation of an imaginary 8-bit processor operating in 256 bytes of memory. This brightly coloured live coding environment is operated by writing assembly code with a gamepad. The processes are visualised while they operate on the memory addresses and trigger sound events. Processes are able to modify themselves and each other, resulting in highly dynamic relationships which are challenging to control.

A more traditional method of programming is employed in Scheme Bricks (Fig. 6), a geometric interface for constructing Scheme programs. Scheme Bricks takes advantage of the isomorphism of code and data in the Scheme programming language, and is inspired by the Scratch language designed for use by children (Resnick et al.; 2009). Scheme Bricks allows you to drag, drop and plug together programs rather than typing. This has some potential side effects; in a performance situation, it is impossible to have a mismatched parenthesis error, as is common in other lisp-like languages. It is quicker to change the overall structure of the program as sections can be removed and reinserted easily by drag/drop actions. Unwanted sections are pulled out of the program and set aside rather than being deleted, and accumulate around the program as ‘spare parts’ which are often later ‘recycled’ by being pulled back into another section.

Scheme Bricks uses visual feedback to relate sound events to the code; the instruction which triggered a sound event flashes as the sound is played. This minimal approach to process visualisation makes the relationship between sound and code structure clearer than Betablocker’s more complete visualisation, and is useful for the performer to immediately locate the code generating a particular sound event.

Daisy Chain (Fig. 7) is an attempt to embrace less rigid structures while maintaining enough of a computational basis to qualify as a live coding performance. It follows a processing system based on Petri nets (Petri; 1966), where executable instruction tokens move around a directed graph. Daisy Chain programs create and modify the graph topologies that they inhabit, producing sounds as a side effect of the computation. The look of the performance was designed to be as far from conventional programming as possible, hand animated flowers and drawn instruction symbols moving around graphs constrained by spring models.

The nodes of a Daisy Chain graph have a fixed lifetime, which was introduced in order to counter a common problem with live coding where the audience watching and performer concentrating on programming tend to perceive time differently. Daisy Chain prevents musical structures from persisting too long, keeping the performance moving forward at a rate the performer can control beforehand.

3.2. Computation in game worlds

Code has a long tradition of use in games as a gameplay mechanic, an early example being Core War developed in the mid 1980s and discussed above in §3.1. More recent games such as Carnage Heart and Marionette Handler are mainstream games for the Playstation which employ programming environments using icons. These programs are used to control robots which battle it out in large virtual arenas. Popular game titles such as Little Big Planet allow the player to construct machines as part of game worlds, complex enough to support Turing complete computation. Kodu, a research project at Microsoft goes even further, as an end-user games programming environment on the XBox.

Al-Jazari is a deliberate attempt to fuse games and live coding performances. It was designed to use a similar visual process to BetaBlocker, but this time mediated through the actions of robotic agents moving around a 3D world, triggering sounds as they do so (Fig. 5). The use of visual agents following commands rather than abstract processes is intended to make the performance more immediately understandable for the audience. Al Jazari has been expanded as an art installation, audience participatory performance and recently as a facebook game – with the aim to increase the accessibility of live coding to the point where anyone can become a live coder.

4. Conclusion

Visualisation is central to live coding. In this article, we have confronted how code is perceived by performers and audiences, and in what ways visual elements contribute to the primary syntax and semantics of a programming language meant for live coding. Consideration of visual elements of code have also become essential as live coding has formed the basis of virtual game worlds. We have introduced a number of novel systems, presented here as explorations of these themes. Visualisation of live code however remains under-investigated in terms of the psychology of programming; while Blackwell and Collins (2005) lead the way into HCI, evaluation protocols are yet to be adapted and applied to experience of live coded performances. This is however fertile ground for practice based research, and we anticipate the changing shapes of code over time, a codeomorphology at timescales from individual performances to lifetimes of artistic and technological development.

Bibliography

[1]

Blackwell, A. and Collins, N. (2005). The programming language as a musical instrument. In Proceedings of PPIG05. University of Sussex.

[2]

Cascone, K. (2003). Grain, sequence, system (three levels of reception in the performance of laptop music). In Kleiner, M. S. and Szepanski, A., editors, Soundcultures. Suhrkamp.

[3]

Collins, N., McLean, A., Rohrhuber, J., and Ward, A. (2003). Live coding in laptop performance. Organised Sound, 8(03):321–330.

[4]

Gallardo, D., Julià, C. F., and Jordà, S. (2008). Turtan: a tangible programming language for creative exploration. In Third annual IEEE international workshop on horizontal human-computer systems (TABLETOP).

[5]

Gärdenfors, P. (2000). Conceptual Spaces: The Geometry of Thought. The MIT Press.

[6]

Jordà, S., Geiger, G., Alonso, M., and Kaltenbrunner, M. (2007). The reactable: Exploring the synergy between live music performance and tabletop tangible interfaces. In Proc. Intl. Conf. Tangible and Embedded Interaction (TEI07).

[7]

Landy, L. (2007). Understanding the Art of Sound Organization. The MIT Press.

[8]

Petri, C. A. (1966). Communication with automata. Technical report, Applied Data Research Inc.

[9]

Puckette, M. (1988). The patcher. In Proceedings of International Computer Music Conference.

[10]

Resnick, M., Maloney, J., Hernández, A. M., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., and Kafai, Y. (2009). Scratch: programming for all. Commun. ACM, 52(11):60–67.

[11]

Wang, G. and Cook, P. R. (2004). On-the-fly programming: using code as an expressive musical instrument. In NIME ’04: Proceedings of the 2004 conference on New interfaces for musical expression, pages 138–143, Singapore, Singapore. National University of Singapore.

[12]

Ward, A., Rohrhuber, J., Olofsson, F., McLean, A., Griffiths, D., Collins, N., and Alexander, A. (2004). Live algorithm programming and a temporary organisation for its promotion. In Goriunova, O. and Shulgin, A., editors, read_me — Software Art and Cultures.

Footnotes

A counter-example would be programming interfaces for the blind, which employ speech synthesis.
The term ‘secondary syntax’ is problematic. Firstly, secondary syntax is only secondary relative to the computer interpreter, and not the human. Secondly, secondary syntax is not syntax in any clear sense; indeed spatial relationships are the basis of semantic meaning as understood in the field of cognitive linguistics. However as secondary syntax is the standard term used in the field of Human Computer Interaction (HCI) we persist with using it here.
In Max, left-right position alters execution order, although relying upon this is discouraged in favour of the ‘trigger’ object.

Formatting LaTeX for on-screen proof reading

2010-04-28T08:27:35Z

A few people on twitter found this useful so here it is in full:

I’ve been writing a few papers lately and going through the cycle of write -> print -> proofread -> write, generating a lot of paper. I’ve text to hard to read on screen, and raw LaTeX somehow feels too malleable to read as a document. Then I thought the obvious; why not format the document for the screen.

I came up with this, two columns with minimal margins:

%\documentclass[compact,twocolumn]{article}

%\usepackage[top=0.1in, bottom=0.1in, left=0.3in, right=0.3in, paperwidth=11in, paperheight=7in]{geometry}

%\setlength{\columnsep}{30pt}

This fits nicely on my laptop screen but adjust for your particular aspect ratio and so on. Then view in full screen or presentation mode and hey presto. Evince in linux is great in presentation mode (rather than full screen mode, which keeps a menu bar), and automatically picks up changes when you recompile your PDF.

If you still find yourself with eye strain, rather than reading from paper, consider adjusting the position of your monitor. There’s a lot of hype around e-paper, and it does look lovely, but I’m unsure that the evidence shows that LCDs are significantly worse for eye strain if at all… Is light really that different when it reflects off something rather than emitting from something? I reckon distance between eyes and monitor is a bigger factor at least.

Upcoming events

2010-04-26T19:51:34Z

A few things coming up

1st May 2010 – Slub VJing with Kirk Degiorgio at Lambda Festival, Antwerp
2nd May 2010 – Slub Live at Lambda Festival again
13th May 2010 – Participating on a Cenatus Panel Session on the Future of Music at FutureEverything, Manchester
3rd September 2010 – Live coding at FACT Liverpool (TBC)

Plus we’re doing a live coding tour of the North of England towards the end of the year.

Also coming up, next month’s dorkbotlondon, probably at a venue near Kings Cross. Dorkbot will likely have some involvement with the Big Chill this year too. Been thinking about doing a another placard or pubcode too, and the date of the annual dorkcamp should be confirmed soon…

The Joy of Interpretation

2010-04-26T11:43:24Z

Without interpreters, we wouldn’t have software, but yet interpreters are also software. This is why we talk about `bootstrapping’, where software pulls itself from the floor by its bootstraps, a paradox settled by the existence of hardware microcode.

Any piece of software exists as a combination of two parts, some instructions in a computer language and an interpreter of that language. Alone they do nothing, put them together and they can notionally do anything. Often there are intermediary steps, commonly interpretation into another language called `bytecode’ to produce `binaries’, but these are just translations into another language, which still needs interpreting for the magic to happen.

Interpreters are fantastic, they allow us to try out ideas beyond our imaginations, adding some instructions, interpreting them to get output rendered as sound or light to our senses, perceiving otherwise impossible worlds, and returning to the source code to twist the encoded structures into new contortions inspired by the results so far. We humans expand the realms of perception through computation, not creating things but writing about things in order to magic them into existence. We’re only scratching the surface of what’s possible, artistic and otherwise, from marrying high speed computation with embodied human experience.

It’s a shame then that the freedom of thought that interpreters give us happen to threaten business models of large companies, who are accordingly searching for the power to make free access to them them illegal. `Console’ computers (a misnomer if I ever saw one) are those where the end user is not allowed access to an interpreter, without paying for a restrictive license and/or expensive hardware. You are not allowed to modify code, certainly not allowed to modify the interpreter, and so must be satisfied with using whatever programs the manufacturer allows you to.

What is frightening is that these business models are spreading — from computer games, to handheld computers and now to tablets. The iPhone (and now iPad) was a particular shock as a device from a company producing hardware traditionally marketed at the creative. You certainly can be creative with an iPad, but only using a surface level interface; you can touch the screen to make a mark, but you can’t write a program about touch, or about marks. iScratch, a project that allowed children to interpret their programs on the iPhone, has been rejected from the apple store. Such interpreters have been banned from the iPhone from the start (apart from the concession of a javascript web browser VM without access to, for example, sound synthesis), but a big media stir has only been made since interpretation was locked down on the development side too, having impacts on some crappy tools by a large company.

This creep towards centralised control over interpretation is deeply worrying, and arguments that end-users get confused or threatened by higher order thought are frankly sinister. Can we have our languages back, please?

UPDATE: If you needed convincing on the point of interpreters allowing higher order creativity, check out Dave’s latest work: http://www.pawfal.org/dave/blog/2010/04/lazybotz/

Bricolage programming and patterns

2010-03-25T14:52:29Z

It’s Ada Lovelace day today, and it turns out my main influences lately have been two women.

One is Sherry Turkle, via her book The Second Self: Computers and the Human Spirit, and her work on Epistemological Pluralism and the Revaluation of the Concrete together with Seymour Papert. Two decades on, some aspects of this work are dated, but the main drive reads like a message from the future; to fully accept computer programming in our creative lives we should consider the feminine, do away with purity and black boxes and work within our code rather than upon it. Computer programming languages and culture is biased towards abstract approaches and against anthropomorphism.

Perhaps live coding is a swing back towards a more feminine approach, where humans are immersed in the same time flow as algorithms, fully engage their perceptual and aesthetic faculties while writing code, and have their whole program before them ready to be manipulated, rather than abstracting parts in black boxes according to grand designs.

Here’s a choice quote from the epistemological pluralism paper:

The conventional route into formal systems, through the manipulation of abstract symbols, closes doors that the computer can open. The computer, with its graphics, its sounds, its text and animation, can provide a port of entry for people whose chief ways of relating to the world are through movement, intuition, visual impression, the power of words and associations. And it can provide a privileged point of entry for people whose mode of approach is through a close, bodily identification with the world of ideas or those who appropriate through anthropomorphization. The computational object, on the border between the idea and a physical object, offers new possibilities.

My other recent influence is Laurie Spiegel, in particular her writing on the manipulation of musical patterns. I’d already found myself making a pattern language when I found her paper, and found it a great inspiration. In this 1981 paper she recognises computation as pattern transformation (which after all is all that it is, a Jacquard loom head that transforms), and applies this insight directly to music; computation then becomes music pattern transformation, the structure of computation forming the structure of music. There’s much more inspiration for musician-programmers to be gained from her other writings.

Pure dyne

2010-01-20T09:39:02Z

I’ve been through a few linux distros over the years, neatly getting progressively easier to install and configure as I get less willing to spend time recompiling kernels, culminating in ubuntu, enjoying the attention to detail and simplicity of use. Recently though, I’ve had to give ubuntu up and go back upstream to the rather higher maintenance Debian again. Linux suffers from creeping featurism in its layers of audio APIs, it started with OSS, a straightforward API based on files, then came ALSA, a wildly complex API with broken documentation in a wiki you can’t edit, and an architecture that somehow means only one OSS application can write sound at a time. It seems to me that it’s a failing of ALSA that further layers of abstraction are piled on top of it, creating a rather complex landscape for sound hackers to navigate.

Ubuntu has joined in the fun by shipping with PulseAudio, which is probably great for general users but a pain for those needing to work with audio on a low level without using loads of CPU. Pulse is not straightforward to remove, and when I removed it had problems with volume controls not working, and the likelihood that future system upgrades wouldn’t work so well. That’s why I switched to debian sidux, but then I couldn’t get laptop hibernation, or my firewire sound card working, and had the stress of maintaining an unstable distribution.

However this week Puredyne carrot and coriander came out, and it’s really great. The kernel is optimised for realtime sound, and jack audio runs solidly without any drop outs, something I haven’t seen before. My firewire sound works reliably, better than I managed under ubuntu. It has a really nice logo and clean look, with no plump penguins in sight. It comes with all the best a/v software beautifully packaged, including all the live coding languages. The people behind it are super friendly and helpful. It’s downstream from ubuntu, so all the software is available. It’s a dream!

They make a big deal out of it being good for booting off a USB key, and I think have worked out some nice practicalities of working that way. This makes it great for doing workshops and running linux in a non-linux lab etc. It installs and works just as nicely on a permanent hard drive though, and that’s what I’ve done.

Anyway, heartily recommended, a dream come true, congratulations to all those involved.

2000 to 2009

2009-12-31T14:16:15Z

Inspired by Christof, here’s my roundup of 2000 to 2009, seriously inhibited by my terrible memory. Will add to this as I remember events.

2000 – Discovered generative music and formed slub with ade, with the aim of making people dance to our code, generating music live according to rigorous conceptual ideals. Most of what I’ve done since has revolved around and spun out of this collaboration. Worked as a Perl hacker with the afore-mentioned Christof during the first Internet boom for mediaconsult/guideguide, a fun time hacking code around the clock in a beautiful office with a concrete floor and curvy walls.

2001 – slub succeeded in getting people to dance to our code, at sonic acts at the paradiso in Amsterdam. It was around this time that I left guideguide for state51 to work on a digital platform for the independent music industry – they were very much ahead of their time then and still are now. Got a paper accepted for a conference as an independent researcher, and met Nick Collins for the first time there, another fine inspiration. Co-founded dorkbotlondon, co-organising over 60 events so far…

2002 – Some really fun slub gigs this year. Followed in Ade’s footsteps by winning the Transmediale software art award for a slightly odd forkbomb, which later appeared in an exhibition curated by Geoff Cox alongside work by great artists including Ade, Sol Lewitt, Yoko Ono and some monkeys. Met Jess.

2003 – Programmed the runme.org software art repository, together with Alexei Shulgin, Olga Goriunova and Amy Alexander. Co-organised the first london placard headphone festival; did a few more after, but didn’t yet match the amazing atmosphere of the first.

2004 – Co-founded TOPLAP together with many amazing people, to discuss and promote the idea of writing software live while it makes music or video. Wrote feedback.pl, my own live coding system in Perl. Bought a house with Jess.

2005 – Started studying part time, doing a MSc Arts Computing at Goldsmiths, with help and supervision of Geraint Wiggins. Dave Griffiths, another huge inspiration, officially joined slub for a gig at Sonar.

2006 – Fiddled around with visualisations of sound including woven sound and voronoi diagrams. Learned Haskell. Co-organised the first dorkcamp, which was featured on french tv.

2007 – Got interested in timbre and the voice, came up with the idea vocable synthesis. Helped organise LOSS livecode festival with Access Space in Sheffield. Went on a camping holiday in Wales and got married to a rather pregnant Jess. Had a baby boy called Harvey a few months after. Got my MSc and carried on with a full time PhD in Arts and Computational Technology, supervised again by Geraint.

2008 – Got interested in physical modeling synthesis, using it to implement my vocable synthesis idea. Got interested in rhythm spaces too, through a great collaboration with Jamie Forth and Geraint. Knitted my mum a pair of socks.

2009 – A bit too close, and in part painful, to summarise. Also, it’s not over yet.

Metaphors of javadoc

2009-12-11T14:26:58Z

Conceptual metaphor theory holds that our understanding of the world is largely structured by metaphor. This presumably includes our understanding of computer programs, which is the basis of Metaphors we Program By: Space, Action and Society in Java, a paper by Alan Blackwell. The paper shows analyses of documentation for some standard Java libraries, looking for the metaphors that structure human understanding of the library components and their interactions. I’ve taken the liberty of extracting the metaphors related in the paper but if you’re interested you should go and read the whole thing, it’s a good one. I feel I could meditate on this list for some time, and I’d love to see comparisons with the metaphors used in the documentation of other languages.

Components are agents of action in a causal universe.
Programs operate in historical time.
Program state can be measured in quantitative terms.
Components are members of a society.
Components own and trade data.
Components are subject to legal constraints.
Method calls are speech acts.
Components have communicative intent.
A component has beliefs and intentions.
Components observe and seek information in the execution environment.
Components are subject to moral and aesthetic judgment.
Programs operate in a spatial world with containment and extent.
Execution is a journey in some landscape.
Program logic is a physical structure, with material properties and subject to decay.
Data is a substance that flows and is stored.
Technical relationships are violent encounters.
Programs can author texts.
Programs can construct displays.
Data is a genetic, metabolizing lifeform with body parts.
Software tasks and behaviour are delegated by automaticity.
Software exists in a cultural/historical context.
Software components are social proxies for their authors.

I’ve written about Alan Blackwell’s research before.

Then this morning I saw this tweet:

<laputean> “hello world” programm is the ‘perfect’ starting point of fastidious and wrong epistemological assumptions that one carries for all life

A neat reminder that the ways in which we perceive the workings of computer `agents’ and source code is very much within a particular social context.