• Machine Listening presents Environments 12 (Joel Stern, James Parker & Sean Dockray): An multichannel reimagining of the classic Environments field-recording series, where human narrators, AI-cloned voices and planetary-scale loudspeakers converge to conjure speculative ecologies, reef lullabies and future-ruined sound worlds.
• James Parker: Delivers a lecture entitled The Planetization of Machine Listening
• Kate Carr: Drawing on her album A Field Guide to Phantasmic Birds, Carr’s work traverses the boundary between naturalistic field recordings and algorithmic fabrication, inviting listeners to question what is real, what is synthesised, and how we encounter “more-than-human” sound.
• Amanda Butterworth & Matt Spendlove: Co-founders of Deep Assignments, will present their new collaborative works of sonic architecture, in which emergent algorithmic processes carve a 3D aural space of phasing minimalism.
• Q&A hosted by Angela McArthur (UCL): In a relaxed salon format, McArthur will guide an open discussion in which audience questions, critical perspectives and peer-to-peer exchange become integral to the evening’s unfolding.

What to Expect

• Immersive Sonic Performances: Multichannel presentations that transform sound into a spatial journey, weaving new sonic architectures.
• Speculative Eco-Sonic Narratives: AI-cloned narrators, glitching field recordings and digital artefacts converge to conjure phantom ecologies and future-ruined sound worlds.
• Open Discussion: A relaxed salon format where the audience’s questions and perspectives become an active part of the event.
• Community: Connect with artists, musicians, technologists and fellow listeners in a supportive, DIY setting dedicated to interdisciplinary listening, thinking and making.

Deep Assignments is a London-based collective founded by artists Amanda Butterworth and Matt Spendlove to foster interdisciplinary listening, thinking and making. Through performances, screenings, happenings, workshops and salons, Deep Assignments cultivates a space for creative discussion, collective support and critical exchange - “a space to honour our assignments, independent of established institutions.” In an age of accelerating technological change and growing social fragmentation, Deep Assignments emphasises the creative endeavour as an existential task, affirming its intrinsic value beyond capital.

Creative use of spatial audio is integral in delivering high-quality immersive experiences and growth for the creative industries. Interoperability, skills deficits and technical complexity between Digital Audio Workstations (DAW) and Game Engines such as Unreal Engine 5 (UE5) currently causes friction.

The industry standard Audio Definition Model (ADM) format is used by market leaders in immersive audio to describe and deliver spatial audio mixes. We’ll build upon existing industry work to research and develop workflows and software to provide interoperability across tools. We’ll create interactive virtual experiences to demonstrate the concepts alongside pre-visualisations of real world experiences.

Composers and sound designers will be able to prototype faster, communicating their ideas in real time through bi-directional editing and transfer. A simplified production pipeline will remove technical barriers, lower costs and reduce the number of specialist skills needed to implement audio assets into 3D virtual environments.

Follow along here for further updates and please get in touch directly.

Our interests continued to align in recent years, building on 3d sound mixing and immersive audiovisual environments for installation and performance to explore XR applications and Unreal Engine/Unity for artistic creation and previsualisation of IRL experiences.

We’re extremely pleased to have cemented the relationship more formally and have been sharing the C&R studio at Somerset House since the start of 2024. We have lots of exciting creative ideas to research and develop for our own projects, clients and the wider community. We have more to announce on that soon so follow along here for further updates and please get in touch directly if you prefer.

Big thanks to Tom Slater for being so generous with his time, knowledge and the studio equipment. Really looking forward to what we’ll create together!

One of the challenges we encountered during the CIF2021 R&D whilst working on the Situationist’s Walkman was getting Apple’s PHASE spatial audio engine functioning as part of our project. We’ve written elsewhere about the more general technical challenges involved in the project, but we felt that it’d be worth writing in more detail about this specific aspect, not least because one of the principal challenges we faced was the lack of written introductory documentation (or open source examples - we think we might be the first!) anywhere else online.

Therefore, I’m going to take you step-by-step through everything we did to get up and running with PHASE - first as a little spike to evaluate its suitability for our project, and later to more fully integrate it into the AR experience we developed. This article is fairly code-heavy and definitely aimed towards developers or anyone else working on the technical implementation of an experience using PHASE, but there’s also information that might prove useful for folks who are more focused on audio too, so I’ll do my best to make sure it’s followable without needing to read or understand the code excerpts. Our process (and the code) very closely follows the process outlined in the WWDC21 presentation video introducing the new API, so if you’ve already worked through that you might find a lot of this familiar. However, as well as thinking it’d be useful to have a written tutorial and working code examples up online, we also encountered a few gotchas that weren’t (to us) obvious from the video, so hopefully there’ll be useful information here even if you have already seen the video. Finally, for folks specifically interested in using PHASE in their AR applications and experiences, we’ll provide details on the particular integration points you’ll need to pay attention to in that particular use case, as well as a little bonus on integrating the head-tracking functionality available in the 3rd generation, Max, and Pro versions of the Apple Airpods earphones.

Firstly, a brief introduction to PHASE. PHASE stands for Physical Audio Spatialization Engine, and was developed to provide immersive sound environments for games, applications, and xR experiences. This (other) video provides a decent non-technical overview of what that entails. The key words here are SPATIALization and PHYSICAL: Spatial audio refers to the ability to place and move sounds in a virtual environment, as well as to move the listener within this virtual environment, with sounds appearing to emanate from their positions relative to the listener (this doesn’t require a special surround-sound speaker setup - just a pair of normal headphones, via the magic of Binaural reproduction and HRTFs), and _physical _refers to the ability to model physical properties of that environment such as reverberation and occlusion. The particularly exciting thing about PHASE, from a creative perspective, is that it allows us to use these tools interactively - to produce spatial audio experiences that change over time, and adapt procedurally or in response to user interaction, all within a common consumer technology platform. However, working with these tools requires both familiarity with the APIs available and their underlying concepts, as well as approaches to mixing and producing audio that are not necessarily obvious from a more traditional audio background. We’ll attempt to summarise all these aspects below.

PHASE is pretty standalone, and isn’t coupled to Realitykit / Scenekit or any other framework for UI, AR, VR or gameplay. As such, while getting started with it is a little complex, the general process shouldn’t vary too much depending on the context in which you use it. The code I’m going to show you is taken from one of our early audio AR experiments - see this revision of this file to see it in-situ. It’s an iOS app written in Swift using UIKit, ARKit and Scenekit, so you might have to adjust a few things depending on your own circumstances, but once you’ve finished reading it should be obvious what needs to be changed. To get started, you will need an instance of a PHASEEngine class - this represents the entirety of your sonic environment and handles all the lifecycle, coordination and DSP required - you will very probably only need one of these in your entire app. In our case, we had a single UIKit UIViewController which handled our app’s AR View, and was the only view which would be making sound, so it made sense to make the PHASEEngine an instance variable on that controller, and instantiate it in the onViewDidLoad method:

The first thing you’ll notice is that this engine is instantiated with an updateMode argument. This refers to the strategy PHASE uses to make updates to its internal state in response to changes we make to its configuration (such as the positions of sound emitting objects or the listener). We’ve elected to delegate the responsibility of timing these updates to the PHASE engine itself, which will schedule them automatically, but in cases where performance / latency is super critical, you can choose to handle these manually, we won’t go into the details of that, mostly because we haven’t tried it ourselves!

The phaseEngine also needs to be explicitly started, at which point, in theory, it would start outputting audio, had we actually set up an audio environment for it to output. We’ll get onto that in a second, but first one more bit of housekeeping:

…when we’re done with PHASE, we need to explicitly stop it, which will stop audio and tear down its state. Since the audio environment in our app only exists within the context of this View, we do that in the viewWillDisappear callback.

One useful thing to know (and of particular interest to those from more of an audio background, is that the PHASEEngine can be configured with a Reverb preset, you can choose from several included for different types of environment:

The important thing to note here is that this is set for the global PHASEEngine object, so it applies across your entire environment, you can’t therefore have different reverberation qualities in different areas of your simulation, or applied to different sound sources. Happily, as we’ll see later, you do at least have the ability to set the reverb send level per-source, so if you need different sound sources to sound more or less wet or dry, this can be achieved no problem.

Now, we can get on with creating our sound environment, and the first thing we will need within it is a listener, using the class PHASEListener. This might seem slightly superfluous coming from a traditional audio background based on mixing sources in stereo, where the listener can be safely assumed to be roughly between the two speakers. In an audio environment such as we’re going to define with PHASE, the listener can be positioned anywhere, facing in any direction, and can move! Therefore, we need to model their position within the environment in order that the PHASE engine can render a mix which corresponds to what they would be hearing at that point:

We also stash this listener object away in an instance variable, as we’re going to need to access it again when we come to update the listener position later, and add it as a child of the rootObject of the engine. This gives us an important clue about how PHASE’s environments are structured - they’re basically a hierarchical tree of objects, which can perceive sound (like listeners), emit sound (sources), or interfere with it (like occluders, which we won’t cover here).

We’re now ready to start adding sound sources. In our case all our assets were pre-rendered as mp3s and included as assets in our application bundle. We’ll now show you how to load an asset and add it to the environment, as well as giving you an overview of the types of control you have over how each sound behaves within the environment. This isn’t quite as simple as it sounds though - there’s a lot of different levers and options that can be configured, and a fairly complex graph of objects that need to be plumbed together. In our own app we ended up writing a couple of classes that abstract over a lot of the PHASE internals, in order to be able to compose our app of simpler, configurable sound sources in a way that is hopefully a little easier to reason about. For now though, I’ll go through everything step by step, first we’ll need a reference to the sound asset in our app bundle which we want to add as a source:

This needs to be registered with the phase engine’s assetRegistry, which manages all the objects used by PHASE:

The most important things we learned here is that the urlmust not be nil, and must point to an existing audio asset that PHASE can read, and that the identifiermust be non-nil and unique. These might sound obvious, but PHASE crashes in a profoundly cryptic and non-obvious way if any of these things are untrue, which caused us several hours of head-scratching. Therefore a sensible first step, if faced with any mysterious crashes, is to double check that you’re registering all your assets properly.

We’ll gloss over the other options here, they’re well explained by the docs, apart from to say that the above is probably a useful starting point for working with mono or stereo sources. To work with multichannel audio, you’ll need to pass a channelLayout explicitly to tell PHASE how to interpret it, but for mono and stereo this can be derived automatically, hence passing nil.

We now have to instruct PHASE how to position and play the audio file we’ve registered. Recall that a PHASE environment is a hierarchical tree of objects representing the entities in that environment, so we have to add a node to this tree which handles playback of our sound asset. The type of node which plays an audio file is a Sampler node, and therefore to create one, we need to instantiate a PHASESamplerNodeDefinition, which defines that node. However, this definition depends on quite a few other objects, the role of which isn’t necessarily obvious until you’ve wired them all together. For that reason, I’m going to work through the following process backwards, starting with the sampler node definition, and working back through its dependencies, explaining them as I go. Obviously this doesn’t make this post particularly cut-and-paste-able, but hopefully will make the purpose of the code clearer. If you want to just grab the code and get on with it, that’s absolutely fine, but remember to paste the following blocks in reverse order :-)

So, without further ado, the final step in the chain is to create your sampler node definition add it to the assetRegistry (as we did with our asset itself), and start it playing:

(I’ve commented the objects here which are for the moment undefined, we’ll get to them in a second)

There’s a fair bit to discuss above, so I’ll start with a broad outline of what’s going on, and a couple of pitfalls we encountered while writing it, then move onto the specific configuration we’re providing and the options it affords. A common pattern in PHASE is that we don’t deal with the objects of its internal representation directly (for example Mixers, Nodes, Assets), but instead create definitions for each of them which are registered as assets, and then referenced by an identifier (allowing PHASE itself to handle the creation and destruction of the objects itself as needed). We see this above, instead of instantiating a Sampler node directly, instead we create a PHASESamplerNodeDefinition and register it with the assetRegistry, before creating a PHASESoundEvent (you can think of this like a cue, an instruction for something to happen within the audio environment) referring to that definition, and firing it by calling its start method.

The principle gotcha here, as before, is that the assetName of your sampler node definition must be non-nil and unique, and crucially, it exists in the same namespace as your sound assets and any other object you register with the assetRegistry, so it must be globally unique. As before, failing to do this leads to some very cryptic errors, so it’s a good thing to check first if you’re having problems.

There’s a couple of options here that give us useful fine-grained control over the behaviour of our audio environment, and we’ll run through those now. The sampler node definition’s playbackMode property defines whether a sample will play back as a loop or as a one-shot audio event before stopping - here we set it to loop indefinitely.

We call the setCalibrationMode method to set the calibration mode for sample playback - this is where we can set the overall level of the sound emitted by this node relative to others. Here we choose to do this by defining the relative SPL of the sound emitted by the node, and set it to 0dB.

Finally, we set the cullOption, which defines what PHASE does when a sound becomes inaudible (for instance because the user has moved outside the sound source’s radius). Here we have chosen that the sound should, in effect, continue indefinitely, inaudibly, muting and unmuting as necessary, with the imagined ‘start point’ being the point where the PHASE environment was started. There’s a bunch of options available here which can be used to interesting creative effect, including restarting the audio each time it comes into earshot, and starting randomly at a different point each time.

So, having described our sampler node definition, we now need to work our way back and look at its dependencies, of which there are two - the similarly-named (and related!) phaseSpatialMixerDefinition, and mixerParameters. Since we’re working backwards, I’m going to start with the mixer parameters, as it _also _depends on the spatial mixer definition:

The mixer parameters identify how a specific sound is mixed, for a given listener, within the environment. Here we pass in the PHASEListener we defined right back at the start, the PHASESpatialMixerDefinition that we will deal with shortly, and a PHASESource, which encapsulates the physical properties of the sound source within our world:

This is reasonably simple - we define a Source, which is made up of several PHASEShapes, each of which is defined by an MDLMesh which gives its shape and volume. A slight caveat at this point - in our fairly ad-hoc and subjective testing, we weren’t able to verify that changing the radius of this mesh had any effect at all on the sound itself (there are other parameters which affect the travel and diffusion of sound which we’ll get onto later). However, we didn’t look into this in a particularly detailed or rigorous manner, so your mileage may vary!

The other important thing to identify here is how you set the position (and scale, and rotation) of the object within the sound environment - by setting the transform of the source. This is one of the points where PHASE needs to integrate with whatever gameplay / xR framework you’re using, so we’ll highlight this and leave it undefined for now. The transform is expressed as an affine transformation matrix expressed as a simd_float4x4 - if this is starting to sound scarily mathematical to you (as it is to me, to be honest), there’s no need to panic - if you’re working with other Apple game and xR frameworks (such as ARKit or RealityKit) and attaching sounds to objects that exist within those (for instance visual assets or AR anchors), this is the same format as the transform property of those objects, so all you need to do is plumb them together. Be aware though, that depending on the lifecycle of your own application, you might not have this information available at the point where you instantiate the PHASE object graph, so you will likely need to keep a reference to this source object somewhere so that this can be updated later. At the end of this post, we’ll look in more detail into how this is done with ARKit as an example.

Having set up our mixer definition the only dependency that remains to sort out is our spatial mixer definition, which, however, comes with its own chain of dependencies we’ll build out as we go.

One thing that tripped me up here was the name of this particular object - from my heavily audio-engineering influenced perspective, you have one ‘mixer’ which handles mixing lots of different sources, and so initially I followed this intuition and created one spatial mixer definition which was shared between all my sources. This was frustrating because it meant that all the sources had the same parameters - FX sends, radius, level, etc. However, after some experimentation, I realised there’s no requirement for each source to share a spatial mixer definition - you can think of it more like a mixer channel or bus - you can create as many as are necessary - either one per channel, or one each for groups of sources that have the same properties. In our case, we create one per channel, which may not be the most efficient, but worked fine for our purposes:

So, the spatial mixer definition itself just links together two other objects, a PHASESpatialPipeline, and a PHASEDistanceModelParameters.

The PHASESpatialPipeline controls the various layers of how the environmental sound is built up, and you control it by passing it a set of Flagswhich control the layers of sound rendered - direct transmission (so the sound arriving directly at the ears of the listener from the source, early reflections, and late reverb. Here, we configure ours to add reverberation (recall we set the reverb preset on the engine earlier), and set the send level for this source:

So, the final piece of the jigsaw is the PHASEDistanceModelParameters, which is where we set the crucial properties that control the behaviour of our sound sources in the world we’re building:

The PHASEDistanceModelParameters class itself is an abstract superclass, and there’s a couple of concrete implementations we can choose from, allowing us to specify the behaviour of the sound source in different ways. Here we choose to use an instance of PHASEGeometricSpreadingDistanceModelParameters, which gives us a decent tradeoff of natural-sounding sound spreading, and ease of configuration, as it only has two parameters - the cull distance and rolloff factor. The cull distance is the distance (in metres) at which the sound becomes inaudible and stops playing (what happens when the user moves back into the source’s radius is defined by the cullOption we set earlier on our sampler node definition). The rolloff factor controls the steepness of the curve with which the sound level decreases over this distance - a value of 1.0 gives a halving of level with a doubling of distance, higher values give a faster decay, and lower values slower. Beware though that these parameters aren’t linked as such, and we found that particularly with small cull distances, we needed to manually tune a bit in order to avoid a very obviously audible hard cutoff at the cull radius. However, this was fairly easy to do, and much simpler than the other distance model parameters implementations, which allow you to, for instance, define an envelope over which the sound decays.

This completes the chain of dependencies for our sound source, so now, all being well, our project will compile, and if we’ve set the distance model parameters and positions of our source and listener such that the listener is in range of the source, we’ll hear some audio!

Without the ability to move our listener around though, the effect of all this setup is rather lost. Thankfully, at least from PHASE’s point of view, that is all very simple - you just set the transformproperty of our phaseListenerobject whenever the listener position updates. This, like the source position we saw earlier, is expressed as an affine transformation matrix expressed as a simd_float4x4. Of course, where you get this from, and where you choose to update the listener position, might be more complicated, and depends entirely on the environment in which you’re attempting to integrate PHASE. In our case, however, integrating with ARKit, this also turned out to be simple, assuming you’re already familiar with common patterns in iOS development, and the ARKit library. In our ARSessionDelegate, we needed to implement the session(_:didUpdate:) callback, from which it’s trivial to get the current transform of the listener in AR space from the frame object that’s passed in:

There’s one important caveat though - this transformation matrix does not take into account the rotation of the device at all, and, while it tracks position in-world accurately, the rotation around the head only works when the device is held horizontally, with the camera in the top left corner. This took ages to identify, and highlights a more general point - that it’s very difficult indeed to debug based purely on audio feedback - it’s easy to tell when something ‘sounds wrong’ or ‘isn’t working’, but quite hard to work out exactly what the issue is. For this reason, we recommend adding a simple visual debug mode to your app as early as possible if you’re working with audio-only experiences like us. In our case, the solution to this problem was to constrain the application to only display in landscape mode, and prompt the user to hold the phone in landscape while in the experience. This is far from ideal, but due to time constraints it was the pragmatic choice. However, with a little more time this should be easily solved - it should be possible to listen for the device rotation, then multiply the ARKit camera transformation matrix through a constant rotation matrix corresponding to the current rotation, before passing it into PHASE. I’ll leave this as an exercise for the reader for now, but will update this post if and when I get back into the project and fix this :-)

As a final bonus, I wanted to quickly show how to integrate the head tracking available in Apple headphones such as the Airpods pro, as this is very easily achieved (which came as a very pleasant surprise to us).

The head tracking information is available from the CoreMotion framework, via the CMHeadphoneMotionManager class:

At first glance there’s quite a lot going on here, but it’s actually very simple. In our view controller where we’re presenting the AR experience, we create a new headphone motion manager, as well as two simd_4x4 matrices, to store the current device position and headphone position respectively. In the onViewDidLoad callback we then request device motion updates from the headphone motion manager, passing in a callback which simply calls the handleHeadMovementfunction with the result. In the definition of handleHeadMovement, we have to do a little bit of song and dance to convert the CMRotationMatrix that Core Motion gives us to the simd_float4x4 that PHASE expects, which we then assign to the headphoneTransform variable we created earlier. Finally, we set our listener position to the device’s position multiplied by the head rotation, which, via the magic of linear algebra (and assuming the listener has their phone close by and roughly in front of them), gives us their listening position including the rotation of their head.

The only other change we need to make is to change our definition of session(_:didUpdate:) slightly, firstly to store the deviceTransform in an instance variable so we can get at it when the head motion callback is called, and secondly to apply the current head position transformation when the device position is updated too (this prevents glitches when the device position updates but the head position hasn’t yet updated).

So, that’s about it for now - you can see a demo of the finished experience in the video on our blog, and the code for it on Github (as well as some previous prototypes and experiments). If you’re within travelling distance of London, have a recent iPhone or iPad and would like to try out the experience, please get in touch with us, and we’ll let you know when we launch! Any other questions and comments are always very welcome and you can follow our updates here. If you need advice, consultancy, creative or development work done on an AR or spatial audio project, we’re available for hire! Thanks for reading.

Photo credit Auguras Pipiras.

Goals

We set out to further validate the hypothesis that using Audio Augmented Reality would create a convincing and compelling immersive experience. Unlike computer graphics running on most consumer hardware, the fidelity of recorded audio is great enough that our ears are already convinced by reality of what we perceive. The missing factor is sound emanating from a fixed point in space, and being rendered spatially. In short, sounding as natural as possible and closely approximating what we hear IRL.

There are plenty of headlocked, “spatial” audio experiences but it was important for us to investigate delivery of audio that is headtracked (doesn’t reposition as you move), has the sense of externalisation (the sound in relation to you) and presence (really being there).

Conceptually, we were interested in creating a playful, non-directed experience that offered surprising, uncanny sonic augmentation to the world around you. Using the city as a playground - a canvas upon which we could project directed audio stimulus that’s unconstrained by the physicality and hardware of a gallery setting, augmenting familiar surroundings with another layer of perceptual information. A 3D graphical score with a participant conductor.

One particular reference we kept returning to was the Situationist idea of the dérive - “a mode of experimental behaviour linked to the conditions of urban society: a technique of rapid passage through varied ambiances.” - an undirected, spontaneous exploration of the urban environment. This seemed particularly germane during the months in which we were working on this project - Facebook had just announced their “metaverse” strategy and rebrand, and a lot of the commentary and discourse around the announcement (as well as other artistic references we’d been inspired by) were concerned with the possibility that AR technologies might lead to further enclosure and commercialisation of public spaces. In this context, the idea that we might use the project both as an exploration of both the playful, open-ended possibilities of the technology for artistic expression, and a means of celebrating the importance of open public spaces and expanding them into AR-space, made the idea of a situationist-like intervention even more pertinent.

The result was The Situationists’ Walkman - a digital dérive, playing out within a small area of East London around Arnold Circus. An updated version of the Situationists’ vision using audio led exploration of the urban environment that prompts us to reconsider our relationship to this constructed or delineated space.

We commissioned a number of artists to make sonic artworks hosted inside overlapping zones around Arnold Circus and the Boundary Estate. Each artist provided a number of individual tracks or stems as well as directions of how to “attach’ them to the topography of the zone.

Strategy

The WWDC21 announcement of the PHASE audio engine from Apple really piqued our interest. That and the head tracking available via their Airpod headphones pointed at tools worthy of investigation for the project.

We evaluated numerous software environments (honorary shout out to Roundware) and frameworks en route to the prototype and what became clear is we had a number of challenges to solve:

1. Listener location - accurately position them in world space
2. Listener orientation - the direction of their head/ears
3. Virtual speaker positioning
4. Sound emission shape, direction and attenuation
5. Sound occlusion
6. 3D or spatial audio

Modern game engines such as Unity and Unreal offer the tools to design audio experiences in a constructed, virtual environment and provide all of the properties listed above. That is, assuming you design a sound stage and navigate around it in a fully immersed Virtual Reality environment via a hardware headset. We require a similar experience but instead, overlaid IRL. One of our guiding principles is for the technology to disappear into the background so the participant can become fully immersed, without getting distracted by the medium. It seemed like a good time to refamiliarise ourselves with these tips for creating audio AR experiences from Henry Cooke and BBC R&D friends before kicking off to avoid any pitfalls they uncovered.

You can take a deeper dive into the technology we used and how we arrived at the final experience over at this post.

Results

After a lot of work finessing onsite, we had an experience that we were really happy with! It’s very effective wandering around and discovering augmented sounds emanating from various points around the site.

Here’s a DIY video of part of the experience. It can’t really capture what it feels like to explore the site but will give you a sense of it. Make sure you wear headphones otherwise the spatial effect will be lost.

During testing, we noticed a really uncanny perceptual side effect whereby after exposure to augmented audio, particularly any material that was clearly not coming from the local environment (that was not always obvious), your ear really tuned back into the natural sounds in a way that focussed you sharply back into the location - almost like a heightened state of sensory perception!

Another interesting observation and an area we’d explore further is we preferred the experience without the Airpods head tracking data! This is counterintuitive but having the hardware device held against your chest meant limiting the amount your head can turn before you twist your body anyway. There also seemed to be some lag between quick head movement and the soundstage reorientation.

We plan to run some tests with a small group of participants onsite in March. After that, we’ll release the experience to a wider group via Test Flight before pushing to the App Store. Drop us a line if you’d like to be part of the test groups and get early access! N.B. - you’ll need a LIDAR capable Apple device and iOS 15 or later.

Thanks and credits

Concept, Design and Development by: Matt Spendlove & Tim Cowlishaw

Featured artists:

• Elvin Brandhi
• Tim Cowlishaw & Constanza Piffardi
• Sally Golding
• Mark Harwood
• Nick Luscombe
• The Nonument Group
• Ruaridh Law
• Spatial & Oliver Coates

View a map of the participating artist zones here.

Produced by Cenatus with the generous support of the Innovate UK Creative Industries Fund.

Thanks to Dave Johnston, Irini Papadimitriou and Henry Cooke for mentoring and advice.

If you’re interested in our work you can follow along here or please get in touch if you need some help designing spatial audio experiences!

To recap and contextualise, these are the challenges we needed to solve:

1. Listener location - accurately position them in world space
2. Listener orientation - the direction of their head/ears
3. Virtual speaker positioning
4. Sound emission shape, direction and attenuation
5. Sound occlusion
6. 3D or spatial audio

Prototyping & technology

One early idea was to construct a 3D “digital twin” of Arnold Circus via mapping and LIDAR data. The hypothesis was we could then work on that model in either XCode or Unity/Unreal to place the virtual speakers and design the artist zones, leaning on an audio engine such as PHASE or Google’s Resonance Audio for spatial sound. An advantage of this approach would be the ability to develop and test offsite on a PC or even in VR. The trick would be to later swap out the first person navigation from joystick/keyboard with realworld listener positional data via GPS. This approach seemed advantageous by leaning on a lot of tried and tested subsystems.

We were warned a few times about the lack of precision from GPS and infrequent updates but what didn’t really coalesce until discussing amongst the wider XR community was that the accuracy was a likely problem for the listener positioning rather than speaker location. The analogy would be a first person shooter video game with a very low frame rate. Rather than jittery graphics we’d likely experience audio glitches as the audio engine calculations tried to keep up with the low resolution updates. We couldn’t afford any playback issues that would break the sense of immersion. I should clarify that we didn’t have capacity to test this out with a prototype so had to reason it through. That’s something we would change in hindsight.

Whilst looking into ARKit, we became aware of ARGeoAnchors as one of the possible content anchors. These offer the ability to anchor AR overlays to persistent world locations, alongside the more familiar surface, image, object and body or face recognition. One important restriction is that geo anchors are currently limited to a number of U.S. cities and London - we just snuck in there!

At this stage the first big compromise started to become clear. To allow us to achieve 1) above - if we lean on geo anchors, then the participant must have their phone out! To enable the kind of accurate location lock AR experiences need, the Apple device and ARKit will get an initial location from GPS data (similar to how you navigate in Maps) and then download pre scanned 3D image data around your approximate location and compare that to feedback from the device’s LIDAR depth sensor. We decided to accept the tradeoff initially and dive deeper into ARKit to see what other advantages it brings. Ultimately, we could still prompt the user to position their phone in a way that doesn’t encourage them to look at it e.g. hold it to their chest and provide experience feedback or instructions aurally, discouraging any need to look at the screen. A third eye, if you will!

To dig deeper into the capabilities of ARKit, we needed to get familiar with creating a model or an ARReferenceObject (*.arobject files) to trigger content anchors. To do this we used some sample code from Apple to scan and detect 3D objects and create the model a water bottle (amongst other things). One can then set up code in XCode to detect a model and trigger AR content.

A far simpler way to test an object or model recognition is to load it into a Reality Composer project. You can see an example in this Github repo where we animate properties of a simple geometric shape to appear via a behaviour once the object is detected. It’s worth noting at this stage, if you create a new project in XCode from the “Augmented Reality App” template alongside SwiftUI, you’ll generate the boilerplate code to run up and load a Reality Composer *.rcproject file into a RealityKit runtime. This theoretically means you can prototype or design in Reality Composer before you fully integrate into your experience.

Next we worked on building an indoor AR experience that triggered audio on object recognition. Starting simple with just one object + sound before adding a few into the scene and creating a mix of audio as you move around.

That worked pretty well but alas, we’re not quite done yet as there didn’t seem to be a way to control the radius and attenuation of the sound using the built in SCNAudioPlayer. We’re putting the finishing touches to another post focussing specifically on the various audio engine options but we trialled and discounted AVFoundation (attenuation settings were global, not per source) en route to getting PHASE going as in our test AR project. The subsequent repo commits tweak the settings and a visualisation wireframe mesh. Stay tuned for a tutorial on using PHASE in projects such as these, as the official documentation is rather sparse, and we ran into some interesting problems along the way.

This gave us some confidence that we could use PHASE alongside an AR experience but also lead to a bit of confusion. The code examples we followed and used as the basis for our prototypes lead us to working directly with ARKit, with SceneKit for rendering. That’s all good but Apple seems to be pushing towards RealityKit as the preferred option for AR and the APIs and delegate callbacks are different enough to require a bit of work to switch between.

Also, the examples we found that implemented ARGeoAnchors for a real world experience were using RealityKit! The current message is a bit mixed from Apple but the important distinction is to recognise that RealityKit has its own rendering engine and doesn’t require SceneKit. This doesn’t really matter when using PHASE for audio but obviously you couldn’t then use the Scenekit SCNAudioPlayer.

Once we verified using PHASE with ARGeoAnchors, we could start thinking about UI. We settled on SwiftUI which is the newer, more familiar, declarative UI toolkit. The structure of SwiftUI projects is a little different from traditional UIKit projects so we needed to work out a bit of plumbing to translate the correct lifecycle elements from our UIKit prototypes in SwiftUI style.

The most important part of this is to have a struct that implements UIViewRepresentable, acting as a bridge to traditional UIKit code. Our ARViewContainer uses a Coordinator class to implement the necessary RealityKit (and underlying ARKit) delegate lifecycle callbacks. This gave a reasonable container for most of our experience code where we could do tasks like set up the AR coaching overlay, parse and load our virtual speaker information from hacked version of the GPX format and receive AR frame ticks to update the listener position in PHASE and generally manage its lifecycle.

We now have the majority of our challenges solved with a couple of caveats.

We ran out of time to dig deeper and test sound shape and direction from 4), only using attenuation, which was enough to design a first pass of the experience. The bigger omission is 5) Occlusion. Whilst this is certainly possible with PHASE, we’d need to manually model all the building geometry in our zone and teach the engine about it. This was just too big a task to be valuable given the time constraints and we are happy enough to drop that for this iteration.

Please check out the intro post to preview the experience and in the meantime, if you’re interested in our work you can follow along here or please get in touch if you need some help designing spatial audio experiences!

​​MetaSounds are fundamentally a Digital Signal Processing (DSP) rendering graph. They provide audio designers the ability to construct arbitrarily-complex procedural audio systems that offer sample-accurate timing and control at the audio-buffer level.

This is a huge leap forward and they provide a graphical patch editor to wire together low level audio components to generate and process sound. These patches can then expose parameters to the engine runtime and enable tight integration between gameplay (or experience play) and sound. The UI is similar in concept to Pure Data or Max MSP and clearly targets sound designers and composers. Colour me very interested!

I really wanted to explore the possibilities for use in immersive environments so as part our CIF2021 R&D work I proposed a prototype to research the use of procedural audio in creating an audiovisual artwork that can adapt and transform in real time within a Virtual Reality “activated environment”, based upon user exploration and interaction.

Creative possibilities

The potential of this style of integrated software is HUGE and affords the kind of control we can only dream about when creating physical gallery installations or expanded cinema work. My experience building audiovisual performances taught me that fundamentally we’re working on a perceptual level using psychophysical principles. Our brains are fantastic pattern matching systems so providing the right cues and mappings can lead to convincing, integrated multi sensory experiences.

A simplistic, concrete example would be using synthesiser envelope data to shape audio and visual stimulus in unison. When used in audio synthesis, an envelope represents the change in some value over time. Mapping that same value data to some visual property e.g. the opacity of a primitive shape, can really bring the visual to life, make the animation dance and solidify the connection in our brains.

Technology overview

As you can see in the header image, MetaSound patches are a graph of nodes and connections that represent an audio signal path. The editor provides access to insert and connect these nodes to build out the patch.

At the time of writing, MetaSounds are only available via plugin as part of the Unreal Engine 5 early access program and are under active development. Our research took place on the EA2 branch but I know from following #metasounds channel on Discord that the API has changed a bit after EA2.

It’s very crucial to grasp that source nodes in your MetaSounds graph could be traditional, static audio wave files for sample playback and processing but far more interestingly can be waveform generators like the classic Saw, Sine, Square, and Triangle shapes. Combining audio sources with filters, envelope generators, LFOs, mixers and maths functions provides building blocks to create everything from familiar subtractive synths to elaborate, custom hybrid synth patches - all with a set of exposed interface parameters that can be controlled by anything else in the game engine.

Alongside these prebuilt nodes exposed via the patching UI, it’s also possible to create your own nodes by writing some C++ and including them in a manual, source build of the engine.

FM Synth

The classic audio generators handily include a frequency modulation input for FM synthesis techniques so I decided to build an FM synth patch as the basis for my interactive experiments.

In FM synthesis, we use a “modulator” signal to modulate the pitch of another “carrier” signal that’s in a similar audio range. This modulation creates new frequency information in the resulting sound, changing the timbre or colour of the sound by adding more partials.

In recent years, I spent time learning synthesis techniques in greater detail via hardware modular systems and software like Supercollider. The latter is an incredible open source real time, interactive synth engine and an essential learning resource for DSP. I revisited Eli Fieldsteel’s essential course materials, focussing on the FM lessons (part 1 & 2) and worked on porting that across into a MetaSounds patch.

I won’t repeat the instructions here but if you follow along with his video it should be simple enough to translate. I’d encourage you to do so particularly if you want to get a sense of the wide range of potential sounds from even the most basic FM synth whilst getting a demonstration of how powerful randomising the control values can be for sound design. This is the crux of why procedural audio in game engines is so appealing for developers.

FM Basic

In the short video below, we can see the MetaSounds implementation of Eli’s basic FM synth from Supercollider:

FM Ratio Index

Following on from that, a slightly more advanced example from the second lesson. We use this as the basis for further experiments. You can use following screenshots to recreate the patch:

The playlist below demonstrate the effect of changing the modulator, carrier ratios and other inputs:

Blueprints

All the examples so far show patching and playback directly from within the MetaSounds editor. To actually publish the MetaSound inputs to other components in UE5 for runtime control, you currently need to wrap the MetaSound into a Blueprint. You can see an example below where we wrap and expose the MetaSound inputs as Blueprint variables:

The next screenshot shows an example of using the UE5 sequencer tool to modulate some of those parameters over time. You could, of course, use any aspect of your game to manipulate the synthesiser:

Summary

We’ve seen how we can build synth patches in MetaSounds and ported an FM Synth from Supercollider over to learn the basics of FM synthesis on the way. Stay tuned for further posts where we’ll show some of the work we built utilising these techniques!

The award is funded by Innovate UK, the UK’s innovation agency. Their remit is to drive productivity and economic growth by supporting businesses to develop and realise the potential of new ideas. They fund business and research collaborations to accelerate innovation and drive business investment into R&D. Their support is available to businesses across all economic sectors, value chains and UK regions. Innovate UK is part of the wider UK Research and Innovation.

eXtended Reality (XR) experiences

Our application is to undertake research and development of two prototype XR experiences, with a particular focus on 3D sound.

The first explores the use of procedural audio to create an audiovisual artwork that can adapt and transform in real time within a Virtual Reality “activated environment”, based upon user exploration and interaction.

The second will investigate the use of head tracked, 3D audio to provide the foundation to overlay exciting sonic artworks, stories and experiences that transport us from, or augment our real world environment.

The funding helps us design and build these prototypes and will bolster a portfolio of work aimed to generate new client services and help identify market gaps for new product development.

Team

Working alongside frequent collaborator and ex BBC R&D colleague Tim Cowlishaw, we’ll create a playful, research lab environment over a three month period to explore these areas.. Our work will be guided by mentors David Johnston, technologist at Digital Catapult, Irini Papadimitriou, creative director at FutureEverything and Nick Luscombe, creative director of Musicity, broadcaster and ex BBC Late Junction host. Alongside our official mentors, we’ll be sharing and evaluating our work within our network of artists and technologists to get valuable feedback.

We are extremely excited to get started and explore the potential of this space and thank Innovate UK for their confidence and support!

If you’re interested in our progress then subscribe to this blog for updates, ideas and code samples as we proceed. You can view all related posts using the CIF2021 tag.

Photo by Tanner Boriack

I nearly left the goodship Apple when they seemed incapable of releasing a laptop that was an actual worthwhile upgrade from my previous model. They redeemed themselves at the 11th hour upon release of the Macbook Pro 16,1 in 2019.

I’d already spent a decent amount of time researching Windows based machines but ultimately I didn’t really want to switch. I have a lot of investment in the Apple ecosystem and my audio production work still leans very much towards Mac. That meant compromise as the 3D and XR world is still very much in Windows land with the Apple GPU profile not really targeting 3D and gaming. In addition, Oculus don’t support OS X with their Rift or Link software and whilst the Steam gaming platform will run, they dropped support for SteamVR on OS X.

My compromise was to buy the Mac, start my XR work on that platform and see how far it takes me. My fallback options are to add an eGPU, dual boot the machine in Windows via Bootcamp or ultimately, consider a Windows desktop or gaming laptop later down the line if required.

That served me well enough for a while but I eventually had cause to reconsider. I attended an online course on ImmersAV for audiovisual art with Bryan Dunphy at the London Music Hackspace. It’s an impressive looking toolkit or custom engine that allows parameter mapping between raymarched OpenGL graphics and the Csound DSP audio engine via a C++ interface. The results can be previewed locally in 2D dev mode and deployed to a SteamVR compatible headset using OpenVR. Whilst I was able to build and test the codebase on my Macbook, I was unable to run on my Quest due to the lack of Oculus Link compatibility.

The second nail in the coffin was hammered in by working on a prototype where I wanted to explore mapping light and sound in Unity using real time synthesis, rather than sample based audio - think something along the lines of Anthony McCall’s Line Describing a Cone, but with live DSP. The subtle tweaks I needed to make to the synthesizer parameter mapping required constant previewing and adjusting so the offline build and deploy process to target my Quest was laborious and unmanageable. I bit the bullet and decided to dual boot my Macbook with Windows. This would allow me to hard link the laptop to Unity via an Oculus Link cable and do previewing directly from within Unity itself.

Windows tools

The Windows install process via Bootcamp was straightforward enough, but I quickly found myself to be very inept at navigating around Windows due to the muscle memory I have for OS X. I added a few tools to make it more Mac-like and make switching between the two operating systems less painful.

After Unity itself, the first thing I installed was Git (for Windows). This comes with the _incredibly_useful Git Bash application which gives you a familiar CLI to work with. So far I haven’t needed another terminal utility. Next I installed Trackpad++ to tame the Macbook trackpad and scrolling. I then installed Sharp Keys but writing this post sometime afterwards I can’t seem to get that app to open so I’m not entirely sure if I used it! I think I likely just remapped the Ctrl and Windows keys on my Apple Keyboard in the Registry so I could use the CMD key as Windows Ctrl key which is what I’d distinctively try and hit. I definitely did use the Microsoft Power Toys utility to remap some of my common keyboard OS X shortcuts. I enabled PowerToys run with Ctrl/Cmd + Space which opens a quick launcher, similar to Spotlight on OS X - the way I tend to open everything. Next I mapped some keyboard shortcuts:

Now that I could sensibly navigate my way around the Windows environment I installed the Oculus Link software so I could enable Link on my device and have a more immediate development flow in Unity :)

I have a suspicion this all might be another stepping stone towards having a dedicated Windows desktop or laptop for XR work but I’ll see how far this next iteration takes me first!

Image Credit: Installation view of Anthony McCall: Split Second at Sean Kelly, New York / Photography: Jason Wyche, New York.

Simulating and expanding the physical installation, participants in the online version will be immersed in a work that plays with perception, interactivity and unexpected encounters – including the viewer’s own reflection captured and integrated within the artwork.

Assembly Now [online] uses the interface of the mirror to elaborate the psychology and technology of emergent algorithmic software, which functions as a contemporary screen filtering our emotions.

We live in an age of ubiquitous photography– selfie culture, and surveillance capitalism– facial recognition and eye tracking. Emotion analytics software used in neuro-marketing and image recognition is a blend of psychology and technology– capturing data on expression, to assume correlations in mood determined by machine learning.

A discussion between artist, Sally Golding, and writer and researcher, Kate Warren, about Assembly Now, can be read here.

Cenatus collaborated with Sally and Tim Cowlishaw on this project from concept through technical implementation and presentation.

• Experience
• Press Kit
• Github repo

Cenatus worked on multi-channel sound design and provided creative technological solutions. We created a series of tools in Supercollider for producing generative sound clips that could be triggered and improvised with in response to the actors movements and changing dynamics of a live theatre scene. It was important for the audio narrative to be responsive to the actors in realtime and to provide interesting variations around sonic motifs.

Alongside original sound design, we worked in collaboration with artist Sally Golding, providing tooling for her own sound design and sonic responses to the creative brief.

You can watch a video over at Oily Cart.

Source code is available via Github.

The Unity audio engine is based around Audio Sources than can either be audio files or tracker modules, routed through a hierarchy of mixers to balance levels and optionally apply FX. The sources can be played back either 2D or 3D. This video from Unite 2015 gives a decent overview of the concepts.

DSP

That’s cool but I was a little surprised (perhaps naively) that there was no built in DSP for procedural audio - a term I was introduced to by Andy Farnell back in 2011. Certainly for the prototype ideas I’m considering, I was hoping to work directly with DSP on device. The approach from Unity seems to be to expose a plugin SDK and leave synthesiser creation as an exercise for the user if required. The audio dev community has filled in some of the gaps - the JUCE framework from ROLI is now supporting Unity as a build target (desktop only) and there are projects to publish Faust and PD via UnityPD or Heavy based instruments as plugins.

For more simplistic scenarios, you can bind to OnAudioFilterRead directly from a C# script as in this example oscillator but I suspect there’s a performance limit to how far you can go with this approach - here’s a quote from the Native Audio Plugin SDK:

“Unlike scripts and because of the high demands on performance this has to be compiled for any platform that you want to support, possibly with platform-specific optimizations”

A higher level commercial plugin solution does exist in the form of Audio Helm which looks really promising and supports desktop and mobile targets. It has a built in sequencer, sampler and synth engine with GUI and scripting support. There’s also a standalone version of the synth engine you can use for sound design and then publish the patch to Unity for native app use. Your patches can then have their parameters scripted which is exactly what I was hoping for. It costs ~70 EUR but seems like a good workflow with minimal fuss.

Worth noting here is that Unreal Engine 4 from Epic seems to be well ahead in this regard as this video from 2017 details. There’s still doesn’t seem to be much documentation around but the API is here. It’s not clear to me if this stuff is out of beta but this functionality alone is enough for me to take another look at UE4!

3D Sound

Audio immersion is essential for realistic XR experiences and Unity ships with spatialization baked in - it includes a basic version of the Oculus Native Spatializer (ONSP) which can be upgraded to the full version or swapped out for solutions from Microsoft or Google’s Resonance Audio project. Check here for the high level 3D sound concepts from Oculus and this guide for the details. I found this Spatial Audio video from Oculus Connect to be a really good overview of the concepts and tooling required for 3D sound.

For a real world, end to end immersive audio production guide check out this recent Spatialized Music for AR/VR presentation. It follows the entire process of producing the audio experience for the interactive Oculus First Steps tutorial that ships with the Quest. It follows the journey from sound designer/composer, through studio recording of the orchestra, post production and to final application integration. It covers the strengths and weaknesses of ambisonic mixing along the way and when to use head tracking for positioning audio or when to just mix in quad.

I also bookmarked the Facebook 360 Spatial Workstation which looks interesting: “..a software suite for designing spatial audio for 360 video and cinematic VR. It includes plugins for popular audio workstations, a time synchronised 360 video player and utilities to help design and publish spatial audio in a variety of formats.” This Spatial Audio for Cinematic VR and 360 Videos article covers creating immersive audio content targeting Facebook and Youtube as well as XR hardware.

Ambisonics

Unity supports the integration Ambisonics out of the box so you can import multi channel pre-spatialized sound files but you’ll need a decoder from either the Oculus or Google audio SDKs I mentioned above.

Middleware

Fully featured audio middleware and end-to-end solutions exist from Fmod and Wwise that are standalone immersive audio production solutions with integrations into Unity and UE4.

I started Sprint 3 keen to build something and revisited the creative coding tutorial in Unity recommended by David Johnston at the beginning of the project. The thing that initially struck me about this tutorial is how it stripped back the Unity ceremony and “complexity”, the latter is actually aimed at simplifying the process for game devs but can be daunting for a new user with so many options available. The focus here is on pixels and a canvas, taking you a little closer to the metal which aligns with previous experience of Open Frameworks and Processing.

What follows is the notes I made for each episode whilst reviewing the tutorial. This is mostly for my own reference but could be helpful if you need to dissect the basics of creating dynamic animations in Unity. I’ve included a direct video link and Github commit for each section, where relevant

Episode 1 helps to orientate you inside the virtual space, explaining your development perspective, the camera perspective and the default skybox. It guides you through the creation of a Cornell Box, and helps explain the default lighting within a scene and how you can take control of it.

video

Episode 2 brings the focus inside the Cornell Box, setting up our own lighting and dealing with optimisations like light prerendering. It also introduces the concept of reusable materials that can be applied to any object in your scene - specifically those that emit light in this case, which are really powerful.

video | code*

*mostly Unity metadata so not much to read, subsequent commits are more useful.

Episode 3 deals with attaching scripts to game objects and the basics of manipulating object properties via transforms over time to create animation. Also fundamentally, the decoupling of scripts and objects and how you must apply a script to an object to get it running. Another core concept here is publishing public attributes within a script that are made available in the Unity editor or other code clients for dev and runtime interaction.

video | code

Episode 4 continues the familiarisation of animating an object in space. The important concept introduced is the creation of a prefab - essentially a prototype game object and script that can be reused from other scripts. In this instance, we create “spinning cube” prefab to allow us to add many cubes to the scene from one reusable definition.

video | code | code

Episode 5 is about creation and management of game objects programmatically. Now that we have a reusable spinning cube component or prefab we need some code to manage the instances. That’s where the game controller or sketch object (in line with Processing nomenclature) comes in - this is the entry point to our scene.

video | code

Episode 6 is more of a general creative coding / animation tutorial covering trigonometric functions and easing techniques. Very useful background and not specifically related to Unity but still really valuable.

video | code

Episode 7 covers processor efficient techniques for ensuring dynamic prefabs respond correctly to static lighting via Light Probes - a mechanism to pre render light at specific node positions that objects can inherit.

video | code

Episode 8 is the final in the series and reviews the work so far before investigating post processing camera effects such as Bloom and Depth Of Field by using pre existing packages for the first time. The tutorial is a little out of date regarding post processing effects so you can read an up to date guide over at the Unity docs.

The tutorial wraps up with a couple of resources that inspired the series. Cat Like Coding by Jasper Flick and the Holistic Game Development book by Penny de Byl.

video

I really learned a lot from these and felt much more confident in Unity on completion. I hope you find the breakdown in my notes useful too. Many thanks to Rick Barraza for the series!

Prototyping Concepts

Energised by the tutorial I started thinking about what I might like to make. An obvious follow up to this would be to port a version of my Transduction project. The geometric animation would be great as a 360 degree XR experience and I now understand the basics of how to implement that.

The event pattern generation for the original project is created via Tidal Cycles and fed to Supercollider (the audio engine) and Processing (the visual engine). It would be pretty trivial to keep a similar architecture and just swap Processing out for Unity as the transport protocol is all OSC, but I’d like application to be self contained in the Oculus Quest.

I’d like to start with event pattern generation and DSP audio engine inside Unity which maybe less dynamic (i.e. there’s no livecoding interface) but I’m sure I can still create something satisfying. The next challenge is to look into DSP and 3D Sound options for Unity so I’ll tackle that next time.

The focus for Sprint 2 was to get a basic “hello world” example running on my Oculus Quest. This type of program is normally trivial but it fleshes out correct configuration of the hardware and software environment, laying the foundations for more interesting projects further down the line and getting confidence with the tooling.

The first challenge was to settle on a version of Unity. The aforementioned Coursera XR intro course suggested fixing to an older version but that was restrictive for other online tutorials so I settled on 2018.4, installed via the Unity Hub. That also meant I no longer needed the Android Studio app and accompanying SDK as it’s now included in Unity as a module via the Unity Hub. I did manage to build and deploy the “VR campus” app from within the XR course but I quickly realised I needed to start with a more basic example that required less hacks and libraries.

The installation was much more of a dance than it sounds as my Internet connection is “challenged” at home so I needed to go find a café with a good connection and ideally, only install the essentials. You really notice how much development tooling assumes a solid connection when you don’t have one!

I generally found the guides from Unity / Oculus not only sufficient but usually better than third party tutorials. The caveat with the Unity guides is the availability of so many versions of their software and help docs. You need to be really careful what version of the docs you’re reading after following Google links. I guess this really highlights the pace at which the industry is moving right now!

Hello World

Once everything was configured I followed the Oculus Build Your First VR App tutorial from the Getting Started docs. This also guides you in creating basic Game Objects in Unity– useful if like me, you didn’t have too much game dev experience before starting to learn XR. It didn’t take long to develop the app and I got it deployed to my headset following the latter part of this Quest dev tutorial - success!

That’s a great start but I already began to wonder about the development feedback cycle. I’m familiar with using tooling (REPLs, live coding etc), tests and the running application to trial my ideas with quick feedback. Do I really have to build and deploy an Android Package (APK) to the device to test it out? The short answer is yes, I’m afraid. The situation is different for a Rift as you might expect with it being a tethered device. More about that in a later post, probably..

Inspiration

Alongside the techy stuff, I also reached out to my network and beyond to anyone working in XR related fields for advice. This started with my third mentor meeting, hosted by Irini Papadimitriou, ex V&A digital programmes manager and now creative director of the brilliant Future Everything festival in Manchester - an organisation I’ve collaborated with in the past. Irini’s mentor role on this project is as a curator and producer. Her enthusiasm was infectious in our meeting and she gave me an extensive brain dump of XR projects she’d experienced.

From Irini’s suggestions there were too many great projects to mention but a couple stood out. The first was Hack The Senses and their 2017 V&A collaboration. The first two installations had a lot of crossover with my own interests - particularly the sensory perception topics I touched on in the previous post. The second was the celebrated Notes On Blindness recordings, film and accompanying VR experience. The film is based around John Hull’s audio diaries that he kept whilst losing his sight. The diaries are an articulate, compelling reflection on his situation and the changes to the world as he perceives it. Following a short film and subsequent BAFTA winning feature film, the VR experience is a triumph - a spellbinding narrative delivered using the strengths of immersive technology. The scenes are visually opaque, abstract yet magical. Careful 3D sound design really accentuates the diary narrative and helps to place the listener inside John’s story and experience - well, as much as anything can hope to. Smart, subtle gestures highlight or reveal parts of the scene. I particularly liked it when John was talking theoretically about the sound of rain hitting an object being useful in emphasising and positioning it with space and lamenting the lack of indoor rain for the same purpose. The VR creation of this scene gradually reveals each new object in the room visually and, more importantly, sonically through gaze based gestures. This additive process subtly forges a wonderful rhythmic Music Concrète soundtrack. The 3D sound design here particularly shines. I really can’t recommend this story, film and immersive experience enough!

3D Sound

Talking of 3D sound, I reached out to Tom Slater at Call and Response, another organisation I worked with a few times in the past. They curated an 8 channel surround sound sonic art programme for our Netaudio festival way back. I also mixed an ambisonic release for Zimoun’s Leerraum project at their Deptford 18.1 studio and completed a great ambisonic workshop they ran last year. Alas, logistics prevented us meeting in person so far but I was pleased to read they were successfully awarded funding for exciting projects from Creative XR and The Jerwood Foundation and hope to follow up later.

Industry XR

Lastly, I got some good insight into commercial XR and game development industry from my old friend Craig Gabell who co-founded the Brighton based West Pier Studios. Of particular interest was the mixed use of VR / AR technology in the My Virtual Space application they developed as a tool for exploring and customising environments.

Once I started investigating XR properly I realised it’s an absolutely vast field, even within one of the disciplines. I’d provisionally secured support from a number of mentors as part of my application so was very glad to lean on their expertise to help distill some of the information out there during my initial sprint.

Hardware & learning

My first mentor meeting was with David Johnston from BBC R&D, a rare and valuable mix of technologist and producer. We had a lively discussion and I instantly felt more confident with his support.

After we met, David sent over a load of resource links covering hardware and software. He recommended the online learning from both Unity and Oculus which seem great and also included a very interesting creative coding tutorial in Unity which aligned with my experience in Processing and Open Frameworks.

David also helped me zone in on purchasing an Oculus Quest which seemed to offer the right balance of ease of use and flexibility for my project. I’m not a hardcore gamer and my aging Macbook Pro is not compatible with many of the tethered headsets on the market so the all-in-one device appealed to me, even at the expense of some rendering power. I was also tempted by some Bose Frames, especially as some BBC R&D friends were working with them for audio AR prototypes but the Quest offers my more bang for my limited buck.

The Quest runs on Android so as long as my PC can manage Unity than I’m good. I did briefly consider other game engines but already had a bias of interest to Unity before I this started and couldn’t find a compelling reason to not follow that.

I enjoyed exploring the Quest - the ease of use for creating a guardian zone was impressive (a safe play space for standing 6DOF VR experience). I don’t have much to compare it to but I’m very happy with the product overall.

Perception Hacking

My second mentor meeting was with Dr. John Greenwood, a researcher in Experimental Psychology at UCL. John’s research is fascinating, focusing on visual perception and it’s disorders via psychophysics - “the scientific study of the relation between stimulus and sensation”. There’s a great overview of his projects over at Eccentric Vision.

He started by demonstrating the power and trickery of peripheral vision and the problems with visual crowding before going on to explain the relationship between conditions such as Amblyopia (often called lazy eye) and diseases such as dementia and Alzheimer’s. A big takeaway for me was to understand failing peripheral vision as a possible predictor for dementia.

We talked about various perception hacks/flaws including the Wagon Wheel Effect, and how the visual system can override the auditory one as demonstrated by the McGurk Effect. I had a written note about “Bunny Hop Illusion” which could refer to either of these; the first is a tactile illusion called Cutaneous Rabbit Illusion and the second is The Rabbit Illusion from Caltech.

John followed up with some slides on Spatial Vision: “the perception of the distribution of light across the visual field” which are “The ‘building blocks of object perception in the early stages of visual processing”.

Whilst studying the slides I also learned that Fourier transforms could be used to analyse images. I understood that sound can be created / distilled from simple sine waves but didn’t know about that generalising to any signal. Whilst looking for a good link to explain the theory I found An Interactive Introduction to Fourier Transforms by Jez Swanson - one of the best things I’ve seen on the web for ages! It breaks down the concepts in a really fun way, making it simpler to understand. The slides referenced Basic Vision: An Introduction to Visual Perception which I ordered instantly.

Concepts

If you’re wondering what all this has to do with my XR research then take a look at my alpha mind map I created before I met John. Gestalt Psychology has been of interest for a while so I’m keen to explore other areas of perception and the way the brain tricks us into understanding the world.

Online learning

I worked through part of the aforementioned Introduction to XR: VR, AR, and MR Foundations Coursera. The videos and general structure were very welcome but I’m not so keen on the peer assessments and a bit suspicious of the “brainstorming XR apps” module. The biggest criticism by far though is the lack of engagement on the forum. They are pretty much a ghost town, with zero input from the tutor so I struggle to see the value? It might make more sense to follow the free Unity learning resources and take your chances in their forums.

Blogging software

Last but not least, I also spent some time extending this very website to give me blogging capabilities. It’s a custom site built with Elixir/Phoenix and I considered adding a static blog generator to the domain instead but I want to limit the amount of software I’m maintaining so was happy to consume the development overhead to keep streamlined and encourage some general maintenance to the site which was well overdue.

For the uninitiated, XR stands for Extended Reality and is an umbrella term for Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR) and concepts like 3D Sound and 3D Video. The distinctions can be blurry, but here’s a one liner on each:

• VR: creates a digital environment that replaces the real world.
• AR: overlays digital content into the real world.
• MR: Umm, a seamless blend of the two?!
• 3D Sound: Either recorded with multiple microphones or reproduced via a speaker array.
• 3D Video: Similarly, multiple cameras record different perspectives on the same shot.

Looking around when kicking off, I found this Introduction to XR: VR, AR, and MR Foundations course on Coursera. Pretty useful so far.

Armed with that, a Unity install, an Oculus Quest and a load of curiosity - let’s see where this goes. Should be a fun ride!

I’ll be posting updates in the blog section and you can subscribe via rss or email for updates.

It has responsibility for researching and developing advanced and emerging media technologies for the benefit of the corporation, and wider UK and European media industries, and is also the technical design authority for a number of major technical infrastructure transformation projects for the UK broadcasting industry.

Cenatus have worked on a number of projects within R&D:

Not A Robot v2

Not A Robot v2 is a multiplayer Audio AR game where participants are invited to deprogram themselves from their digital rituals and practice real human interactions, to listen to each other and the world around them. A voice assistant guides the group through a number of exercises, which gradually increase the level of human interaction. The project allows users to interact in real-time and space with each other, without the barrier of a screen by using Bose Frames.

Extending the initial research project, Cenatus started working with the team to brainstorm new concepts and chapters in the experience. We provided placeholder Supercollider generative audio sequences during the workshops that were later incorporated within the game.

After the initial workshops, we implemented ambisonic, adaptive audio sequences and gameplay in Unity for further user testing, including a chapter allowing collaborative music making session amongst the participants.

Editorial Algorithms

At the heart of the experimental system created through this project is a scalable pipeline for the ingestion and analysis of online textual content.

Cenatus worked on an API to allow the creation and storage of arbitrary streams of data ingested by the system. We also built separate UI component allowing end users to configure streams of data based upon a subset of data sources ingested by the system, filtered by entities from DBpedia. [more info].

CODAM

CODAM is a collaborative project funded by Innovate UK in the area of video fingerprinting and visual search. The partners in the project are the University of Surrey and Visual Atoms. The project aims to deliver a toolkit to track and identify footage in very large video archives. This could be of significant help to broadcasters who quickly need to find a piece of footage for legal reasons, or for programme makers who want to find shots of a particular object or scene.

Cenatus worked on developing and enhancing the existing web front end that allowed video ingestion and users to search for frames or video clips ingested by the system as well as searching for ‘similar’ objects within a particular frame. [more info].

Quote Attribution

Citron is a BBC R&D exploration into understanding how a machines might help judge the veracity of a news article. In order to understand the veracity of a claim, it is helpful to know who is making it and what their interests are, and this is something a machine can assist with.

Cenatus worked on an prototype API / UI to expose a searchable database of all quotations cited in BBC news articles over the past few years. This was used to demonstrate the value of the underlying data to journalists and audiences - creating tools that allow people to understand the types of claims made in the news media and their provenance.

DSRP

The Data Science Research Partnership is a five-year research partnership with eight UK Universities to unlock the potential of data in the media, commencing October 2017. The Data Science Research Partnership aims to be at the forefront of machine learning in the media industry, helping create a more personal BBC that can inform, educate and entertain in new ways.

Cenatus built a survey application that will be used to evaluate participants viewing / reading habits within the BBC platform against their perceived behaviour, in a collaborative project with the University of Manchester.

Decompression unravels in the spirit of a ‘happening’ – an encounter between audiences, space and the elements of cinematic abstraction, explored via the artists’ longstanding interest in expanded cinema and sound system culture.

Decompression brings scope to the examination of cinematic conditions by introducing live coding resulting in electronic sound and LED light compositions, and improvised direct to camera manipulations of frame rate interference, eliciting stroboscopic and sonic states. Threaded throughout the performance is an examination of the original text Expanded Cinema by Gene Youngblood (1970), which re-purposed, poses questions concerning contemporary technological-utopia, pressing the issue of the viewing experience and mass entertainment into relevance with regards to today’s globally connected audiences. The performance also features a ‘prepared’ light sensitive screen, tactile projection beams, and surround sound.

Decompression considers the point at which contemporary expanded cinema meets algorithmic event. An ongoing point of enquiry, the live audiovisual composition seeks to highlight: The ‘dematerialisation’ of technology via spatial activation of the sound system, programmed LED lighting and manipulated projection beams; ‘deterritorialization’ touching on the audience’s potential role as participant; and ‘detemporalization’ in which the concept of time-based media is instead driven by generative software processes.

The performance explores the frequency of light and sound as both reductive and complex - resulting in a sensory and hypnotic live set which challenges the viewer’s expectation of auditory and visual perception.

Materials: 4 x Raspberry Pi LED wall mounted lights, live coding, web camera, Strobotac, library sound FX, various synthesisers, data projector, haze machine.

Cenatus worked on this project from concept through technical implementation and performance.

• Project page
• Github repo

Photo by Dee Chaneva for Tyneside Cinema

Unconscious Archives Festival took place between 20th - 30th September 2017 over four unique shows. Exploring the relationship between sound, art, and performance.

Four unique events make up the first Unconscious Archives Festival, from experimental live film events at Shoreditch’s intimate Close-Up Film Centre, to electronic explorations focused on the hand built and coded at beloved South London club Corsica Studios, it’s an expansive programme in place across the ten days, brought to you by UA founder; artist-curator Sally Golding. Breakout artists merge with established names from the noise / art circuits to allow for discovery, with a large focus on the boundaries of audience / performer interaction and DIY music culture. UA2017 also features a special focus on the Austrian experimental music and art scene, operating in partnership with the Austrian Cultural Forum London.

Cenatus worked as associate producers on the festival and also provided the website and technical support.

Image: Myriam Bleau - Soft Revolvers, photo by Severin Smith

The Cyanometer is both a monument and software that periodically collects images of the sky. The monolith gathers data about the blueness of the sky and the quality of air and visualises them, thus becoming an instrument that raises awareness of the quality of one of the crucial elements of life. Together with air quality data, the Cyanometer website is creating a special kind of online archive and retrospective calendar that is measuring and documenting the changes to our environment.

Cenatus worked on concepts for the project, designed and built the online infrastructure to store and visualise the collected data. Each installation communicates with an API to store data and two web components visualise that data via the main website and an archive.

Github:

• msp/cyanometer
• msp/cyanometer-archive

Cenatus worked in collaboration with artist Sally Golding, creating concepts and realising technological solutions for dual installations at the South London Gallery. Both works had generative properties in the evolution of the sonic and visual components and included a simple interface for artist to configure timelines of events..

Github:

• msp/generative-projections-slg
• msp/generative-lights-slg

Cenatus headed up engineering during 2015-16, changing the culture to be one of close collaboration between product & technology, ensuring focus on regular delivery working towards the business KPIs. We created visibility of work streams, fostered accountability of tasks and allowed developers to work with clear focus. We owned the technical architecture and created a suite of repeatable automated tests to allow agility within the product & codebase. We developed and successfully delivered many new features into production that significantly enhanced the product.

A new immersive audiovisual performance, taking the audience on a hallucinogenic dark carnival ride exploring the slippage between parapsychology and technology.

Interweaving science and superstition, philosophy and pulp through sound composition, live 16mm film projection and light environments, Breaching Transmissions invites the audience to enter a hypnotic, sensory zone. Through themes of ‘transmission’ and ‘medium’, this live performance implicates the audience as a participant by harnessing abstract infrared camera images, and reconfiguring the typical performer-technology-audience relationship.

Cenatus worked on the project from concept to performance. We realised the technology required to create the desired atmosphere and created a complimentary audio narrative.

More info & video clips here.

Cenatus conducted a quality review of their back end and user facing web systems to ensure robust engineering practices that are covered by good integration tests. We worked on various enhancements and new features including integration with the OAuth authentication component, used by the front end website and VST software registration systems.

Primitives uses custom made software to explore sonic and optical intensity articulated by simple geometric figures and extreme frequencies. Projected images drive a sensory assault, consumed by your eyes, then ears and existing somewhere between perceptions.

Cenatus conceived, built and executed the project, touring the live performance and international events and festivals.

There’s more information for curators, including video clips here. There’s also a microsite for the project that was launched along with the vinyl/DVD release.

Productions, exhibitions and performances have included Melbourne International Film Festival (AU), BBC Research & Development (UK), The Roundhouse (UK), Club Transmediale (DE), South London Gallery (UK), Sound and Music (UK), Future Everything (UK), Mutek Festival (CA), Serralves Museum (PT), Cafe OTO (UK), Digital Culture Centre (MX), Contemporary Art Madrid/CA2M (ES), Unsound (PL), Museum of Transitory Art (SI), High Zero Festival (US), E-FEST (TN), Tyneside Cinema (UK) and SHAPE Platform (Sound, Heterogeneous Art and Performance in Europe).

Matt can be found on Github, Twitter, Last.fm and Linkedin and you can check out the latest projects here.

We promote new music and media, develop artists and enable wider public use of innovative digital technologies for creativity.

Cenatus produced Netaudio London, the UK’s foremost digital culture festival incorporating strands for Live Music, Sound Art, Conference and Broadcast that operated in 2006, 2008 and 2011.

Cenatus was originally set up as a not-for-profit membership organisation in 2005 and incorporated as a community interest company in October 2009 by Matt Spendlove and Andi Studer who lead the company as directors and were supported by a network of associates and staff. Cenatus grew into a creative technology company in 2012. You can read more history in the archive.

What would you like to hear? (insert current industry meme here). I make things, using technology. I have done since 2000.

Pragmatic, adaptive, self-organising creative technologist with an eye for elegant, simple solutions?

Value focussed?

Generalising specialist?

Interested in process and interaction?

Worked across industries and on research projects: web, art, music, finance, advertising? From start small ups to FTSE100 giants

C via Java & Ruby to Functional Programming, mostly Elixir/Phoenix these days and the usual HTML/CSS & Javascript niceties. Infrastructure is Git and cloud services like Heroku & Amazon, but I’ve had plenty of Linux/OSX experience before they existed?

I run a bunch of creative projects, including being a director of Cenatus and previously the Netaudio Festival. You probably don’t need to hear about that but might make me more fun to work with. Who knows?

That I sound far less arrogant in person?

I can be found on Github, Linkedin, Twitter and Last.fm. Sometimes. Here’s a very old CV for the more traditional amongst you.

The creative agency Dandelion & Burdock contracted Cenatus to implement a company website for the Brighton based arts organisation.

We implemented an image led responsive site, designed by Dandelion & Burdock and created a custom content management system. We transferred knowledge to the in-house team who now maintain the site.

• 24 March 2011: Michel Chion will present the world premiere of Live In Prose, A Symphony Concrète. This piece was composed in the period 2006-2010 and has been previously been only in extracts and drafts in Paris, Montreal Canada and Yokohama, Japan. The entire work is “for fixed sounds”. Michel Chion was born in 1947 in Creil (France). In the 70’s he was assistant to Pierre Schaeffer at the Paris’ Conservatoire national de musique, producer of broadcasts for the GRM, and publications director for the Ina-GRM, of which he was a member from 1971 to 1976. Parallel to these activities, he composed important musique concrète works in the studios of the GRM including the classic ‘Requiem’. 25 March 2011: Ghédalia Tazartès will present an idiosyncractic solo performance for voice and electronics. His public appearances remain exceptional events as he rarely performs in concert.

• The autodidact Ghédalia Tazartès, born 1947 in Paris has spent 30+ years within musical practice and experimentation, letting his musical work wander from chant to rhythm, from one voice to another. Utilising magnetic tape recorders, he paves the way for the electric and the vocal paths, between the muezzin psalmody and the screaming of a rocker. He traces vague landscapes where the mitre of the white clown, the plumes of the sorcerer, the helmet of a cop and Parisian an hydride collide into polyphonic ceremonies. Don’t become a black, an arab, a Tibetan monk, a jew, a woman or an animal but to feel all this stirring deep inside of you.” Additional acts to be announced.

Support on the night will be provided by the excellent Rashad Becker who “plays traditional music of imaginery species”.

Venue: Cafe OTO, 18 – 22 Ashwin Street, London E8 3DL

Tickets: £10 in advance / £18 for 2 day pass

Commissioned by Sound and Music and Cafe OTO.

Documentary video on the SAM/OTO series.

Cenatus worked with DiS on numerous occasions. The biggest project was the 10th anniversary redesign that included a rebuild & re-architecture of much of the user facing sections of the site. The last project was to implement a CDN to drastically reduce server costs.

More info can be found in the archive.

In 2008 Netaudio took over the spacious Shunt Lounge for 4 days. The programme included 8 audio-visual installations and 45 live performances with a total of 81 artists involved. The festival welcomed over 2700 visitors.

2009 appearances: Netaudio floor at Club Transmediale, Berlin, 27 January 09 Netaudio stage at Freerotation, Wales, 14-16 August 09 Netaudio London stage at Netaudio Berlin, 8-11 October 09

In 2010 we undertook an extensive R&D project.

2011 saw Netaudio taking over the Roundhouse Studios for Short Circuit with accompanying shows a KOKO, Cafe OTO and the Apiary Studios – check out the full programme