Machin-X: Digital Puppetry

Thoughts on Generative A.I., Gutenberg, Cameras and What Comes Next

2026-06-04T10:47:22.320-04:00

This is a concept trailer for A.I. filmmaker Rodrigo Soares' Remains: The Only Survivors. He describes it as film about "an empty world destroyed by machines...a dystopian cyberpunk future where only machines remain, and young hybrid/android girl survives by repairing broken machines. Whatever your feels about generative A.I., it's a very impressive piece of work.

And it raises the question, what happens when A.I. films become just as impressive - or even more impressive - than traditionally made ones? That's a question that Rodrigo himself seems to be wrestling with:

"Do you think a creator who uses A.I. can also be considered a real artist?

For me, the real challenge in A.I. filmmaking is not generating the most impressive frame. It is shaping those frames in to something meaningful. A.I. can open the door to incredible worlds, but the filmmaker still needs to decide what the audience should feel, where they should look, and why each image matters."

- Rodrigo Soares, Fableforge.ai

When Johannes Gutenberg introduced movable type and the printing press to Europe in the 1400s, the immediate reaction from academics and professional scribes was a mix of awe and absolute horror, not unlike the reaction to Generative A.I. today. Before Gutenberg, books were copied by hand. It was a sacred, slow, highly artistic process. Critics of the printing press argued that mass production would ruin the human mind and that the flood of new books flooded the world with "foolish, ignorant, malignant... and impious" texts.

They were complaining about, essentially, 15th-century print slop.

This happened again in the 19th century with the invention of the camera. Painters were horrified by cameras. The critic Charles Baudelaire famously called photography "art's most deadly enemy," fearing that a machine requiring "only a chemical process and a button push" would destroy the livelihood of true artists. But the camera didn't kill painting, just as the printing press didn't kill scholarship. Neither form was destroyed, but the goalposts shifted, from preservation to critical thinking and original synthesis.

In fine art, painters no longer had to act as human photocopiers documenting reality, they were forced to ask themselves what can a brush do that a lens cannot? The resulting existential crisis gave birth to Impressionism, Cubism, and virtually all modern art.

This is, of course, well worn ground in the ongoing debate over generative A.I. Likewise, The observation that A.I. doesn't automatically make you a great filmmaker — any more than a printing press automatically made you a great author — isn't a particularly new one. However, Rodrigo is suggesting that the goalposts in filmmaking are rapidly shifting from execution to curation.

When generative A.I. models produce work this impressive, we have to directly engage with the question: why do we need traditional forms like puppetry?

Rodrigo's argument illuminates something important: notwithstanding the complex legal and ethical issues raised by generative A.I., it is a technology that ultimately removes barriers and rewards people who actually know how to perform and direct. When he says 'the filmmaker still needs to decide what the audience should feel, where they should look, and why each image matters' - that's a description of what puppeteers and other artists already do. Generative A.I. creates the possibility of doing it with fewer restrictions.

Personally, I think that just like filmmaking, generative A.I. is going to ultimately push puppetry - both traditional and digital - to exciting new places. The transition won't be smooth, and will likely be more than a little terrifying...but it could also be terribly exciting.

How OK Go Used Blender As a Puppet Stage

2026-05-28T00:04:00.000-04:00

One of my favourite music videos of the past year is OK Go's Impulse Purchase, but it's not just an animated music video. It's a small, but important demonstration of how Blender can be used as a live puppet stage.

OK Go frontman Damian Kulash described the origin of his collaboration on this with directors Lucas Zanotto and Will Anderson at Blender Studio to Live Music Blog:

“It began with my love of Lucas Zanotto’s short animated loops — they’re so inventive, so full of joy, always delivering little doses of the kind of wonder we’re always searching for in our own videos. So I thought the universe flowing from his brain might be the perfect setting for a lyric video, but when I reached out to him, he had a more ambitious project to propose: a ‘live performance’ with me AS one of his characters. Suddenly it was a much weirder, more wonderful project than what I’d envisioned,”

To make the video, Damian performed the character via a live facial motion-capture app on an iPhone. The app used OSC (the Open Sound Control protocol) to send his real-time eye, lip, and head movements into Blender, where they drove the character’s erratic facial expressions on the fly.

What makes this true digital puppetry, not just digital animation, is that Damian isn't simply supplying reference material for an animator to interpret; he is performing the character through a real-time control system. His face is the input device. Blender is the puppet stage.

Best of all, in keeping with Blender Studio's open source filmmaking ethos, a demo version of this set-up has been released for anyone to try:

To run the free demo you will need to download the following:

Blender 4.5 LTS (the latest long-term support release, not the newest version of Blender)
The .blend demo file
Blender Studio's bespoke Foscap add-on
The Live Link Face app for iOS or Android

The full .blend production file used to make Impulse Purchase is also available for Blender Studio subscribers. Complete step-by-step instructions can be found on the Blender Studio Blog.

Gadzooks Studio Shows Why Performance Beats Prompting

2026-05-26T15:19:45.173-04:00

Gadzooks Studio, a Manchester-based stop motion animation studio founded by animation directors Steve Boot and Haydn Secker, has been sharing some really delightful digital puppetry experiments on Instagram. They’re building a Blender-based pipeline that utilizes TouchDesigner, which lets artists and technologists build interactive animation systems without writing traditional code.

What they've shared so far is pretty impressive:

View this post on Instagram

A post shared by Gadzooks Studio (@gadzooks.studio)

What Gadzooks is doing here is using TouchDesigner to feed their live performance in to Blender, which acts as the character/animation environment. This allows them to create digital puppets that respond to a performer in real time: facial expressions, head turns, timing, reactions, and little bits of character business.

TouchDesigner, created by Toronto-based Derivative, is a big part of Gadzooks’ not-so-secret sauce (although, to their credit, they’ve been sharing much of their workflow on Instagram). TouchDesigner is able to take a variety of live inputs - everything from camera footage to audio, MIDI, sensors, controllers, or just about anything else that outputs a signal - and use that input to control a character in real time without traditional hand-keyed animation. It’s a more sophisticated version of what I was trying to build with Panda Puppet many years ago - a way to make it possible to use almost anything to perform a digital character.

It’s a really great example of the advantage digital performance has over AI-based prompting: the immediacy of puppetry, with the flexibility and control of a digital animation pipeline. The best of both worlds.

I'm excited to see what others might be able to do with TouchDesigner, as well as what else Gadzooks is working on. To keep up-to-date with them, follow Gadzooks.Studio on Instagram!

Inside the Digital Puppetry of Jim Henson’s Puppet Up!

2026-05-22T10:22:00.000-04:00

The Henson Digital Puppetry Studio is a digital puppetry platform the Jim Henson Company has been building for decades, with roots that go back to Waldo C. Graphic in the late 1980s. The technology has come a long way since then, and in recent years has become good enough that the Jim Henson Company regularly uses it in Puppet Up!, their touring live improvised puppet comedy show.

In this video from Tested, Henson puppeteers Sarah Oh and Dan Garza show host Adam Savage how they use their puppeteering and improv skills to blend a digital character seamlessly - and hilariously - with a group of practical puppets in a single performance before a live audience.

What makes this so impressive, of course, isn’t just the technology. It’s that Oh and Garza are skilled puppeteers doing what skilled puppeteers have always done: listening, reacting, finding the joke, and creating the illusion of life where it doesn't actually exist at all.

The octopus may be digital, but the performance is remarkable because it's unmistakably human.

If you want to see the Henson Digital Puppetry Studio in action yourself, Puppet Up! will be playing the Montalbán Theatre in Los Angeles this summer. Visit PuppetUp.com for upcoming tour dates and full details.

The Magic Behind ILM's Real-Time Rocket Is the Performer

2025-11-14T18:25:00.000-05:00

ILM's Christopher Jones (left) and creative director Michael Koperwas with Real-Time Rocket in the background (source: Industrial Light and Magic).

Industrial Light and Magic made a splash last year with Real-Time Rocket, an immersive digital puppetry experience for fans of Marvel Studios that was featured at Disney's D23 Expo.

Marvel fans at D23 were able to directly interact with Rocket Raccoon from Marvel's Guardians of the Galaxy, thanks to a collaboration between Marvel Studios, Disney’s StudioLab, and ILM, that enabled improv actors to perform Rocket using markerless motion capture technology.

ILM showcased an updated version of the activation at the Disney Accelerator Demo Day earlier this month:

Real-Time Rocket won a VES Award for Outstanding Visual Effects in a Special Venue Project, which is impressive, although having live audiences interact with a digital character is far from a new idea. Real-time digital characters have been featured in attractions like Turtle Talk With Crush at the Disney theme parks for over twenty years.

The specific challenge for ILM here was that due to the inherent limitations of markerless motion capture, they had to engineer a way to take a relatively limited amount of performance data from a performer and translate that into natural, biologically accurate poses appropriate for Rocket's raccoon anatomy.

@ign Rocket doesn’t know what IGN is, but you can interact with Rocket at the TVA at D23! #disney #d23 #marvel #marvelstudios #guardians #rocketraccoon #gotg #guardiansofthegalaxy #tva #loki #ign #mcu #movie ♬ original sound - IGN Entertainment

Amidst all the hype about employing Unreal Engine, markerless motion capture and the use of a LIMINAL 3D display, it's easy to forget that while the technology behind Real-Time Rocket is remarkable, what audiences responded to so strongly has very little to do with a technology pipeline, and everything to do with the performers using it.

Real-Time Rocket really works because of the classic Disney character performer training behind the scenes. Actors were trained extensively on Marvel lore, Rocket's speech patterns, and improvising with a live audience. The performers had a two-way video feed, which enabled them to spot specific people in the crowd to react to. That is just the latest iteration of what puppeteers have been doing digitally in various forms going all the way back to Jim Henson and Waldo C. Graphic in the 1980s.

So the actual magic at work here isn't digital - it's human.

Sergei Obraztsov famously said that his hand was the soul of his puppet. Here, the performer provides the character's soul. ILM's digital pipeline is, ultimately, just the puppet.

Experimental Digital Shadow Puppets in Blender

2024-04-08T13:45:00.000-04:00

Lyle at LJ Puppetry recently shared this digital puppetry experiment. Lyle was inspired by the Henson Digital Puppetry Studio and its use of a Waldo, but wanted to see if he could create digital shadow puppets that were controlled in real-time via motion capture inside Blender. Lyle himself admits that his results were mixed, but I think his project may indirectly suggest some really interesting technological possibilities.

Of course, experimenting with digital shadow puppets is a well-trodden path. I began my own journey in to real-time animation experimenting with digital shadow puppets when I first started writing Machin-X, although back then we had to rely on colour blob tracking and custom scripting in Adobe Director because there were no off the shelf tools.

Today, it's much, much easier to do this with off-shelf open source software like Blender, which is what Lyle used to perform this multi-scene shadow story - a magi, a tiger that transforms into a woman, a figure that coalesces from stars - entirely through hand and finger movements captured by a webcam:

The shadow characters in the video are flat black figures built using Blender's grease pencil tool. Lyle controlled them in real time by mapping specific finger positions to specific puppet movements — thumb open stands the magi up, a closed fist brings him to his knees. What we're seeing in the video isn't playback; it's a live performance into Blender, with the puppeteer off-camera and the output on screen.

Although his experiment wasn't entirely successful, Lyle's documentation is an excellent starting point for someone else who wants to try experimenting with this on their own. You can read an in-depth write-up of his workflow here.

This might be the right approach with the wrong tool

I've been following motion capture developments in Blender for a long time. I was very excited by the BlendArMocap plugin when I first heard about it, but I've since discovered that it has some real limitations. Dropped frames (the "jerky mocap" that Lyle mentions) and the ability to only perform one puppet at a time are two of the known issues.

Personally, I suspect that BlendArMocap might be the wrong implementation of the right technology. It's notoriously difficult to manage motion tracking inside Blender itself (which is what BlendArMocap does), but the underlying technology that makes that markerless motion tracking on a standard webcam possible, Google's open source Mediapipe framework, is brilliant.

TouchDesigner's node-based interface can look intimidating, but it's a popular tool used by artists all over the world.

Mediapipe offers real possibilities for digital puppetry. The ability to use markerless motion capture - especially in a tool like Blender's innovative Grease Pencil - is tantalizing, but a better way of utilizing it might be with a program like TouchDesigner. TouchDesigner is used by artists and performers to process, and react to live data — from motion capture to audio to video — in real time. If you've seen a live concert where the visuals pulse and shift in sync with the music, that was likely done with TouchDesigner, and it can also send clean motion data directly into Blender in real time, via OSC.

Frankly, it's much easier to fix problems like "jerky mocap" in TouchDesigner before the data ever reaches Blender, which in turn creates a much more streamlined workflow.

We need more experiments like this

I love what Lyle has shared here, and that he is encouraging others to learn from his project. He's given me some things to think about, and I'm very excited to see what others do with Blender and tools like TouchDesigner and Mediapipe.

More than anything though, this experiment serves to remind us that creating great puppetry is hard, creating digital puppetry is especially hard...and recreating shadow puppetry in particular is deceptively hard.

Digital Wayang Kulit

2015-04-13T20:30:00.001-04:00

Here's a nine minute demo of a Digital Wayang Kulit (Indonesian shadow puppetry) program developed at the MSc in Digital Education program at University of Edinburgh. The 2D figure is controlled by the digital Dalang (puppeteer) using a Gametrak controller and a Wiimote.

There have been a lot of shadow puppet inspired digital puppetry demos created over the years (this one is from 2013), but I love how fluid the movement of this one is.

Special thanks to Jane for submitting this!

Character Animator - A 2D Real-Time Animation tool from Adobe

2015-04-12T22:34:00.001-04:00

Adobe has just unveiled a new 2D digital puppetry - or have we all agreed to call this field real-time animation now? - application called Character Animator. A demo of the software is provided in the video above, but essentially it's a tool for animating 2D bitmap (Photoshop) and vector (Illustrator) still characters in real-time using a camera, microphone, head tracking and facial mo-cap.

In addition to basic mo-cap and lip sync capabilities, it also allows users to create programmable behaviors and will support the creation of 3rd party plug ins. I haven't tried it myself yet, but it looks like it could be a very easy-to-use and potentially powerful tool for creating basic 2D animated characters in real-time.

Adobe Character Animator is currently in beta and available for testing by After Effects CC users. You can learn more here.

Via The Labyrinth.

Unbelievably impressive demo of the Unreal Engine 4

2015-03-20T00:20:00.000-04:00

It's been far, far too long since I posted an update here on Machin-X, but I may be (finally) returning to some digital puppetry work in the near future and when I saw this demo for the Unreal Engine 4 I had to share it.

It is, well, pretty unreal:

The Kite open world demo created in Unreal Engine 4 features a diverse and beautifully realized 100 square mile landscape. Everything is generated completely in real-time at 30fps and includes fully dynamic direct and indirect illumination, cinematic depth of field and motion blur, and procedurally placed trees and foliage.

Real-time 3D sure has come a long way since I first worked on a TV pilot for a proposed kids' series using cardboard cut-outs and blob tracking with a webcam to create some 2D flash animation almost a decade ago.

I wonder where this technology will go in the ten years or so?

Henson Digital Puppetry Studio

2014-01-28T15:03:00.000-05:00

This is a brand new promotional reel for the Henson Digital Puppetry Studio, the patented real-time animation/digital puppetry system developed by Jim Henson's Creature Shop.

2013-09-09T03:13:00.002-04:00

This is a simple, but nonetheless very effective example of 2D digital puppetry created using Unity and a Kinect.

Animatic Digital Puppetry System

2013-07-03T17:05:00.000-04:00

A look at "Animatic", a digital puppetry system that was developed by Luis Leite (see previous post) using 3D Studio Max and Macromedia Director in 2006. The system was developed as part of his research thesis Marionetas Virtuais.

Digital Puppeteer Mario Mey

2013-05-23T10:09:00.000-04:00

This is a new demo reel for Argentinian digital puppeteer Mario Mey that shows off his digital characters performing en Español at various live events (his character Pinokio 3D was mentioned here back in 2010) . He creates and performs his "Marionetas Digitales" (digital puppet) characters using Blender 3D and PureData, a real-time graphical dataflow programming environment for audio, video, and graphics.

You can see Mario at work and get a look at his production process in this video, however it was recorded in Spanish.

Faceshift Markless Motion Capture

2013-05-16T14:30:00.000-04:00

Faceshift is software that promises "markless motion capture at every desk". It works with consumer-level cameras like the Kinect to track and analyze the facial expressions of a performer and uses them to animate a virtual character in real-time. It also offers the option of recording a performance so that it can be edited and polished in post-production.

There are lots of potential applications for this kind of software in game and film production and, of course, digital puppetry applications!

You can learn more at www.faceshift.com.

Hakanaï: Dancing with Digital Puppetry

2013-05-14T09:46:00.000-04:00

Hakanaï is one of the more unconventional examples of a digital puppetry performance I've discovered (although, is there anything truly "conventional" about any form of digital puppetry?). Its creators describe it as a "haiku dance performance taking place in a cube of moving images projected live by a digital performer".

The performance involves a dancer performing live, whose movements are tracked in real-time and used as the basis for an interactive, digitally animated environment that is projected around them:

It was created by the French Company Adrien M / Claire B using their proprietary software eMotion. Here's more from their description of the project:

...Performed by an artist as a “digital score”, it is generated and interpreted live. The dancer’s body enters into a dialogue with the moving images in motion. These simple and abstract black and white shapes behave according to physical rules that the senses recognise and to mathematical models created from the observation of nature.

The audience experiences the performance in several stages. They first discover the exterior of the installation. As the dancer arrives, they gather around to watch the performance. When the choreography has ended, the audience can then take some time to wander amongst the moving images.

Through a minimalist transposition, this piece is based on images drawn from the imaginary realm of dreams, their structure and their substance. The box in turns represents: the bedroom where, once the barrier of sleep is passed, walls dissolve and a whole new inner space unfolds; the cage, of which one must relentlessly test the limits; the radical otherness, as a place of combat with an intangible enemy; the space where impossible has become possible, where all the physical points of reference and certitudes have been shaken.

Through the encounter of gesture and image, two worlds intertwine. The synchronicity between the real and the virtual dissolves and the boundary that was keeping them separate disappears, forming a unique space filled with a high oneiric charge.

Very cool, no? You can learn more from the video's description on Vimeo.

Activision unveils impressive real-time character demo

2013-03-31T01:35:00.000-04:00

Activision unveiled some new real-time rendering technology for human characters at the Game Developers Conference last week.This is the result of several years of research in to creating photo realistic human characters for video games. Although the animation itself a bit off and suffers from the infamous "Uncanny Valley" effect, just on a purely technical level this is pretty impressive.

From the video's description on YouTube:

This animated character is being rendered in real-time on current video card hardware, using standard bone animation. The rendering techniques, as well as the animation pipeline are being presented at GDC 2013, "Next Generation Character Rendering" on March 27.

The original high resolution data was acquired from Light Stage Facial Scanning and Performance Capture by USC Institute for Creative Technologies, then converted to a 70 bones rig, while preserving the high frequency detail in diffuse, normal and displacement composite maps.

It is being rendered in a DirectX11 environment, using advanced techniques to faithfully represent the character's skin and eyes.

More technical details can be found here.

Via Cartoon Brew.

A digital dragon puppet

2013-03-27T18:30:00.002-04:00

A nice example of a digital shadow puppet, made by Luis Leite using Kinect and Unity 3D. To animate the puppet, a human body is tracked in real-time using the Kinect sensor, with one hand controlling the head and the other controlling the tail. The physical movement of the performer's body is remapped on to the virtual shadow puppet using Inverse Kinematics via Unity's Mecanim animation system.

Luis was also responsible for a Kinect-based digital puppet that was mentioned in a post about Kinect-based digital puppetry on Machin-X two years ago.

Digital Puppetry with the PS4

2013-02-22T16:20:00.000-05:00

Earlier this week game development studio Media Molecule gave a R&D presentation at the launch event for Sony's new PlayStation 4 (PS4) video game console, which appears to have some amazing new capabilities like the ability to sculpt, create and animate in real time that offer phenomenal potential for digital puppetry applications. Media Molecule won't too much about what they're working on (yet), but their demo utilizing the PS4 and the often-derided PlayStation Move controller looks amazing (skip ahead to the 5:15 mark to see all the digital puppetry goodness).

Very exciting!

Via Puppeteers Unite.

Karagoz goes digital

2013-01-30T01:21:00.000-05:00

Indonesian shadow puppetry has gone digital, so why not Turkish shadow puppetry too?

iKaragoz is an app for iPhone and Android developed by a Turkish firm called Anakule. They are promoting it as the first puppet application for mobile devices, but it's definitely not (several others including iPuppeteer and Pollock's Toy Theatre app have been on the market for years).

The app allows the user to control the characters onscreen intuitively by simply moving their smart phone or tablet.In addition to the traditional Turkish Karagoz puppets, additional packs with characters from Cambodian, Chinese, Indian, Indonesian, Thai and Greek puppetry traditions are also available.

Here's the Wayang Kulit version in action:

iKaragoz was designed by Uğur Doğan with the assistance of Turkish puppeteer Mehmet Saylan. It's available to download from the iTunes Store and Google Play. I haven't had a chance to try it out myself yet, but if someone does please let me know what you think!

Via Puppetry News.

Bryn Oh, Imogen and the pigeons

2013-01-25T16:56:00.001-05:00

Bryn Oh is the avatar and pseudonym of a professional oil painter who lives here in Toronto who has been creating mesmerizing and challenging multilayered installations inside the virtual world of Second life for several years (see previous post). I'm especially impressed by Bryn's latest work Imogen and the pigeons, an "immersive narrative exhibited in the virtual world called Second Life...a layered story told through poetry."

I find it difficult to classify work like this. Is it Machinima, immersive interactive art, digital puppetry, all of the above, or something else entirely? While I'm not entirely sure what the answer to that question is, I do know that I like it. A lot. It's inspiring to see the innovative ways that Bryn Oh is exploring and expanding how this still new medium can be used.

You can explore Imogen and the pigeons inside Second Life (click here and follow the instructions to join Second Life) and/or see more of Bryn Oh's previous Second Life builds on YouTube.

Digital Shadow Puppet Installation

2013-01-20T21:05:00.001-05:00

A nice look at an interactive, Kinect-based digital shadow puppet installation created by New York based motion designer Yang Yang.

Digital Guignol Theater

2013-01-10T00:06:00.000-05:00

I've been doing a lot of Punch and Judy research lately and I stumbled across this Digital Guignol Theater that was created by Wizarbox in 2009. A proof-of-concept demo intended to aid in children during rehabilitation, it allows them to control a digital Guignol (the French equivalent of Punch) show using their voice and one or more Wiimotes.

Digital Wayang Kulit Music Video

2012-12-05T03:17:00.002-05:00

We don't see a lot of music videos that feature digital puppetry, so when one comes along it's worth noting!

This video for Boshra Al Saadi's Snowyman was created using a modified version of Antonius Wiriadjaja's Java-based Wayang Kinect project. Hit the link to learn more about his work and download the digital Wayang Kulit applet so you can try it out yourself!

Flaming Skull Face Tracking Demo

2012-11-23T02:02:00.003-05:00

Some clever face tracking from a Spanish studio called Paradox D&D. They've written a custom application that positions a 3D skull over top of the user's head, capturing the user's body movements and using them to control a 3D model in real-time. This is an example of a "virtual dresser" that allows a user to "wear" 3D elements in a manner very similar to augmented reality.

Via KinectHacks.

Researchers Demo New Kinect-based 3D Puppetry System

2012-11-03T13:28:00.000-04:00

This video highlights a new Kinect-based 3D puppetry system developed by researchers at the University of Washington and the University of California, Berkeley that allows users to create 3D animation with minimal experience.

What's somewhat unique about this system is that unlike most Kinect-based systems, instead of the using body movement to control an onscreen character or object, users manipulate physical objects to control corresponding 3D models on a virtual set. I like this approach to digital puppetry a lot, because it retains the basic concept of puppetry (real-time manipulation of a physical object). To demonstrate how effective they believe this approach is, the researchers used novices in puppetry and animation to test the system.

Although the system these researchers have developed appears to be somewhat limited - note that the objects in the demo are solid objects without any kind of skeleton or use of Inverse Kinematics or Forward Kinematics - systems like this are very positive developments for digital puppetry. It would be great to see this get out of the lab and be developed in to some kind of open source platform.

A technical paper on this project can be found here.