Welcome back! In this part we will start sculpting a humanoid character using ZSpheres. ZSpheres is a very useful (and powerful) tool in that it offers you a super-fast and highly flexible way of getting starting the sculpting process (often the hardest part). This means that you don’t have to create a base mesh in a 3D-modeling package first, you can just hit the ground running. So, let’s get started!

ZSpheres

Open the ZBrush application and create a new document. Hit the SimpleBrush icon in the Tool menu, and then the ZSphere icon as shown in the picture below.

Click and drag with LMB to draw the ZSphere on the canvas. Your screen should look like the picture below. Hit T to enter edit mode. Forget hitting T now and you will draw a new sphere every time you click on the canvas. If your accidentally draw new spheres, hit CTRL+N to clear the canvas, LMB-click-and-drag a new sphere, hit T, and you are OK.

The red ball you have no drawn on the canvas is your root ZSphere, to which you now can add new ZSpheres (in a tree-like structure that branches out from the root). All you have to do is add new spheres, and adjust the position, scale, and rotation of each while looking at your shape from different angles. Your goal is to create a structure in this manner that resembles a basic foundation for the shape you have in mind, which you will use for starting the sculpting process.

Here are the actions you need to create a base mesh using ZSpheres:

• Add new sphere: Hit Q to draw sphere. LMB-click-and-drag on a location on the sphere where you want to add a new one. Hold SHIFT if you want the new sphere to have the same size as its parent.
• Delete sphere: Hit ALT+LMB.
• Move sphere: Hit W to move sphere.
• Scale sphere: Hit E to resize.
• Rotate sphere: Hit R to rotate.
• Enable symmetry: Hit X.
• Preview: Hit A to preview the result.

So, as I said the the beginning, we’re going to sculpt a humanoid character. The way to do that is to use ZSpheres to create a shape like the one on the left in the picture below, and then proceed to make ZBrush convert the ZSphere object into a polygon mesh for sculpting (shown to the right).

The process

To create a base mesh using ZSpheres, you can use the following steps and repeat until you are satisfied with how your preview looks:

1. Add a new sphere on an existing one (or create a root sphere).
2. Move, scale, or rotate this new sphere while looking at your mesh from all views. (Enable symmetry if needed.)

The picture below demonstrates the process used for creating a humanoid shape. (Remember to look at the object from all sides and not only the front view!)

OK, so you have followed the process through several iterations and are satisfied with the preview. Time to convert to polygons!
Navigate to Adaptive skin in the Tool menu and hit Make adaptive skin. The polygon mesh will now be added to your tool menu, named Skin_… Select it and start sculpting! :)

That concludes this part. Feel free to add comments or pictures of your works in the forums. Have fun sculpting! :)

Hi and welcome to this series on sculpting in Pixologic ZBrush. In this series, we will investigate some of the game-changing techniques that have made ZBrush so popular among digital artists.

First, a quick primer on what ZBrush is. ZBrush is a digital sculpting and painting application (Autodesk Mudbox is another software package that offers similar capabilities). 3D sculpting means that you work on “digital clay”. You start out with a clay-like basic shape, like a sphere, and then you use a set of brushes to “sculpt” that clay. What makes ZBrush so powerful is that you can work on millions of polygons. In short, 3D sculpting is a very efficient and fun way to create high-polygon 3D models and/or normal-maps.

OK, enough with the sales pitch: let’s get to it!

There are basically two ways to start sculpting in ZBrush. One: make a simple “base mesh” in an external 3D application (such as Maya). Two: start with a simple shape or ZSpheres. ZSpheres are very cool, and we will save them for a later tutorial. In this part of the series we will start by importing a base mesh from Maya. You can refer to this tutorial if you need a refresher on Maya.

Creating a base mesh

There are basically three criteria for the base mesh:

1. Keep it simple
2. Shape it into a very abstract representation of your design
3. Make sure the mesh is clean and quads only

So, you may either have some concept drawings or a mental image as reference for your design. Then, block out a simple mesh in Maya that resembles that design, using only a handful of polygons. Now, this is very important: keep it clean and all quads. You will face quirky problems in ZBrush if your mesh does not satisfy this condition. All quads means that each polygon should have four edges (this has to do with the subdivison algorithms in ZBrush). Also: the mesh should not contain any holes, gaps, overlapping vertices or polygons with flipped normals. A good reference might be the section on cleaning up the mesh in my Maya tutorial series.

In the image below, I have created a base mesh for “Headmistress Daggerhorn”, which is one of the key characters featured in the game Evil Junior which I’m currently working on. It was created on top of great concept art from Atle Mæland as the reference.

Create your own base mesh, and then export your base mesh as .OBJ in Maya. Brace for impact in ZBrush!

Intro to ZBrush

Time to open ZBrush. Hit “Import” in the tool menu and browse to your .OBJ base-mesh. “Draw” your base mesh onto the canvas by click-dragging in the viewport. Then hit “T” to edit the object.

Now, a brief heads-up on the ZBrush interface (see the picture below). To your right you will find the “Tool” and “Geometry” menu. Current brushes and materials are located to the left. Here is what you need to know:

• Tool: Here you “Import” your base mesh. You save your work as ZTools by hitting “Save As”. Saved ZTools are loaded with – surprise! – “Load Tool”.
• SubTool: You can also import several sub tools and append them to your main tool, so that you can more easily hide and select individual objects on screen.
• Geometry: ZBrush “subdivides” your mesh when you hit “Divide”. Each division operation adds resolution to your mesh, measured as the number of “active points” in the HUD of the viewport. You can easily jump between subdivision levels by hitting “Lower Res” and “Higher Res”.
• Brush and material: Here you can edit your current settings for brushes and materials.

ZBrush both looks and feels like something out of this world. (You will probably either love or hate the interface.) I would highly recommend you to use a pressure-sensitive pen and tablet for sculpting in ZBrush. Here is how to navigate:

• Sculpt: LMB onto the mesh.
• Smooth: SHIFT+LMB onto the mesh.
• Quick-menu: RMB.
• Move: ALT+LMB in the background.
• Rotate: LMB in the background.
• Zoom: ALT+LMB in the background, then release ALT and hold LMB.
• Snap rotation into orthogonal view: Rotate and then hold SHIFT.

Here are a few hints to kickstart the sculpting process:

• Symmetry. Hit “X” to enable automatic mirroring.
• LazyMouse: Hit Stroke -> LazyMouse to enable computer-aided stability to your strokes. Hands like a surgeon!
• Brushes: Some brushes you may find useful are the Standard, Smooth, Move, mPolish, and Slash brushes.
• Strokes and alphas: Try experimenting with these to customize your brushes. You can add high-res details by painting an alpha in Photoshop and then apply it to your mesh using the Drag Rect stroke.
• Material: Try using other materials other than the very “ZBrushish” MatCap Red. All the MatCap materials are good. Switch between them to apply different effects to your mesh.
  
• Iterate: This is perhaps the most important thing in ZBrush. Only subdivide your mesh when you have maxed-out the current level of resolution. Iterate and add more detail for each subdivision level.
• Reference: The use of reference (anatomy guides, bodybuilding magazines, images on the web) can make all the difference. Study the anatomy of bones and muscle if you find the time. Be aware of nudity though.
• Masking: Hit CTRL+LMB or ALT+LMB onto the mesh to paint masks. Very powerful for adding details. You have more options under the “Masking” menu.

Basic sculpting in ZBrush

The sculpting process in ZBrush is actually very simple. It all boils down to iteration. In the beginning of the sculpting process, try using the Move brush to drag the mesh into the shape you want. You can also ALT+LMB with this brush to push or pull. When you are satisfied with the overall shape, use the Standard brush with LMB and ALT+LMB to paint detail at the surface. Use the Smooth brush in between to make the surface nice and clean. (You may want to dial down the brush modifier on the Smooth brush to make it less sensitive.) When you are happy with your sculpting at this level, subdivide and repeat the sculpting process. You can keep on subdividing like this till you drop – or more probably – you run out of RAM. Two million polygons should suffice – any more than that and your RAM will probably choke. As I gradually add more and more resolution for each iteration, I tend to use the brushes in the following manner:

• Move: I use this mostly at low resolutions to pull and push the clay into shape.
• Standard: This brush I tend to use at all resolutions, gradually decreasing the brush size for each subdivision.
• Smooth: I use this ALL the time.
• mPolish: Great to use at higher resolutions to flatten out a surface. Especially if you aim for a cartoony look.
• Slash: At very high resolutions you can use this brush to add scars and such.
• Pinch: Very nice for pulling two edges towards each other, for example if you want to “tighten” a scar.

In the picture below you can see how Daggerhorn takes shape from a simple base mesh to a high-ploygon bust of the Headmistress of Gloomcraft Academy.

That’s it

Piece of cake. Now, post an image of your sculpt in the gallery and stay tuned for the next installment! See you around :)

Oh, and one last thing: Here is an all-exclusive and yet unreleased work-in-progress of Headmistress Daggerhorn from Evil Junior!

Getting started

So you have a demo in JS. It’s pretty. It’s perfect. It’s 3k in a 1k competition. Well, damn.

You start with the obvious and run it through a minifier and you shorten all your variable names to a single character, you get it to 2500 bytes. Great, that’s progress, but you’ve still got 1476 bytes to go. You merge some functions together, fold a couple loops into each other, and soon you’re at 2200 bytes. Long way to go.

Rule #1 of shrinking demos: removing a byte becomes more difficult every time you remove a byte. It starts off trivial and runs very quickly into a brick wall.

Quick wins

These are a couple useful techniques to trim the fat at this point.

var a = g.getAttribLocation(x, 'pos');
g.uniform1f(g.getUniformLocation(x, 'time'), t);
g.vertexAttribPointer(a, 2, g.FLOAT, 0, 0, 0);
g.enableVertexAttribArray(a);

What’s wrong with this picture? Well, if we ignore whitespace (minification takes care of that): we’re using var (pfft, correctness), we have a shader attribute with a 3-byte name and a uniform with a 4-byte name, we’re using g.FLOAT (more on this in a moment), and we’ve got a lot of zeros.

g.vertexAttribPointer(
    a = g.getAttribLocation(x, 'p'),
    2,
    5126,
    g.uniform1f(g.getUniformLocation(x, 't'), z),
    0,
    0
)

Much better! We killed var removing 4 bytes, we removed 2 bytes by inlining the assignment to a in the first use and killing the semicolon, we eliminated a zero (the return of uniform1f is treated as a zero by the WebGL API in this case) and a semicolon for a savings of 2 bytes, we changed our shader attributes/uniforms to have 1-byte names (saving 5 bytes), and we inlined the FLOAT constant for a savings of 3 bytes. That’s 16 bytes removed from a block of 151 (after whitespace removal), or a reduction by 10.59%! Let’s do it again.

Before we move on, I just want to make a note that WebGL constants really are just that. You can inline them and trust them not to change.

Abusing globals

Look at the latest code above. Count the instances of ‘g.’. Four instances just in that tiny little snippet. If we could get rid of those in a small way, this could be a huge win, so let’s do it.

What is the global namespace in JS? Normally, it comes from the window object. So when you call foo(), it’s going to look in window if you haven’t declared it locally. So why don’t we shove our methods into the window object? Or even better, into top (in our case, they’re identical)?

Right now the declaration of g is as such:

g = z.getContext('experimental-webgl')

But let’s change it to:

for(k in g = z.getContext('experimental-webgl'))
    top[k] = g[k].bind && g[k].bind(g);

So we walk over every key in the WebGL context and put it into top, but only if it has the bind method and we can bind it to the WebGL context. Without that, it tries to treat the window as a WebGL context. The window doesn’t like this.

At this point, we just cut all instances of ‘g.’ out of the picture. This was a win of around 30 bytes (counting the cost of the globalization code) in my demo’s case, but we can go much, much further.

What’s in a name?

Let’s look at a list of all the WebGL methods I called in my demo:

attachShader
createProgram
linkProgram
createBuffer
bindBuffer
bufferData
viewport
createShader
shaderSource
compileShader
useProgram
getAttribLocation
getUniformLocation
uniform1f
uniform2f
vertexAttribPointer
enableVertexAttribArray
drawArrays

Holy moly, that’s a lot of bytes! But these are defined in WebGL, how can we change this?

Welllll… let’s look back at our globalization code. We get a name, k, and then bind g[k] intotop[k]. But we control what k goes into top. Having a map of long name into short name would be expensive, but what about a regex? I’m going to spare you the gory details, but after a while of tinkering with it, I determined that the optimal code for this is:

t[k.slice(1, -5).replace(/[ntalruicoh]/ig, '')] = t[k] = g[k].bind && g[k].bind(g);

In this case, t is top (because hey, 4 bytes!). We still assign the original name into t, but we also put the sliced and diced name into it. Why? Because this mangles two names we need to disambiguate: uniform1f and uniform2f. But that’s fine.

So what do the names look like after this?

S
eeP
kP
eeB
dB
ffe
e
eeS
deS
mpeS
seP
eb
efm
f
f
eexbP
beVeexb
w

Night and day difference, as you can see. You’ll obviously want to change the regex if you do this yourself, since your code will have different balances.

Other random wins

Arrays of digits tend to be pretty wasteful:

new Float32Array([0,0,2,0,0,2,2,0,2,2,0,2])

What about this instead?

new Float32Array('002002202202'.split(''))

The more digits you have, the bigger the win from this replacement is. Until we get to compression and everything changes.

Compression

Ok, we all know HTTP requests are trivially compressible, and this can improve load times and blah blah blah. Doesn’t matter when it comes to demos: if the size on the wire is 1k and the size on disk is 3k, your demo is 3k.

In the past, people have compressed their demos into images. This is an easy way to get a good size reduction — PNGs are just zlib’d data with a little header, basically. But these demos all had a PNG and an HTML+JS file that would load the PNG, draw it to a canvas, pull the pixels out of the canvas as a string, and eval that string. How can you make this into a single file demo?

Introduction to PNGs

The PNG file has an 8-byte signature followed by a series of simple chunks. The chunk format is as follows:

4 byte length
4 byte chunk type
<length> byte chunk data
4 byte CRC

Most of these are pretty clear, except chunk type. Chunk type is a FourCC, e.g. ‘IHDR’, which is mostly generic, with some exceptions which I’ll talk about shortly.

I’m not going to go into detail on the IHDR chunk (we can’t really mess with it, but I will say I use greyscale for simplicity purposes), but here’s what we start with.

8 byte signature
13 byte IHDR with 12 byte chunk header
X byte IDAT (covered below) with a 12 byte chunk header

The IDAT format is dead simple: 1 byte filtering method followed by your zlib deflated data.

Abusing PNGs

We start by defining our own chunk type (we’re allowed to do that!) before the IDAT chunk.

4 byte length
4 byte "jawh" (Just Another WebGL Hacker (TM))
X byte bootstrap
4 byte CRC

What is the bootstrap? Well, it’s what turns our PNG into code and runs it. Here’s the one I use:

<img onload=with(document.createElement('canvas'))p=width=4968,(c=getContext('2d')).drawImage(this,e='',0);
while(p)e+=String.fromCharCode(c.getImageData(0,0,p,1).data[p-=4]);
(t=top).eval(e) src=#>

The 4968 here is really the size of the decompressed data in bytes times 4 — there are 4 components to each pixel (red, green, blue, alpha) but we’re only using grayscale, so we need to offset that. If you look carefully, you’ll also notice that it walks backwards across the data, which should create a string in reverse, but I compensate for that when I compress the data, and reverse it before doing so. This saves a couple bytes. The image source is also important: rather than hard-coding the image filename and wasting space, it uses #, enabling it to treat itself as a PNG.

So what do we have now? We have a PNG containing some HTML. The browser first opens it as HTML (file extension is important here), then sniffs the MIME type and figures out it’s a PNG when it gets loaded into the image tag. We have a self-extracting PNG. But we can do better.

Why specs don’t matter

We’re currently using a chunk type of “jawh”. Clever and a nice little insider reference, but that’s 4 wasted bytes! The chunk type comes right before the data, so why don’t we make the chunk type into <img instead?

The spec says on the subject:

Four bits of the type code, namely bit 5 (value 32) of each byte, are used
to convey chunk properties. This choice means that a human can read off the
assigned properties according to whether each letter of the type code is
uppercase (bit 5 is 0) or lowercase (bit 5 is 1). However, decoders should
test the properties of an unknown chunk by numerically testing the specified
bits; testing whether a character is uppercase or lowercase is inefficient,
and even incorrect if a locale-specific case definition is used.

It then goes on to tell you that bit 5 of each of the bytes is:

First byte: 0 = critical, 1 = ancillary
Second byte: 0 = public, 1 = private
Third byte: reserved
Fourth byte: 0 = unsafe to copy, 1 = safe to copy

Since we could call it <iMg or <imG or any other combination of uppercase and lowercase, clearly we only care about the impact of < on the first byte. But if we look at bit 5 of <, we see that it’s already set high, so it’s an ancillary chunk — we’re in the clear and just saved 4 bytes!

Wait, no, neither Chrome nor Firefox load the image anymore. What the hell? This is the point where you realize: the spec doesn’t matter. No browser out there follows the PNG spec, whatsoever. Perfectly to-the-spec images won’t work, and horribly broken images will work. So let’s horribly break it. Note: Both Chrome and Firefox use libpng for image parsing, so you could go through the code and look for ways in which it’s mishandling data and go down that path, but for my purposes I found that just experimentally changing things and seeing how they break was Good Enough (TM).

Breaking PNG for fun and profit

Messing with the chunk type is out due to browser incompatibility, so what other options do we have? Well, who cares about CRCs.

Set the CRC to c=#>, save, refresh. Hey, it works in Chrome and Firefox! 4 bytes saved.

Well, we can’t push back towards the front due to the chunk type, and we can’t push forwards towards the IDAT chunk because of the size. But how do browsers handle chunk size mismatches?

If we set the size of the IDAT atom to c=#>, that unpacks to a size of 1042496867, which is obviously more than the size of the IDAT chunk. We shift ) sr into the CRC and set the IDAT size to c=#>, save, refresh. All browsers will set size = min(reportedSize, endOfFile – startOfChunk)! That saves us another 4 bytes.

So sure, we couldn’t get rid of jawh, but who cares? We just saved 8 bytes!

Compression caveats

Now you know how to make the PNG container small, but this says nothing about actually making your demo small. A few of the tips above are actually harmful in the context of compression. Why? Because repetitive data compresses really, really well. For instance, the Float32Array trick is nice when you’re dealing with uncompressed data, but ends up eating a couple extra bytes after compression. Reusing function and variable names is also very, very important. The more repetition you can introduce, the better off you’ll generally be.

Wrapping up

All in all, these tips (and general insanity) will let you build really small demos. My demo, Magister, went from around 3k down to just over 900 bytes when all was said and done. I ended up adding a message in the demo (“1k should be enough for anybody.”) to get it up to exactly 1024 bytes, which was my goal. Feels good to have to work up rather than down.

I hope this helped shine some light on the subject.

Above: Evan Wallace has once again created a much talked about tech-demo called, with great efficiency: WebGL Water. The tech demo features raytracing, soft shadows, caustics and “analytic” (meaning “fake” :) ambient occlusion. It all comes together in an interactive WebGL app that looks good and has a very decent framerate, even on middle-of-the-road hardware. Well done!

Above: Another piece from Evan Wallace, this time taking on path tracing to create more realistic lighting and shadowing in a scene. The tech-demo allows you to change the material, move the lights and add/remove objects in the scene, as well as moving them around. Fun to play with, and again, surprisingly fast.

Above: MKVDB has made a WebGL port of his native Windows 1k executable “Nautilus” intro. Goes fullscreen and is a little heavy for most mid-range computers, but if you have a fast one it looks really gorgeous.

Above: The famous Cadillac Ranch art installation in Texas has found a home online in the virtual Cadillac Ranch. You can move around (granted, the controls are not exactly top notch) and spray-paint the car-wrecks in the sand. Built on WebGL.

Welcome back! We created a character in part 1, using various 3D-modelling techniques inside of Maya (or your preferred modelling package). We now have our 3D-mesh and UV-layout ready to go, though our character looks kind of dull just with that default-gray Lambert-material in Maya.

Colors anyone?

In order to unleash your creativity and bring your character to life, you need one (or several) texture maps. We will start out with the diffuse map. In a nutshell, the diffuse map is a 2D picture that gets projected onto your 3D mesh, telling your application (or game) what color should be assigned to which polygon. Similarly, you have tons of other possible maps for things such as specularity, transparency, bumps and what not. The picture above depicts the notorious Sir Nigel Rochester (known from “Evil Junior” of course) as a 3D mesh with a diffuse-map containing all of his color information.

Let’s see how you go about making one of those diffuse maps!

Enter Photoshop

Photoshop is perhaps the application for painting 2D texture-maps (but feel free to give me a shelling in the comments field if you disagree). You may absolutely use other packages (or even hand paint your textures and scan them) but in this tutorial series I’m going to stick to Photoshop. Many 3D packages also provide more or less useful texture-painting capabilities, most notably ZBrush and BodyPaint. I will actually showcase some techniques in BodyPaint later on, so stay tuned for that.

Back to the action – fire up Photoshop and create a new file. I recommend the following settings for your texture maps:

• Width x height: 2048 x 2048 pixels
• Resolution: 300 pixels/inch
• RGB Color with 8 bits

This should result in a PSD file of ~12 MB. 2048 x 2048 pixels = “2K” in pixel-pusher lingo. 1024 = 1K. You get the idea. Now one piece of advice (which I learned for myself the hard way): don’t be cheap on your initial texture resolution — ever! By heeding this advice you will hopefully avoid the painful mistake of starting out with a texture resolution that is too low, only to find yourself starting from scratch because the texture you spent several hours working on looks blurry on your mesh. If your initial texture-file is 2K, you can always export your texture as 1K if need be. But a 1K master-file upscaled to a 2K texture map is surely an abomination.

Ok, so 2K should do for this project. Save the newly-created file as a PSD in the sourceimages directory of your Maya project. Name it something like: yourcharacter_diffuse.PSD.

Texturing preparations

Now you should have a clean 2K-canvas facing you in Photoshop. Let’s go through a few hints before you continue on your texturing business.

First, make sure you have exported a “UV Snapshot” from the UV Texture Editor inside Maya, as depicted in the picture below. I will use this snapshot as reference in the painting process.

Next, you can either save your texture map during the process as an image file and insert it into your map in Maya (as we discussed under “Assigning Maps” in part 1) — or — you could utilize this nifty trick: create a new material in your Hypershade Editor in Maya by selecting “PSD File”. Browse to your PSD file in “Image Name”. Select your mesh in Maya, then RIGHT-CLICK the “place2dTexture” node in Hypershade and click “Assign texture’s material to selection” (see image below).

You should be able to save some time by hitting “Save” in Photoshop and see the PSD material instantly update inside Maya. With that out of the way, let’s set up some layers inside of Photoshop.

Layers are among the texture artist’s best friends: they help creative (and often disorganized) people be effective (and in the process: more organized). They eat RAM, so don’t go crazy on your layers. But a few will help you experiment and save you a lot of time. Remember that you may also be part of a team and that other artists may need to work on your files. How to win friends in the art department: keep your files clean, properly named and organized.

The picture above shows my layers for Nigel’s PSD file.

• Put your UV snapshot at the top, with “normal” or “multiply” as the layer blending-mode.
• Create a folder for ambient-occlusion effects. This is a very nice trick I will dive into next.
• One folder for your “diffuse”, i.e. where you put all your paint layers.

Alright, now let’s bake that ambient-occlusion map…

Ambient occlusion

This step is optional, but with the process being a snap and the results being good — why not give it a try?

Ambient occlusion (AO) is an effect in which you let the renderer determine shadows for you. A white material is applied to everything in the scene, and then the scene is lit. Lighting and shadow information is calculated and the renderer “bakes” an AO-map containing only shades of black and white. By doing this, you basically let the renderer figure out shadows on your mesh. It can really add that little extra and also save you a lot if time in the texturing process. Lazy artists are good artists. I will briefly walk you through the process of baking AO-maps in Maya right now..

Ok, so this is what we will do: open up your 3D mesh in Maya and apply the ambient-occlusion material in the Hypershade Editor:

• Create the surface shader: Maya -> Surface -> Surface Shader.
• Create the AO shader: mental ray -> Textures -> mib_amb_occlusion.
• Connect the AO to the surface shader in the Work Area by MMB-dragging AO output to surface input (see picture). Choose default (outValue -> outColor).
• Select the AO shader and open the Attribute Editor (CTRL+A).
• Play with different settings under “Samples” (quality of output) and “Distance” (effect). Try setting “Samples” to 128 or 256.
• Apply this material to your mesh.

Now we need to “bake” (or in other words: render) this texture:

• Select your perspective camera in the Outliner. In the Attribute Editor set Environment -> Background Color to white.
• Select Rendering in the menu selection.
• Select Render -> Render using -> mental ray (you may need to enable mental ray in the plug-in manager) in the main menu.
• Select your mesh.
• Hit Lighting/Shading -> Batch Bake (mental ray) Settings. Set the resolution to your texture map resolution, e.g. 2K.
• Hit Convert when done.

The resulting map will be baked to renderData -> mental ray -> lightMap in your project directory. The result should look something like this:

Awesome. Now paste your freshly-baked AO-map onto its own layer in Photoshop, set the layer to “Multiply” and make sure it’s placed on top of any and all diffuse-layers you might already have. Adjust the blend-percentage to taste. Voilá!

3D texture-painting in BodyPaint

This step is optional but highly recommended if you have access to an application supporting 3D-painting. Painting a texture on a 2D canvas can be both limiting and frustrating because of distorted and slow feedback. 3D texture-painting enables a much more intuitive and responsive process because you paint directly onto the surface of the mesh. There are two methods for 3D-painting used by different solutions. ZBrush offers polygon-painting where you paint individual polygons directly. BodyPaint stores color information as an intermediary 2D-image. Maya also offers 3D-painting capabilities. I will use BodyPaint in this example. The principles are the same, regardless of application used. Here is what you need to do:

• Export the mesh you want to 3D-paint to an .OBJ file in Maya.
• Open BodyPaint. Create a new project. Import your .OBJ file.
• Select all mesh objects in the “Objects” tab.
• Create a new material in the “Materials” tab. RIGHT-CLICK the new material and hit “Apply”.
• Click on the material icon. In the “Attributes” tab click Color -> Texture -> Create New Texture. Alternatively, load your existing texture.
• RIGHT-CLICK the material icon and hit “Enable 3D Paint”.
• Create a new layer under the “Layers” tab.
• Start painting!

Now you can either make a texture from scratch or enhance your existing textures. 3D-painting is also great for painting in reference for 2D-painting. This is what I have done in the picture below. This way I can easily figure out where I want to paint Nigel’s shadows and skin in Photoshop.

When you are finished simply RIGHT-CLICK the texture icon next to the material under the “Material” tab and press Texture -> Save Texture as. Save as .PSD if you want to keep your layers.

Painting in Photoshop

Contrary to popular belief, you don’t need a PhD in Photoshop in order to paint awesome textures. All you really need is the paint brush. Here are a few additional tricks you may find useful:

Layer masks: Click in the background (outside the grid markings) in your UV-snapshot layer with the Magic Wand Tool (W). Click CTRL+SHIFT+I to invert the selection. Create a new layer. Then hit “Add layer mask” in the layers panel. You can see the result in the picture below. Next to the mini-version of your paint layer, you have a black-and-white representation of your layer mask. 100% black in the mask = 100% invisible, 100% white = 100% visible. In other words, layer masks are an extremely flexible and powerful tool in the painting process because you can “paint” in or out what should be visible in this particular layer. This is very nice when you want to blend your layers manually.

External textures: You can achieve insanely nice and lifelike effects by adding external textures and images onto a separate layer and experiment with the blending mode of the layer, e.g. “Multiply” or “Overlay”. Does your character wear armor? Try adding a metal texture with scratches and wear-and-tear.

Warping and liquify: Sometimes you need to push things around, like if you import a 2D texture that needs to be warped according to the grid lines of your UV snapshot. You can use Edit – > Transform -> Warp or Filter -> Liquify.

Pressure-sensitive tablet: This is a no-brainer. Texture-painting solely with a mouse is possible but extremely limiting. A pressure-sensitive tablet is a good investment (if you do not already have one) and will definitely make your day easier as an artist.

Pressure-sensitive brushes:  If you have a tablet, try experimenting with the brush settings under “Brushes”. Especially “Shape dynamics” (size) and “Other dynamics” (opacity).

Basic brushes: I tend to use a basic hard round brush, a soft one, and combinations of these with sensitivity for either size or opacity. Try making other brushes by painting a black-and-white pattern on a square canvas and hit Edit -> Define Brush Preset (or simply download free brushes on the net).

With those few tricks freshly memorized, it’s time to paint!

• Start with a color filling the entire background (left image).
• Import reference from 3D-painting if available.
• Using your UV snapshot, and popping in and out of Maya to check the effects of your actions, start painting in the basic colors with a hard round brush for each mesh piece or body part (middle image). You can also use your AO layer for reference.
• Once you have painted in all your desired colors, try enabling your AO layer to see how it looks (right image).
• Want more detail? Experiment with more advanced brush settings or add external textures with blending.

When satisfied, save out your work. I like saving as .PNG (which also stores transparency), though you may find a better choice depending on the requirements of your project (consult your art director or lead programmer if you have one). If you have an established pipeline, it will most likely do batch conversions of textures and assets at some point, so keep your original textures in high quality is always a good thing. Save the file in the same resolution as your PSD. Name it something like yourcharacter_diffuse.

Specularity and other maps

If you want additional control over other properties than just color, you can create other maps as well. For example, you may want Sir Nigel’s top hat to be extra shiny. Shininess, or rather specularity, is determined by a specular map. Create a new layer in your Photoshop texture-file and paint with black-and-white, black representing matte, gray semi-glossy and white very glossy. Paint as in the picture below:

You can use the same technique for controlling other surface-properties such as bumps (e.g. for hairs and fur) and transparency (the glass of Nigel’s monocle). A much-used map in video games is the normal map. Normal maps are color images providing rendering information for how light should be reflected on the surface of the mesh. They are often used for giving the illusion of indentations and protrusions on flat surfaces, such as muscles, bones and wrinkled clothing and so on. Normal maps can be created from high-polygon versions of your mesh.

Wrapping it up

OK, let’s conclude this part of the tutorial by adding our diffuse and specular maps in Maya:

• Create a Blinn material and assign it to your mesh.
• Open the Attribute Editor (CTRL+A) to edit the material properties.
• Enter your diffuse map in Common Material Properties -> Color.
• Enter your specular map in Specular Shading -> Specular Color.

If you want to render an image of the result, do the following:

• Add a primitive plane to your scene. Stretch it out so that it acts as the ground. Select two sides of the plane (the ones visible in your perspective camera), extrude those edges and “lift” them to create walls. When done, hit “2″ on your keyboard to smooth it. Apply a white Lambert-material to it.
• Then add an ambient light to your scene and position it.
• Set your renderer to mental ray.
• Position your perspective camera, open the IPR renderer and select a small portion of the scene for live preview-rendering.
• Now, play with the settings of your mesh’s material and the lights/background until you are satisfied with the result in the IPR renderer. Also try experimenting with the render settings such as quality.
• When happy, render the current frame with mental ray.

That concludes this tutorial part. We now have a 3D mesh with maps for diffuse and specularity. I hope you have enjoyed this tutorial, and please feel free to ask questions or add your own work in the forum!

Hi, I’m Ollie “cosmic” Borgardts, and I am the creator of the animated short “The Lacquerer”. I’m going to do my best to make this an interesting and entertaining read, but I should probably say up front that I will go into detail on how the movie was made. I’ll also touch upon the creative process and my personal experiences from making this production. At the bottom of this post I have attached a 30-minute walkthrough video in which I explain some of the techniques I used.

Behind “The Lacquerer”

The whole production took about 2 months (full time) to complete, and with such a long production time I feel I have plenty of tips about the production process and the ideas and techniques that went into the piece. The short was released at the Revision Demoscene Party 2011 where it won the Animation Competition.

The whole project started two months before the competition was to be held, when a good friend of mine, Pro from the demo group Nuance, told me about a promotional campaign for a 3D package named Messiah Studio at a very good price. I have been a faithful Maya-user for many years, but when I checked out the package, some of the features it offered were very interesting. Even though I had never used it before, I promptly ordered a license in the belief that I could learn what I needed to know in good time before the competition deadline got too close.

One trick I learned was to watch tutorials at double or triple speed to save some time

I really wanted to finish a cool production for Revision, so with hindsight being 20-20, I probably picked a bad time to start learning a new 3D package. I proceeded to spend quite some time just watching various tutorials online, to get to know the application and the differences from Maya. One trick I learned was to watch tutorials at double or triple speed to save some time (no joke, it’s what I did and it was very effective).

Just to touch on the fact that I decided to switch 3D packages. I do simply love Maya. I use an older version (6) which I bought several years ago and I’ve been using it so long that I know the program in and out and can therefore work quite efficiently in it. However, I do have some issues with it.

When creating digital characters I would begin in ZBrush and then bring it into Maya for rigging. To bind the bones to the geometry Maya requires you to go through a process called “weight painting”. This tells the skeleton how much influence each single bone has on the bound geometry. There are a few scripts that help you automate then I need to bind the bones to the geometry and do a process called weight painting. This tells the inserted digital skeleton how much influence every single bone has upon the bound geometry. There are scripts to partly automate the process, but to be honest: most of them suck, and doing it manually sucks even more.

There are most likely newer versions of Maya that has new features or techniques for rigging and skinning, but for me, taking the step over to Messiah really helped me out. One example: in Maya, if I want to change something after the bone structure has been set up and bound to the mesh, I would have to unbind all of it and repeat the process on the new or changed mesh.

For an artist, this workflow is killing productivity. I want to be free from the very beginnng, the planning stage, the modeling, texturing, animation and right up to the final export or the encoding process. I want to be able to make changes at any point in that toolchain. Period.

Another example: changing the character from a humanoid biped to, say, a 3-legged beast with 10 arms, I’d have to repeat the whole process. Again, this doesn’t work for me. This is one of the areas where Messiah shines. It has a very clever way of separating the binding/rigging-stage from the animation. If I wanted to add more bones, I’m able to do so, and the animation and other parts remain untouched. Awesome.

Thoughts on creativity (and art in general)

I was now in a place where I knew how to use Messiah — which is exactly the time that the creativity black hole hit me. I’d like to share some thoughts about creativity in general and about the approach I use when realizing my ideas.

My tools of choice are Maya (or now: Messiah), After Effects, Photoshop and ZBrush. Not necessarily in that order.

I believe that there are really no “new things”. Pretty much everything has been done before in some way. Even so I look at other artists and what they doing and I’m definitely learning something new every single day, and that’s a good thing. However, if it’s one thing I know it’s that creativity can’t be forced when you are on a deadline.

I want to surprise the people at the event with my production, but how could I surprise them if I can’t even surprise myself? For each project I start, my personal goal is always to surprise myself in some way — to create something I’d never thought of before it’s actually finished.

So how can this be achieved? Well, I know what I’m able to. I know what I’m too unskilled or too lazy for. I know my tools. I know my skills and my preference when it comes to art. I know these things inside and out, so how do I “trick” myself? Well, one approach is to combine existing things. Take, for example, three existing things and combine them in some random fashion. It doesn’t matter which way you do it. The point is to not be logical and simply negate the facts.

Just to give you a little example of the process that works well for me:

Look out your window at the scenery outside. Then go: “Why do trees have their roots planted in the groud? What if it was upside down?” Argument one: defined.

Continue with the simple assertion: “What would happen if the tree was planted upside down?” Well, for one, it wouldn’t be able to access any water. So the consequence of this could be that the roots would need to stretch to the clouds to access water. Argument two: defined.

But what with the birds that usually sit on the soft branches? They don’t want to sit on the ugly, muddy roots. No, they want to sit on the branches and leaves as before. Therefore the result could be that they hang themselves upside down instead, reaching for the branches. Argument three: defined.

..and boom — there’s your scene. The rest is just a matter of a technical completion, however you choose to proceed.

The image to the right isn’t exceptional, but I hope you get the idea how everything in an artistic world could be created in this way. Just let the things happen. You don’t want to control everything or be too logical here. That simply doesn’t work — or, to be more precise — it simply doesn’t work for me. I know a thing or two about the technical stuff, but in the creation process of something artistic, taking a logical approach is just disturbing the art.

However, when the creative process is completed, you can come back and go on working on it in a professional, logical and disciplined way.

When perfectionism and ego gets in the way

At an early stage of making “The Lacquerer” I just wanted to use ZBrush, model something and then animate it. I asked a friend of mine, Marc Ewald (a musician from the Netherlands), if he’d like to do a soundscape for it (just like he did for “Julie” my last short movie experiment for Breakpoint 2009). I believe he’s a genius in what he’s doing and so the brief I gave him went something like this:

Please think of a concept for a short film (but don’t tell me what it is) and then create a soundscape for it. No identifiable sounds like bird chirps, car horns and such. Just make something abstract that will fit with loosely defined terms of contrast, such as “fast”, “slow”, “large”, “small”, “oppresive”, “liberating”. Again: don’t tell me the meaning of it. I plan to let the story unfold itself while I’m making it.

The reason for this was of course to have no preference or preconceived notions when starting up. I could listen to the sounds as a small child would and just let the imagination kick in. What would I think of those sounds — what would they look like? Then I could concentrate on creating something visual around the sound completely disconnected from the visuals.

What I got from Marc was very good. Bombastic, epic, almost score-like in nature. I could picture it in a game such as “Metal Gear Solid”. It was absolutely fantastic and I was speechless. My problems started when I was trying to visualize them. Suddenly my creativity was .. gone.

I had vivid pictures in my mind, such as huge boss-fights in epic PlayStation 3-games. The sound was great, but I kept thinking: “How on earth am I going to be able to match this, visually? How can I live up to the expectations the sound is setting the scene for?”

I thought about all those things for a couple of days, and then, as it does, a happy accident occurred.

I stumbled upon some audio samples I had laying around on my harddrive. They sounded interesting so I chopped them up randomly to a 3 minutes collage. I also, for some reason, started modelling an old guy in ZBrush.

At this point I was slightly tired of the whole thing, but stuck to it anyway. Working on something was better than thinking about it. I believed that a solution would present itself, as it usually does.

I threw some bones onto the pretty sloppily created mesh and started animating it in a random way. No plot — just trying to forget how I would go about handling the miserable situation I was in. I had no idea on what to create, and I felt really bad for Marc who had created a great piece that simply wasn’t what I needed at the moment. I really wanted to use it, but it was dawning on me that it wasn’t what I was looking for this time.

I rendered out some seconds of crap and brought it into After Effects where I started playing with color correction and other generic things. I just wanted to keep working, waiting for the solution — the idea — the special technique.

This is when I more or less randomly started playing with a feature in After Effects called..

Time Remapping

Time Remapping squeezes and stretches your video footage by changing the timing and playback rate. You can slow down things or speed them up, and it does the same with the sound — just like a record on a turntable. It’s a very basic feature, and pretty much the first thing you learn when you start using it. However, I had I never thought about using this in a creative way. It was too simple, far away from the “epic effect” that you see in the movies. It’s totally overused, boring and just too simple.

In fact it’s so simple that I thought everyone using After Effects is using it. This is where ego kicked in.. “I wouldn’t use that — not me — the best digital artist in the universe — the one that only uses super-complex techniques — not even people at ILM do what I do..”

What utter bullshit! Ego struck again! The little bastard in my head took over and was controlling me in a very destructive way that lead to total stagnation — and I didn’t even realize it at the time. Instead of having fun with the new software, the digital tools and playing around with them to create something cool in a fun way, my ego — the perfectionism and logical approach — ruined everything.

This cheap and simple trick which I just played with by accident ended up as the concept for the shole short. All the content I was to create in the coming four weeks would be held together by this very basic tool. I decided to write it down so as not to loose it: “Create a very detailed creature, animate it in some way, sync it all with Time Remapping afterwards!”

This cheap, simple trick, found “by accident” was the concept for the whole movie. This would hold everything together. So I wrote it down that this won’t be lost again: “I’m going to create a creature with the best skills I have at this time – I’m going to bring it into messiah and animate it in some way, but in perfect sync to the audio to “destroy” it afterwards with simple time remapping“.

Then another problem presented itself: I still wanted Marc to do the soundscape. He has studied composition after all, and he’s by far my musician of choice to work with. But he needed some clips of the concept to create the audio, and that was a problem because none really existed at this point. My solution ended up being that I would animate the short with the glitchy audio I had already created, and he would create a new, better sound for it. Remixing it, in a way, but keeping the same major sync points.

Unfortunately, that never happened.

I was working non-stop with animating right up until the last day before Revision was about to begin, and therefore I couldn’t expect Marc to be able to create everything needed in only 12 hours. I’m quite sad about that, but I didn’t want to put him in the same position as I was in earlier — what if he wasn’t inspired with what I had created? Lesson learned: the next time I’m going to involve him at an earlier stage, promise.

Giving life to the creature (blendshape creation with 3D layers)

I spent some time modelling the figure, texturing it and creating the blendshapes for it. All of this was done inside of ZBrush. I love ZBrush, it has never let me down. In other applications you can very easily end up stuck in a one-way street somewhere down the pipeline or the production process, but with ZBrush I have never had this happen. At any point there is a way to backtrack and get back to what you need to change.

So what are blendshapes? You can think of them as layers in Photoshop, except in 3D space. You create different “versions” of your model on layers and with a slider you can switch between these and create stuff like facial expressions this way.

These are then exported with the original mesh and connected inside of the 3D application. Every major-vendor 3D package has those – Blender, mMx, Maya etc, but their lingo might differ (like “morph targets”). These imported blendshapes are then invisibly connected to your original model and can seamlessly be blended into each other. This again can then be keyframed and animated.

Principles of complex effects (or: “Hollywood has time and money”)

One look I wanted to create for the project the figure was one of having the character look like he was coated with something like clay or mud. If you’ve seen any of the X-men movies then you probably know the character “Mystique” (if not: this is what that looks like). Her skin morphs in a pretty crazy way. It’s not just blended though, it looks like there’s more going on under the surface, like tiny, greebly, flaky stuff. It looks really hard, but actually it is — once again — very simple.

I couldn’t find the technique described in any tutorial, book or in the 3D forums online, so I couldn’t get it exactly right. My effect isn’t as elaborate or worked-through like the one I mentioned, but I think it worked out well and I’m going to show you the basic principles behind it.

The basic idea is to create two morph targets — one untouched (a clean slate, if you will) and one for the tranformed shape (a mask or one with scales or whatever you like). Step one is taking these two models and blend them using influence modifier objects like a sphere or a cube or any shape you can think of. That gets you started, and you have great control over the look of the animation, but the desired “greeble-look” isn’t there yet. So step two is to add complexity where the blending areas of the structure are switching from one shape to another. For this I created a third map that distorts the model in a very overdone way. Very random and crazy.

At that blending point this third map is basically used to “destroy” the geometry in a small rim. This gave me that “wrapping around” effect you see when the figure is coated. I could have used an animated map as well to enhance this effect even more (and I think this is how they create the skin effect on “Mystique” as mentioned above), but I was satisifed with the way it looked.

Another, simpler, technique would have been to render out two versions of the animation and then simply blend them in post-production. However, this won’t give you correct shadows and the compositing job would have been at least triple the work in the end. Real geometry is nearly always best, if you can get it right.

The devil is in the Multi Displacement Map layering (aka “details”)

As you might know, the models you can get from ZBrush can have a very high poly-count. With an average computer with 4 GB of RAM you can easily make models containing up to 100 million polygons or more.

ZBrush can handle this because internally it’s not using polygons, but something they call pixols — more commonly known as voxels. These are pixels with depth information. It’s a clever fake so that you can work comfortably inside of ZBrush without lag, but once you actually export the model, it is converted into real polygons. The most common approach to maintain a high detail level is to create a displacement map or normal map and place this onto a lower poly-count version of the model.

Displacement maps are nothing more than grayscale images (colored ones for normal maps) that create the illusion of having the same high detailed level. A little clarification: with normal maps this is the truth, but with displacement maps the high poly count is acually created during the rendering process.

I took these mapps and added them to the blending process in Messiah with the same influence blend modifieres to bring in details in the model and color-space as well. However, a single displacement map didn’t yield the result I was looking for so I had to research the technique of using so-called “Multi displacement maps”.

Imagine a base model, like a sphere with 100 polygons. You can create a displacement map that deforms this sphere in a good shape, like the basic shape of a human head. With only one displacement map you can just pull “outwards and inwards” to create indentations. The side planes of each newly created “extraction” can then not be affected anymore by the same displacement map and this is a shame.

If you extract a normal with a displacement map that results in some sort of “cubic” form. Then you have no information left in the map to extract the faces again in direct way, because the “detailing side parts” are simply just being created during the rendering process. With this in mind, a single displacement map wasn’t really that useful. You need another map that provides that information, and the good thing is that you can stack tons of maps onto each other and get really deep details out of it. At the time I started this project I didn’t know how to do this, but as it turns out, it was surprisingly simple (as in: just the click of a button). You can watch the process in the making-of video at the bottom of this post.

Compositing is everything (or: “How to make crap look good”)

With the model, morph targets and textures done, it was time to getting it to look good. While I’m proficient in creating 3D models, it’d be a lie to say that this is my favourite thing in the process. I see it more as a necessary evil, because creating things in virtual 3D space eventually brings you to the point where it starts to suck. Or to put it differently: you can never expect to finish your projects in your 3D programme. You need post-processing. Even if your lighting setup is perfect, you animation looks good and the models are perfect — the true power comes when you step out of 3D space and enter post-processing land.

When I started toying with 3D modelling and animation almost 15 years ago I thought that the way the professionals worked was to set up a scene, model, animate it, set up the lights and click the render-button. In reality, no movie is made in this way, because it’s simply neither practical nor yield good enough results. Think about it like layers: what if the base rendering takes 15 hours to finish? What are you going to do if the shadows are too dark or the colors are slighty off? You can’t just render it again — a very bad workflow.

The correct (and only) way to do it is to composit different layers on top of another, with the base rendering being the lowest lever in your chain. Everything is done in layers, and when combining them (following some simple principles and a few standard rules) is what nails the look and feel of a movie. Once you know these rules you are also capable of breaking them, which is what I usually do. In all my pieces they were at some point at a stage that I just played around in Photoshop to try the find a look or a feel, but the basic workflow and the principles remained the same.

Warning — soapbox moment! It’s really about the basics. Turning on a plugin isn’t just lame, it’s counter-productive, because it means you’ll never know what’s truly going on under the hood, and therefore unable to control it well. Some plugins can make your life easier when you are in a hurry, but is no substitute for having a really good set of basic skills. Do your homework. Buy books on anatomy and lighting. Knowing these basic principles will make your work better.

Oh, and a top-tip: always render to still image sequences. If you suffer a crash or a powerloss during rendering to a movie-file, all is lost, while if you worked on a still image sequence, you can just set your renderer to continue where it left off.

Shading the character

The best looking renders, in my opinion, are the ones I get directly out of ZBrush. ZBrush is fast and uses lighting baked into so-called MatCap Materials. The only problem with ZBrush is that it has no real 3D space. Again, these are only pixels with depth information and so you can’t “go around them” (or I should rather say “walk inside of a model”), because you can’t rotate 2D. In ZBrush it’s actually 2.5D information you’re dealing with

To make skin look real you need a feature called subsurface scattering. Think of it as holding your hand in front of a strong light-source at night, where you can see the light through our fingers. This is due to the scattering of light inside of your skin. I really wanted this look in the movie, so to enhance it I set up a simple shading network inside of Messiah to get a rim shader.

For stills I get everything out of the programme: depth Pass, diffuse pass, SSS, specularity and so on. The trick lies in using these elements to make it look real in the compositing stage in Photoshop (if you’re working on a still image) or After effects (for movies).

The key things are: color, perspective, depth and hue.

I’m going to show you a simple way of getting a simple, really ugly and random scene to look realistic in some way by just following a few simple steps. Mind you, this technique is for still images only. You can see the end result in this image to the right, and for the how-to, watch the making-of video at the bottom of this post where I go through each step of the creation of this image.

Getting the look down to a sensible render time

I wanted to use the so-called MatCap Materials for the shading of the skin. Matcaps handle the way of shading in a very clever and fast way, and they look really cool, so I wanted to use these inside of Messiah. In Maya or Blender the setup for such a shading network is pretty easy, but not in version 6 of Maya. In messiah I had no clue on how do it, but luckily there is a nice and friendly Messiah-community from which I got help an tips on this. One of them even set up a full shading network for me, but sadly had no time to implement it due to the looming deadline. In my next production I’m going to use these techniques.

However, I still want to show you how the MatCaps can be used inside of Blender. The workflow can be most likely be applied to every other major 3D package.

In ZBrush, just render out a sphere with the desired material applied to it. Make sure the document width and height are at the same settings and that the sphere is perfectly placed in the center. Oh, and turn of perspective in ZBrush — using the orthogonal view will give you the same result. Export that texture and map it using to the color channel using the normals. Make sure that your shader is turned to shadeless because the light is already baked into the the texture, and the painted color information from the actual model is simply added on top of that. This way, you’ll get the exact same look as if you were still in ZBrush. That’s it.

With everything rendered out, it was time to have fun! I used Time Remapping here and there in After Effects and was fascinated by the sense of randomness and oppressive mood that it caused. The character was absolutely animated to the sound, but with the added effect of having the sound also stretched and squeezed where I used Time Remapping. I really liked how everything was coming together at this point.

One thing I haven’t touched upon is the glass panel in the movie. It was entirely done in post. Just a simple 2D tracking of the hands touching the glass and then a few dirt textures on a null object.

In terms of the story, my first idea was to the character break the glass when he hits it, but I found it more creepy and scary if he was still stuck behind the wall of glass.

“Making-of” and explore the scene files

If you’re interested in the scene files for this movie you can download them (project files & head), play with them and explore how they were done. I have also created a 30-minutes “Making-of” video which touches on many of the techniques described above.

I want to say thank you for your time and your interest in how all this was done. Of course not every single aspect and every problem that occurred during the creation process could be described here. Even if I’d tried to cover them here to warn you before embarking on your own projects, other problems will still occur. The devil is in the details.

It was really, really hard work to create this movie. Pain and fun and I earned a lot by going through the process. If they happen to read this, I would like to offer an apology to my friends and family who were really forgiving during my mood-swings. :) Also, thanks to the people sharing their comments and critique about the movie so far — I hope you will take time to watch it and comment below as well.

So, stay tuned, keep creating you own productions, don’t let discipline and perfectionism ruin your creativity — just don’t take life too serious and have fun (yes, I know — this applies to me as well :)

Getting started

Since I’ve become involved with Quartz Composer (also known as “QC”), I’ve read countless posts to Apple’s Quartz Composer Developer list or Kineme where users complain about a lack of documentation. I don’t think that criticism is entirely valid; I think it’s a matter of history, and this article is going to go into that a bit. What follows will be a bit of broad overview of some issues surrounding Quartz Composer, while future articles will be more in depth and certainly have a great amount of info that one can use in many domains, not just Quartz Composer, though there will be a focus on implementing in QC.

Back to the common complaint of lack of documentation; a cursory browsing of the internet will reveal countless pages that attempt to explain Quartz Composer, they just don’t explain the problem the user wants to solve. Users often disregard Apple’s documentation as a starting point as well. I suggest Apple Quartz Composer Programming Guide as a very good piece of reference material.

Apple has done a great deal of documenting of Quartz Composer, provided tons of example projects that show how to work with Quartz Composer graphs (qtz files), how to write plugins for Quartz Composer, or integrate it into applications that use the technology. Doing a search through Apple’s Developer code will reveal numerous examples and articles, all of which I would recommend that anyone interested in Quartz Composer examine. I think these are the absolute best examples and documentation of Quartz Composer in existence.

The documentation workaround

That said, there is a very major reason why people tend to immediately ignore Apple’s documentation, and it’s somewhat valid.

With Apple’s standard API, one can indeed expand the modules that Quartz Composer loads into it’s environment by writing plugins that one can construct graphs with, but it also places numerous embargoes on that QCPlugin API. There is a large subculture – or perhaps the dominant culture – of QC users that utilize what has become known as “the SkankySDK” to circumvent this. For example, Apple may prevent plugins from loading inside of certain environments like the Web Browser, or from utilizing built in QC data types like “QCMesh” (because of proprietary reasons or simply a lack of updating the spec. Speculation is probably unhealthy!)

Before Apple had actually published a plugin API, users were eager to extend Quartz Composer’s functionality. QC was based on Pixelshox, which was also an extendible technology, and number of users had noted that Apple had also designed Quartz Composer to load “GFPlugins”. By backwards engineering this API, this soon became a way for users to extend Quartz Composer’s functionality. By the time that Apple had actually introduced a plugin API, several developers were already using the “skankySDK”, and numerous plugins that conformed to this standard “non-standard” had been released. Interestingly, Apple continues to create GFPlugins for use in their own projects, choosing not to eat their own dog food. The infamous tape where Apple calls the reverse engineered SDK the “skanky” SDK while introducing their hobbled SDK is entertaining enough though.

What was most problematic is that through use of GFPlugin/skankySDK one could create new data types inside of Quartz Composer, and leverage control over the entire system. QCPlugin uses a wrapper that precludes one not only creating new data types, but also creating certain types of macros, or even utilizing certain built in data types.

While it is far from useless, and one can create a great deal of things in that box, the fact that QCPlugin is limited, but accepted by Apple, has a significant impact on Quartz Composer development and culture. If one wishes to create new modules that are allowed in apps to be sold on Apple’s app store, one must write plugins in that box. If you wish to write modules in a way that leverages complete control, you use the skankySDK, fully knowing that this isn’t viewed as an awesome thing by Apple, and also, that they do not support.

With a little help from your friends

Over the years, Kineme codified the non-standard plugin API/skankySDK into an actual framework, which is available at their site. Kineme’s forum is also the largest community of Quartz Composer users, and a great place to get feedback and help, as well as an excellent source of add on modules. Another very noteworthy developer in the QC community is Anton Marini, a.ka. Vade. His v002 plugins are considered to be some of the most useful plugins available for Quartz Composer, and he also shows how far boundaries can be pushed even when using Apple’s standard QCPlugin API (and he also has a forum for support).

It may seem quirky to lay all of that out on the front end, before even getting too far into “what Quartz Composer is”, but I never said I wasn’t quirky. I think it’s important to understand this when delving into Quartz Composer, and good to understand it before getting deep into interacting and programming with the system.

The inner workings of QC

Quartz Composer is multifaceted. It presents itself in the form of a framework for OS X, as a technology, and as an application to leverage the technology. A good way to think of Quartz Composer is that one is creating a file, and that by linking together modules (patches) with cables (noodles), one can send information from one module to another. Anyone that is familiar with node based editing systems may also think of these as nodes, and of Quartz Composer as a node graph, and that’s an accurate description. Others may think of this as a decision graph, or an execution graph. There are strong parallels with non-programmer paradigms, especially signal flow. Musos that have chained together multiple effects boxes, or modular synths have a good running start to understanding the method of interaction with QC. Users typically work with the Quartz Composer.app, which is included with Apple’s Developer Tools, to create and manipulate Quartz Composer graphs.

In Quartz Composer, a method is used that is called “lazy evaluation”. QC graphs tend to lay out from right to left on the Quartz Composer editor application. When one connects the output of a QC Video Patch, to the image input of a QC Sphere, execution happens thusly: The QC Sphere is a “Consumer” patch. First, a check happens to see if it is enabled. Then, all input port values are surveyed.

If there is a value at the port, it’s used. If there is a connection to another patch upstream, like the Video Patch, that patch then executes. If the Video Patch output was connected to nothing, it would never execute at all. If there was a switcher in between (known as a multiplexer in QC parlance), and connections existed, but the Video Patch was not currently chosen, the Video patch would still not execute. If that Video Patch has an input port, connected to another patch’s output, then it will trigger the next patch in the line to execute. This is likely an important distinction to make to people familiar with graph environments that use a Push model (like Max or VVVV).

The fact that one can program plugins, whether skanky or standard API, that there is a defined evaluation mechanism, that one can incorporate any of Apple’s frameworks fairly easily, use past work modularly, or write GLSL shaders, OpenCL kernels, JavaScript programs in the Quartz Composer editor while live compiling and seeing the output results render in the QC Viewer while working makes QC a tool for some projects that is indispensable.

I’m excited to delve into that more. In coming installments, we will learn about creating graphs, prototyping shaders and CL kernels, extending functionality with the standard and skanky SDK’s, integrating Quartz Composer into standalone apps, as well as broader issues in coding and how they interplay with this technology.

Displayhack is a site for everyone participating in or enjoying real-time graphics, visual arts, motion graphics and the demoscene. Often adduced as the reason or motivation for expending effort toward a seemingly useless goal, the point being that the accomplished goal is a hack. As Louis Armstrong once said when asked to explain jazz: “Man, if you gotta ask you’ll never know.”

..and here is the long one:

Niches

Niche-sites are cool. They are good at what they do, they cater to their audience, and they specialize. They’re niches after all. There are many such sites and while all good and focused, almost all of them either focus too narrow or too wide.

We wanted to create a site where all those interested in the demoscene, web-based graphics, motion graphics, visual arts and other visual fringe technologies could meet, discuss and inspire each other.

Past, present, future

As long-time demosceners, we know very well that the history of the scene is exactly that: history. If we want to play a part in the next generation of visual artistry, we need to do two things: 1) recruit, and 2) embrace. We need more people to enter the scene, actively, and we need to look outside the traditions of packaged executables downloaded to your computer. The next generation of the demoscene is already here, and it’s on the web, in movies and all around us.

Enter Displayhack. It is a site for everyone participating in or enjoying real-time graphics, graphics programming, visual arts, motion graphics and the demoscene, and aims to be a friendly and informative meeting place for beginners, seasoned professionals and everyone in between.

The name

So, where does the name come from? Well, as all truths, it stems from Wikipedia:

A display hack is a computer program with similar purpose to a kaleidoscope: to make pretty pictures (symmetrical or otherwise).

“The hack value of a display hack is proportional to the esthetic value of the images times the cleverness of the algorithm divided by the size of the code.”

Crack intros, display hacks programmed by software crackers for the home computers of the 1980s, evolved into what was to be known as demos and demo effects. The creation of demos later became a subculture of its own, now known as the demoscene.

Contribute

We’re in the middle of adding a lot of content to the site, but what we really want to achieve is a constant flow of new content, driven by our users. If you’d like to be become a contributor, please contact us – we’d love to hear from you! To become a user, simply use your existing OpenID or SceneID — oh, and read the user FAQ too.

Welcome to Displayhack.

For me, as a musician and designer, the most exciting part of the visual aspects of making a demo is undoubtedly the direction, editing and synchronization with the music. The problem is of course that I’m not a coder. Enter GNU Rocket – the general purpose, ad-hoc tool that gives non-coders an intuitive interface to play around, in a very powerful way.

What is it?

GNU Rocket is what we have given the nickname “sync tracker” – that is because all of this is commonly referred to as “syncing” in the demoscene. For anyone with a background in electronic music on the Atari, Amiga or PC using “trackers”, the concept should be very familiar. Instead of having a set of horizontal channels, like working in Logic, Pro Tools or the likes, you work vertically. Each column is a channel, and each channel consists of a series of horisontal “steps”.

Each step can contain a note or an effect, and when you press play, the tracker traces from top to bottom, playing each note or effect when it encounters it.

This is what the GNU Rocket interface looks like:

Instead of notes or samples, each column represents a controllable variable in the graphics engine — and that’s pretty much all there is to it. If you understand this concept then you can safely move on. If you don’t, you can watch this somewhat dull video where someone uses Renoise to make “music”. When he starts inputing samples — just imagine that’s a keyframe in a video. That’s right – that’s how it works.

The tech

GNU Rocket consists of two elements: an API and an editor. The API is non-intrusive, allowing coders to be coders, and GNU Rocket tries very hard to stay out of their way. The API is versatile, engine agnostic and anti-semantical by nature – the tool doesn’t care what you edit, everything is just floating-point variables. The downside of this is that you can’t do high level editing, but that can also be seen as a positive, because it keeps complex decision-making out of the way.

Now to the juicy center: the client is a network server, and the API library is a network client. That’s it. Very simple, but very powerful. This is a typical workflow:

• The client connects to the server and tells it what variable it uses
• The editor sends the data for each variable it has recorded to the client
• While both ends are running, any change done in the editor is instantly communicated to the client
• In addition to that, some control data are also sent, including play position (“current row”) and run data (“play/pause”)
• The control data can be expanded to incude more elements in the future

The API, explained

When using the API, there are really only a few elements you need to keep track of. “sync” is the path-prefix, and the actual file-name is [prefix]_[track-name]. “localhost” and SYNC_DEFAULT_PORT is the host and port to it.

struct sync_device *rocket;
rocket = sync_create_device("sync");
if (sync_connect("localhost",
SYNC_DEFAULT_PORT)) {
/* Handle error */
}

You need to implement hooks to your music-player to use GNU Rocket. It ships with example implementations for BASS by Ian Luck (an excellent library – when in doubt, use it). “rbp” is “rows per beat”, a value of 8 is usually sufficient, and “sync_update” does the network updating.

static void pause(void *d, int flag)
{
/* player specific */
}
static void set_row(void *d, int row)
{
/* player specific */
}
static int is_playing(void *d)
{
/* player specific */
}
static struct sync_cb cb = {
pause, set_row, is_playing
};
/* ... */
while (1) {
  double time = /* player specific */
  double beat = time * (bpm / 60);
  double row = beat * rpb;
  int irow = (int)floor(row);
  sync_update(rocket, irow, &cb, ...)
  /* ... */
}

“sync_track” is a handle to a variable in the track data. “sync_get_track” can only fail when loading from disk, if a track has not been (correctly) stored. The “row”-variable depends on your music-player, but is a function of the playback time, your BPM and “row rate” (i.e how many rows to have per beat in the music).

const sync_track *red_track;
red_track = sync_get_track(rocket, "red");
/* ... */
while (1) {
  double row = /* ... */
  sync_update(rocket, ...);
  /* ... */
  float red;
  red = sync_get_val(red_track, row);
  /* ... */
}

Now you should have a working editor-executable – congratulations! At this point I should mention that while the editor GUI is compiled for Windows, it should be very easy to port to OSX or Linux. The GNU Rocket system itself has been used in productions ranging from full-size demos, down to 64k and 4k intros. It has also been used in demos for Flash and Nintendo GameBoy Advance.

Using the editor

Now that we have all of the tech-talk out of the way, it’s time to actually use the editor to get some work done. The overall principles are frighteningly simple:

1. Start the editor
2. Start the demo executable (available tracks should appear in the editor)
3. Edit tracks
4. Stop the demo, modify source code, add tracks
5. Restart demo
6. Changes and new tracks should appear
7. The demo should start up at the same position as you left it

..and when the time comes to release your masterpiece to the general public:

1. File -> Remote Export
2. Build release version

Hotkeys and tricks

While using the editor, these are your available hotkeys:

Bulding for release

When making a release version, you need to export your tracks to external data-files. With the demo and editor connected, just use File -> Remote Export in the editor. You’ll also need to define a pre-processor variable called SYNC_PLAYER in your demo source. Guard calls to “sync_connect” and “sync_update” with an #ifndef on this, because these functions do not exist when in playback-mode.

The hooks (“sync_cb” etc.) can also be removed if you want, and link to the player-version of the library (this is just the normal version built with SYNC_PLAYER set). Keep in mind that this differs if you’re using the .NET bindings.

That’s it – you’re now done with the implementation. I will write a follow-up post in the near future on how to use the tool in a good way, but for now: get integrating — download GNU Rocket here and use it in your own projects.

Begin

It was easter. We made a new demo for The Gathering 2011.Yea, that’s right – in Norway, not in Germany. I really wanted to do a new demo because I’ve been collecting new routines all winter, and it was high time they got into the wild. So about 3 weeks before easter Jani and I started bouncing ideas around (“something with fluids” was the sumtotal of that I think). Then we went on the hunt for music. As some may know, we don’t have an active musician we work with regularly in Fairlight or CNCD anymore; we have to outsource. So I dropped a message on Facebook half-jokingly asking if anyone had a spare soundtrack. I’m not sure whether that was a good idea or not but I spoke to Ruairi (RC55), who put me in touch with Tom Wright (aka Stereo Wildlife). He’s produced a beautiful new album and agreed to let us use one of the tracks – and even did a bit of remixing to make it fit the demo. So, music was ready from day 1. This is such a huge bonus when making a demo; it meant we could completely design around it, plan out what scenes we wanted straight away and know they’d fit.

The demo was envisaged as a “small project” – a relatively low budget production. Low budget meaning less development time, fewer resources. Weeks to make by a small team. Frameranger for example is a very “high budget” demo – lots of people, over a year in the making, tonnes of art assets and specifically made effects, and lots and lots of wasted work. This one is very different; there’s only one hand-modelled mesh in the whole thing that’s “rendered” properly (the head at the start and end), although there’s lots of meshes used for other things in the demo. We wanted an effect-led production. The first thing that happened was that Jani designed the numbers scene in Lightwave: creating meshes for each number, placing them in the scene, timing them and making a camera path for the whole lot. Meanwhile I was working on effect development. Then Jani developed the introduction part with the head more or less on his own, and modelled and tweaked the tracks for the fluid parts while I worked on fleshing out the numbers scene with elements and effects. Then we integrated and worked together to finish. With a week or so to go there was a touch of panic and it looked like we weren’t going to get there; but in the end we found ourselves more or less done 5 days before the competition. For once we had time to polish, tweak and optimise. Hope it shows..

As an aside: the Gathering was a great event for us not least because they also held the Scene.org Awards, which recognises the best demoscene productions from last year. We got 11 nominations and after a very rock & roll ceremony full of glitz and fireworks came away with 4 awards: Ceasefire for best music, Agenda Circling Forth for best effects, technical achievement and the cherry on the cake: best demo of 2010. Ooooh. Apparently we just missed out on Public’s Choice by a few points – but hey, no accounting for taste.. ;)

32. Particles. Again?

I’ve realised over time that I’m not really a traditional “democoder”. I’m a graphics researcher who happens to prefer to show his new work off in whatever demo we make next. That probably goes some way to explaining why I do things the way I do: researching and improving on certain areas (like particle systems or fluid dynamics. but not ribbons. bitches.). Some would say that fluids or particles are effects: you “do” fluids for a scene in a demo, then you go “do” something completely different. I don’t subscribe to that. For me the achievement in a demo like this is not to implement fluids: we first used fluid dynamics in a demo 5 years ago. The challenge is to move the field on – to do something new with it that nobody else has managed to do in realtime yet, or not on the same scale. Of course there’s a point where this gets lost on the viewer, and maybe it does just become “nice particles” to the uninitiated.

Although the natural reaction of some people will be “oh, particles again – nothing new!” – this is probably the biggest technical leap we’ve made for a demo since Blunderbuss. Instead of concentrating on the amount of particles and simply using them to render 3D scenes with a few modifiers on top, we concentrated on the cleverness of the particles: the simulation itself and the rendering/shading. In this demo the particles are smart. They’re going somewhere.

Particles are just a primitive like polygons or lines – not interesting in themselves. Creating and rendering a lot of them is easy. Making them do something interesting and look good is a completely different kettle of fish. So lets talk about what we did this time to make particles do something interesting and look good..

93. Smoothed Particle Hydrodynamics (SPH)

SPH is a form of fluid dynamics which uses particles for storing the fluid and the transport of the forces/densities, rather than a grid. This allows you to represent more detail at higher resolution than a grid would allow given the same memory / performance limitations, it’s not limited to a certain area of space, and it makes collisions more practical and it’s a better fit for liquid effects. It’s the scheme used in professional offline packages like Realflow, used for all those nice liquid splashy effects you see in ads and movies – which take hours to simulate, let alone render. Good SPH is for me one of those holy grails of  effects development (like realtime radiosity). The thing is, the quality and scope of effects you can do with it is directly dependent on the number of particles – and so is the difficulty in pulling it off. If you have a few thousand you can make some droplet effects; with 10s of thousands you can make some nice splashes; and with 100s of thousands or millions, you can start to make really amazing running water simulations.

The problem with SPH in realtime is it’s really really hard. The simple explanation of the algorithm is: “take all the particles near my particle and perform some force exchange between them”. The force exchange is easy; the “all the particles near my particle” is a bitch. On GPU it’s even more of a bitch; and in 3D it becomes an order of magnitude more of a bitch.

Other demos have featured SPH before; FR-063 performed it on the CPU with (what looks like) between 1000-10000 particles. The current bleeding edge for 3D SPH in realtime is around 250,000 particles, working on a top end GPU using CUDA and with simple point rendering (and no effects or anything else on top). The current bleeding edge for 3D SPH on DX9 – i.e. with no compute shader / CUDA – is erm.. I dont actually think it’s been done.

The problem is simply the neighbourhood search. You end up with a variable amount of fast-moving particles affecting each particle, where it’s hard to pick an upper bound – so the spatial database is hard to construct. If you solve the neighbourhood search, you can solve SPH.

The demo features up to 500,000 particles running under 3D SPH in realtime on the GPU, with surface tension and viscosity terms; this is in combination with collisions, meshing, high end effects like MLAA and depth of field, and plenty of lighting effects. On DirectX9. It’s fast. Almost impossibly fast. How? We found a new approach to SPH where we can re-form the neighbourhood search term to something much easier to solve on a GPU. Meaning we can, honestly, get very close to what a program like Realflow can do over hours of simulation – but in realtime. And that, for me, is what demo coding (and realtime graphics) is all about.

There are 4 scenes which are directly showing “fluids” in the demo; a couple more using SPH in places for the great quality it has that it makes the particles spread out really nicely rather than bunch together randomly. In each of the fluid scenes it’s basically a load of particles dropped at the top of a very long track, and left to get on with it. The camera captures only a part of the action at any time – the great battle of “design vs showing off code” resulted in something that probably doesn’t completely sell the effect, but it does make something more enjoyable to watch. And that too is what democoding is about..

I thought it’d be nice to show it in isolation, so I put a couple of screenshots and a video above. Aside from that one embedded video – apparently wordpress is a little bitch and won’t let me embed more than one video link into a blog post – you can also check the reverse angles here and here. Those and the above screenshots show an initial test shot we did with 3D SPH – we drop 250,000 particles, and let them run with SPH and collisions against a mesh (handled as a signed distance field). Look, it splashes about and shit like that. All completely in realtime. Oooooooh. If nothing else, being able to run it in realtime makes it a lot easier to tweak. You get instant results – you don’t have to wait for any simulations to calculate. In these days of youtube and the prevalence of netbooks, perhaps high end realtime graphics doesn’t have the same relevance to the audience that it did 15 years ago – but it sure matters a huge amount when you’re actually making something. The benefit to the workflow is huge.

12. Signed Distance Fields

I touched on this for Ceasefire, but it was this production where we finally got them working and used them in anger: the use of signed distance fields for arbitrary collisions (and attraction) with particles. We take polygon meshes, convert them into signed distance fields using distance to triangle measurements and place the results in a volume texture, giving us the means for fast collision ray tests. This is absolutely invaluable when using fluid dynamics because otherwise the particles fly off merrily into space. So we have particles flowing around a head; particles flowing down a track carried by SPH; and particles being blown by a 3d fluid effect into the form of a word. All using signed distance fields.

We used them for a lot more besides particle effects, though. They’ve become an integral part of our rendering pipeline. That will become more apparent the next time we do something featuring a lot of solid 3D.. but they’ve opened up a lot of doors.

One clear example of SDF usage comes in the first “fluid” scene – falling drops collide with invisible words. This also neatly demonstrates the “art vs code” issue – we’re simulating 250,000 particles under SPH running down a long 3D track, and the camera shows a small subsection of those. The collision with the words actually uses two affectors: we used a collision node to make the particles bounce off the 3D words (using an SDF version of the mesh), which worked great – but it means you only see the top of the words. So we added a second affector – a low weighted mesh attractor which pulls the particles towards points on the faces of the mesh. This helped the particles slowly run down and also pulls them in from 3d space towards the words. It also added to the surface tension effect by keeping them attracted to the words even after they fall off the end.

65. Particle Shading

In my original post on my particle system a year or more ago I talked about how we had support for opacity shadow maps for self shadowing on particles. Since Blunderbuss we didn’t actually use that much – we’ve mainly got away with unlit particles, using the shading and lighting from the source meshes. But I’ve been working on some new techniques and had to make use of them..

The major problem with opacity shadow maps is depth aliasing – you only have a limited set of depth samples (16 in my case) for which to represent the scene, and it’s not enough. They tend not to be spread evenly across the particles either. So I tried a few new methods:

252. Volume Shading

This method borrows heavily from slice-wise volume rendering: the particles are sorted in light space by depth, nearest to furthest, and rendered in slices to composite the image. In this case though we only care about the shadow result: the values are written into the per-particle shading buffer used in the final particle render.

The sorted particles are rendered into the shadow map in batches – typically we used 64 batches per particle system. Per batch we additively render the batch particles into the shadow map, then project the shadow map onto the particles into the next batch: the value read from the shadow map is considered the amount of shadow on that particle from particles closer to the light.

This clever bit is, this method doesn’t care about the actual depth of the particle : it only cares about the position of the particle in the sorted sequence. No depth writes are required and transparency is supported without any problems. One additional benefit of the technique is that we can blur the shadow map a bit after each batch, giving a scattering effect. If one had the power to do it and could render one particle per batch, it’d give a perfect shadowing result. As it is, the batch sizes give some slice aliasing.

Unfortunately the slice aliasing was too much of a problem with large sytems and the technique is also a bit too slow – and generates a lot of render target swaps. So I came up with something better..

15. “Stochastic” Shadow Mapping

This isn’t the same as the stochastic shadow mapping paper that was recently presented, but the name makes a certain amount of sense for the effect anyway. The basic idea is something I’ve tried a few times on and off since 2009. The idea is that if your particles don’t overlap pixels in view space, you could render them as solid – using regular shadowmapping and lighting techniques. Of course this is rarely the case in a render – because particle systems rely on lots of small elements overlapping and blending to look solid and nice. However, what if you do render them as single pixels and make them not overlap, and then perform a full screen 2D operation to upscale each point and make them overlap and blend?

We applied that approach to shadow maps generated from particles. The particles are rendered as single points to a very large shadow map; this gives us a reasonable chance that the particles won’t overlap. It’s just like a spatial hash – with a very simple hashing function and no collision handling.. Then, when sampling, we read from the map using a large kernel and sum up the amount of filled pixels which pass the shadow map test to give a shadowing result.

But there’s a twist: in order to improve the quality, cope with hash collisions and reduce aliasing, we perform a temporal reprojection step. When writing the shadow map each frame a random sub-pixel offset is applied to each particle which varies every frame; this means we get a different set of collisions, so different particles become visible each frame. Then when sampling the shadow map we blend the result with the previous frame, so the results adjust smoothly over time. By combining these two things we get a very nice, soft, reasonably alias-free shadow solution which is also efficient to render. No sorting required. The final shadow value per particle is written into a buffer and used at particle render time.

I also experimented with the technique for the actual rendering of the particles to the main frame – rendering single points with Z test and blurring the buffer out, with some per-pixel sorting during the composite, to create softened particles but without the need for a full particle sort. Unfortunately it didn’t give us the visual fidelity we needed; we relied on the blending of particles, the variable sizes and the sprites used. Could be more applicable in a future project though.

536. Meshing (Marching Cubes)

I suppose it’s the obvious step, isn’t it. Democoders love metaballs. Being able to render particles as meshes using metaballs is something we’ve wanted to do for ages because it moves us towards the “liquid” look – the Realflow-style look. We’ve been here before: in Frameranger we rendered around 50,000 metaballs in realtime by generating a potential field, converting it into a signed distance field and raymarching it. Results were promising but not perfect: being able to generate an actual triangle mesh has some side benefits, like being able to post process the mesh and adjust it with tension – something we really wanted to do to get closer to that Realflow look I keep going on about.
Marching cubes gives two issues to solve: generating the potentials, and then triangulating them. We already worked out how to generate the potentials some time ago for Frameranger, although a bit of work was required to scale it up to 250,000 particles.

The second part is more difficult: you need to generate an arbitrary amount of geometry data from that potential field with triangle and vertex counts that change every frame. Naturally, we could quite easily make an implementation which just generates the worst case: treat every cell in the volume as if it was contributing triangles, then write degenerates for the invalid ones. That actually works – but it’s prohibitive for large volumes. One cell can contribute up to 5 triangles, and with a 128^3 volume we’d be looking at 10 million triangles – which isn’t great. 256^3 volumes would effectively be impossible. What we need is a way to only process and send triangles for the cells that are active.

This is problematic because we can’t generate index or vertex buffers on the GPU, we can’t generate drawcalls on the GPU (so we can’t vary how many primitives are rendered on the GPU) and we can’t use the CPU – because the potential field is on the GPU and it’d be far too slow to get it back to CPU. And even if we could, the CPU probably isn’t up to the task of generating the geometry fast enough anyway. And even if it was, we’d have to send all the triangle data back to the GPU again. So we’re stuck with the GPU – and yet we don’t have a way to vary the number of cells we render triangles for.

It seems impossible. However, Gernot Ziegler came up with a nice solution a while ago: histopyramids. This is a way of performing stream compaction on the GPU: it takes a big sparse buffer, and moves all the filled elements to the start of the buffer. A bit like a sort, but much more efficient. This gives us exactly what we need: we generate the (sparse) potential grid and use histopyramid compaction to move all the filled elements to the start. Then we use an occlusion query to count the number of active cells and use the CPU to generate batches which give enough triangles for the count to generate. The actual vertices are generated using a pixel shader and vertex texture fetch is used to read them.

Result!

4. Bokeh Glows

I’ve had this effect on the back burner for a few years but finally got to actually finishing it up.. Bokeh is the term relating to the effect of circular or shaped highlights in a depth of field effect, caused by inaccuracies in the shape of the lens of a camera. Or something. They make DOF look really nice. I’ve tried before by using a really big circular kernel for a regular DOF effect with an HDR input and leaving it at that and it actually does work, but I wanted to see if I could get some shaped bokehs and really overblow it. So I tried something with point sprites.

The basic idea is to work out where on screen bokehs would happen, and render point sprites at those points. I did this using the following method:

• Use those 2d positions to read a blurred version of the original frame. Perform some thresholding to pick out the points which pass. Generate colour values for the points.
• Temporally smooth positions and colours using positions from last frame, apply some attack and decay.
• Render a load of point sprites using vertex texture fetch to read the positions and colours, rendering the sprites to the screen. (With some additional magic to make it look good.)

72. Post Process Antialiasing (MLAA)

This is the first demo since 2009 (Frameranger, in fact) that we’ve released which actually features polygons being rendered as polygons. Happily, time has moved on, and so has our renderer. One of the major bugbears I had with the deferred renderer is lack of antialiasing – but fortunately a whole bunch of post process antialiasing techniques got invented in the last couple of years. MLAA is the technique du jour, and we use an implementation in our renderer. It’s great.

We do two little twists in our version to make it cool: firstly we use a lot of stencil optimisation so only the active edges get the big-ass shader applied to do the actual MLAA (or in fact get any of the process after the edge detect applied). And secondly.. there’s an ugly problem with MLAA in that it actually cocks up quite badly in a certain case. The technique relies on checking for horizontal or vertical edges. But where you have a pixel which is both a horizontal and vertical edge, it messes it up. Which breaks about 1/4 of the diagonal edges you have to deal with, so its pretty noticable. Our oh so clever technique for fixing that is.. do the MLAA twice. The second time we flip the whole image in x and y, then MLAA it and flip it back. Genius huh? .. no? Well, it makes the polygonal scenes look good, and fortunately the stenciled version is so fast the extra hit isnt really noticable.

42. Stereoscopic 3D

We really wanted to do something with 3D for a while, but sadly we dont have any true 3D hardware (*cough* donations please *cough*). We decided quite early on that we were going to go for a pretty much black & white look – so it would actually be feasible to use the good old red / cyan anaglyph method. 3D isn’t as easy as just turning it on, though. It takes some effort to make it work well, give a good effect and not strain your eyes. We tuned it quite carefully and the setup of the scenes really helps – the first scene is slow and quite static so it lets your eyes adjust, the camera movements are quite smooth and in a single direction so they’re easy to track, and so on and so on.

Do watch the demo in 3D, it’s really made for it. We’re going to make a proper HD 3D video with left & right splits soon for those with real 3d setups.

End

I guess what’s interesting for me about this demo is that it was so much easier to make than many we’ve done. It just kind of came together; we started early enough, we got the music at the start, we didn’t have any major problems, nobody disappeared or dropped out, everything showed up on time, we didn’t completely overstretch ourselves and come up with some ideas that couldn’t be done, and we had time at the end to go over it and tweak and polish things, and we’re really happy with how it turned out. It’s like the way it’s supposed to go but never does. It doesn’t work for everyone (not very bombastic, you see) but it seems the people who got it really got it and like it, which is what matters. Maybe we’ve actually cracked it.. or maybe next time’ll be a royal screwup. Have to wait and see..