c0de517e

Replacing google share: zootool

2012-01-19T11:11:00.000-08:00

There are quite a few bookmarking services on the web today, and I never felt the need to find one until Google shut down its handy reader share feature. So, kinda randomly I picked Zootool and I've been working on sharing some links for a few weeks, so now there is a bit of content and I can publish its feed. Let me know if there are problems, otherwise I'll stick to this one to share my finds in the "rendering et al." category, at least until Reeder does not add support for Google+ :)

http://zootool.com/feeds/user/c0de517e/type:all

Prototyping frameworks

2012-01-18T17:01:00.004-08:00

Every now and then, I look around for prototyping frameworks for my rendering work. I always end up with very little but maybe I'm too picky or too lazy. Here are some I've found:

I actually use:

FXComposer, both 2.5 and 1.8 (Nvidia does not host it anymore, so you have to google it) as they both crash/have problems in different ways. In particular, 2.5 seems to have problems clearing rendering targets (I use a pass with ztest always to clear), 1.8 crashes in some situations. 1.8 is also nicer for a programmer, but 2.5 is fairly usable. I use SAS and NVidia has some documentation (finally!) about it, to script render passes. In theory both also support proper scripting, but the documentation is thin. A few times when I wanted to look inside FXC 2.5 I used something like ILSpy or .net reflector to delve in the undocumented parts (that's to say, almost everything).
Wolfram's Mathematica. I wrote a couple of articles on this blog about it, it's great and I love it, I love the language, it's not what you would expect if you're a mathematician but for a programmer is pretty neat (well at least, if you like lisp-ish things, which you should, syntax apart)
SlimDX or SharpDX, to tell you the truth I mixed them up a few times, the names are similar. Bottom line, a DX wrapper for C#, I love C#. SharpDevelop if I don't have an updated visual studio which supports the last .net framework.
Processing. I wrote an article on the blog about using it with Eclipse for live-coding, it's neat, it's simple, it has a ton of libraries at this point, even to do 3d stuff and shaders but I use it mostly for 2d prototypes.
3D Studio Max. It has an horrible support for shaders (at least it used to, and I suspect not much has changed since) and I never loved it (I love Maya even less though), but I used to know it (six years ago or so) and know maxscript, so I ended up prototyping in Max a few things. It can be handy because you can obviously manipulate meshes all the ways you want and define vertex streams and visually paint attributes on meshes. You can't really control the rendering passes though, so doing non-surface shaders or stuff other than the most basic post-effects is hard. Nowadays I don't use it much if at all.

Seem promising:

PhyreEngine, if you have access to the Sony stuff... Might be a bit overkill as it's a fully fledged engine, so the learning curve is not so steep per se but there are tons of examples.
Erik-Faye Lund published the sources of his "very last engine ever" which is used in a bunch of great demos (as in demoscene) I didn't have the time to look into it much yet, but the name sounds great!
Hyeroglyph 3, it's the 3d engine that "ships" with the Practical Rendering and Computation with DirectX11 book (which is nice). It still has a bit more things that I'd like to (more of an engine than a framework) but it's nice...
Cinder looks still a bit young, it has many nice things but it lacks some other which I would consider "basics". I feel the same about openFrameworks and to me Cinder looks nicer. Plus I don't love C++ :)
Humus Framework 3. This is great, it's simpler than a full fledged engine, it's easy to read and it has tons of examples and Humus is notorious for his graphic demos, which all come with sourcecode and were made with his framework!
Intel's Nulstein. I didn't spend much time looking at this, Intel "makets" it as the most bare-bones, next-gen engine source possible. Which sounds good, but from the little I saw, it's still not simple enough for my purposes (I really would like not needing to learn anything :) otherwise I could just write my own stuff). Also Intel with its Visual Adrenaline initiative seems committed to providing great support, articles, sourcecode and tools for graphics development.
VVVV. It's a node-based graphic thingie. Which would seem like the least suitable thing for rendering prototypes, but it supports shaders, and it supports "code" nodes where you can write C#, so it might be worth a try...
OpenCL Studio looks great to me but I didn't have the time yet to test it out properly
Python. Lots of people love it and there are tons of libraries, environments, wrappers and so on. The latest someone told me about is python(x,y) for scientific computing. I don't use python. Maybe I should...

What I'd love? I'd love to have a very simple wrapper, just slightly more high-level than the base API to get me some of the most common stuff already coded (i.e. a camera system, parameter manipulation, mesh management and loading...), and I'd love if this wrapper was embedded in a live-coding language, like Lua with script hot-swapping, and if it supported hot-swapping of all the assets as well (shaders, textures etc...). I always think I should code such a thing sooner or later but I never find the time to do it :(

Videogame photography

2012-01-17T23:58:00.009-08:00

For some reasons, videogames and photography are more linked that people might suspect. In my career I've noticed that many if not most of the top engineers I know, also dabble quite a bit in photography (well, I'm biased as I'm a rendering guy and my friends tend to be in the rendering circle). That includes myself (NFSW). But there is more...

Some notable videogame photographers are Robert Overweg, Duncan Harris (Dead End Thills), Francois Soulignac and Iain Andrews, but I'm sure there are way more out there, if you have links to share please comment on this post.

Robert's work focus on exploiting the limits and bugs of games to create artistic fine art series (which is not uncommon either), while Dead End Thills and Iain Andrews are more "conventional" screenshot artists: games get hacked to gain more control or improve quality and the end results are quite amazing!

Professional photographers also dabble in the media time to time. Of course, the main attraction seem still to be the players, and not the games, as in these series by Phillip Toledano and Robbie Cooper (also here), but there are examples of photographers leaving their cameras for the PC to record for example this Skyrim timelapse (was this inspired by a similar video on Eurogamer? or viceversa?) oe using videogame equipment in their photography.

Videogame references is photography are of course countless, like this fashion editorial in Neon Magazine.

Lastly even gamers are becoming photographers, not only in games that specifically feature this profession in their characters (Fatal Frame series and Beyond Good and Evil come to mind but I'm sure there are more), but by twisting game rules.

Virtual war photographers enter multiplayer shooter games with the intent not to kill but to report (and not die), like Brunet Thibault (who also enjoys other forms of videogame photography). There are staring to be quite a few examples of this, which is probably going to piss the other players though, it would be great if modern FPS started to support this option (spectator modes are sort-of there, but you can't die and thus don't really "simulate" the role, there is at least one game that is all about war photography, but it doesn't seem on the contrary to allow other roles, so it's still not quite the same).

A notable game which supports photography (and fashion!) without being centered on photographer is Second Life. These guys go so crazy that they actually photoshop their work! Eve Online is another game worth mentioning, not only its avatar creator is amazing, but taking your avatar picture puts you in a (simple) virtual photography studio and the results are fairly decent.

If only Red Dead Redemption or Alan Wake could get more love... Unfortunately being console only titles (I still hope for an Alan Wake on the PC), they lend themselves less to photography, but at least for the former there are a couple of amazing videos out there, including a short film made by John Hillcoat...

Current-gen DOF and MB

2012-01-04T15:58:00.001-08:00

I was thinking to publish this somewhere else, but Crytek at the last Siggraph already disclosed a technique that is close to this one, so at the end I guess it's not worth it and I'm going to spill the beans here instead.

Depth of field and motion blur, from a post-processing standpoint, look similar. They both are scattering filters, they both need to respect depths. So it makes sense to try to combine both effects in the same filter. And if you think about doing it that way, you're almost already done, it's quite obvious that it's possible to combine the DOF kernel with the MB one by skewing the DOF towards the motion-blur direction.

This is what Crytek does, they use a number of taps, their filter is circular and they just transform this circle with a basis that has the motion-blur axis as one vector an a perpendicular one, scaled by the DOF amount, as the second. It's pretty straightforward, the only thing we really have to take care in this process is regions were the motion blur is zero and thus we won't have the required first axis, which might be a bit of a pain.

As you know, doing these effects in post is an approximation, so when I look at these problems I always think in terms of the "correct" solution first and use that as a background in my head to validate any ideas I have.

In this case what we really want to do is to sample some extra dimensions in our rendering integral, temporal in the case of the motion-blur and spatial, on the camera's film plane, in case of the DOF. This leads quite naturally to raytracing, but thinking this way is not particularly useful as we know we can't achieve comparable results to that, we don't have in post the ability of sampling more visibility (and shading) than what we have already in our z-buffer, so we should look instead at a model that gives us "the best" of what we can do with the information that we have (to be fair, there is in literature an method which subdivides the scene in ranges with multiple z-buffers and color buffers, and then ray-marches on these z-buffers to emulate raytracing, which I found pretty cool, but I digress even more). With only a z-buffer and its color buffer, that would be scattering.

You can imagine that as placing a particle at each image point, stretched along the axes we just described, with an opacity that is inversely proportional to its size, and then z-sorting and rendering all these particles (and some PC ports of some games did that, having a lot of GPU power to spare, in general placing some particles is not a bad idea especially if you can avoid having one per pixel but selecting the areas where your DOF highlights would be more visible). This, plus a model to "guess" the missing visibility (remember, we don't have all the information! if an object is moving, for example, it will "show" some of the background behind it, which we don't have, so we need a policy to resolve these cases) is the best we can do in post and should guide us in all the decisions we make in more approximate models.

Ok, going back to the effect we are creating, so far it's clearly a gathering effect. We create a filter kernel at each pixel, and then we gather samples around it. Often, this gathering only respects the depth sorting, we don't gather samples if their depth is farther from the camera than the pixel we're considering. This leads to no missing visibility problems, we assume that the surface on the pixel we're considering fully occludes everything behind it, but it's not really "correct" if we think about the scattering model we explained above.

A better strategy would gather a sample only if the scattering kernel of the surface at that sample point would have crossed our pixel, this strategy is what I call sometimes "scattering as gathering". Now the problem with this is that unless we know that our scattering kernels have a bounded size, and for each pixel we sample everything inside that size to see if there is something that could have scattered towards it, we will miss some scattering, but unfortunately doing so requires a large number of samples.

In particular, as we size our gather kernel using the information at the surface point of the pixel we're considering, we easily miss effects where a large out of focus object scatters on top of some in focus background pixels, for which the gathering kernel would be really small.

Now there are a variety of solutions for that. We could for example fix a minimum size for our gathering kernel, so in the in-focus areas we still "check" around a bit, and we might end up gathering nothing if we found nothing that scatters towards us. This sort-of works but still won't handle large scattering radii. We might think to cheat and get more samples by separating our gathering filter into two passes, each filtering along a line (axis).

This works well for gathering, even when the filter is not really separable (DOF and MB are not, as the kernels, even if we use a gaussian or square one, which are separable, are sized by a function which varies per pixel and is not itself separable) the artifacts are not noticeable, but if we push this with a scattering-as-gathering logic it starts to crumble as the second pass does not have the information of where the first-pass gathering took its samples from, so it can't decide if these samples would have scattered towards a given location and it can't even separate the samples anymore. Digression: In the past I've solved this by doing DOF in two passes, using separable gathering in a first pass while detecting the areas where it fails and masking them using early-z to then do a second pass on them with a large gather-as-scatter filter.

So what we can do? The solution is easy really, we can write our basis vectors to a buffer (which is required anyways if you want to consider MB due to object movement and not only camera movement, the former can't be computed with the information we have from the colour and depth, we need to render the motion of the objects somewhere) and then apply a fixed radius gathering-as-scattering filter there, which as it's only searching to expand the subsequent filtering radii and not sampling colour, can be done with fewer samples without causing too much artifacts, pretty much as "percentage closer soft-shadows" do.

So far, Crytek does something along the very same lines. The twist where the effect I crafted diverges from theirs is that I still employ a separable filter to compute the final blur, instead of using some taps in a circle. The first time I saw this done for MB and DOF was in EA's first Skate game, so it's nothing particularly novel. Skate's implementation though was "driven" by DOF, the motion blur was only added for the camera and it was only present if the DOF was there too (at least, afaik).

Extending this to behave well with the two effects separately requires computing the "right" gathering weights, or as I wrote above, reasoning about the scattering. Also, once you get the ability of doing motion blur without DOF, you will notice that one of the two blur passes will do nothing in areas of pure MB, as the second axes will have zero length (but you are still "paying" to sample N times the same area...). To avoid that waste, I filter along two diagonals, in case of pure MB these coincide but are both non-zero, so we get a bit better filtering for the same price.

I don't think you can do much better than this on current-gen (without proper support for scattering) but I'd love to be proved wrong :) Example code below (it doesn't include many details and it doesn't include the axis scattering pass, but the "important" parts are all there):

// --- First pass: Compute camera motion blur and DOF base axes
float depth = tex2D( DepthSampler, TexCoord );
float2 MB_axis = ( currentViewPos - previousViewPos ) * MB_MULTIPLIER;
float DOF_amount = ComputeDOFAmount(depth);
float MB_amount = length(MB_axis);
float DOF_MB_ratio = DOF_amount / (MB_amount + EPS_FLOAT);
float2 DOF_axis = MB_axis.yx * float2(1,-1) * DOF_MB_ratio;
MB_axis *= max(DOF_MB_ratio, 1.f);
// Compute the 2x2 basis
float4 axis1xy_axis2xy = DOF_axis.xyxy + MB_axis.xyxy * float4(1.0.xx, -1.0.xx);
// Make sure that we are in the positive x hemisphere so the following lerp won't be too bad
//axis1xy_axis2xy.xy *= sign(axis1xy_axis2xy.x);
//axis1xy_axis2xy.zw *= sign(axis1xy_axis2xy.z);
// We have to take care of too small MB which won't be able to correctly generate a basis
float MB_tooSmall = 1.0 - saturate(MB_amount * MB_THRESHOLD - MB_THRESHOLD);
axis1xy_axis2xy = lerp(axis1xy_axis2xy, float4(DOF_amount.xxx, -DOF_amount), MB_tooSmall);

// --- Second and third passes: separable gathering
half2 offset;
if(is_first_separable_blur) offset = GetFirstAxis(TexCoord);
else offset = GetSecondAxis(TexCoord);
half amount = length(offset);
half4 sum = tex2D(ColorSampler, TexCoord) * FILTER_KERNEL_WEIGHTS[0];
half sampleCount = FILTER_KERNEL_WEIGHTS[0];
half4 steps = (offset * TEXEL_SIZE).xyxy;
steps.zw *= -1;
for(int i=1; i < 1+NUM_STEPS; i++)
{
half4 sampleUV = TexCoord.xyxy + steps * i;
// Color samples
half4 sample0 = tex2D(ColorSamplerHR, sampleUV.xy);
half4 sample1 = tex2D(ColorSamplerHR, sampleUV.zw);
// Maximum extent of the blur at these samples
half maxLengthAt0;
half maxLengthAt1;
if(is_first_separable_blur)
{ maxLengthAt0 = length(GetFirstAxis(sampleUV.xy)) * (NUM_STEPS+1);
maxLengthAt1 = length(GetFirstAxis(sampleUV.zw)) * (NUM_STEPS+1); }
else
{ maxLengthAt0 = length(GetSecondAxis(sampleUV.xy)) * (NUM_STEPS+1);
maxLengthAt1 = length(GetSecondAxis(sampleUV.zw)) * (NUM_STEPS+1); }
half currentLength = amount * i;
half weight0 = saturate(maxLengthAt0 - currentLength) * FILTER_KERNEL_WEIGHTS[i];
sum += sample0 * weight0;
sampleCount += weight0;
half weight1 = saturate(maxLengthAt1 - currentLength) * FILTER_KERNEL_WEIGHTS[i];
sum += sample1 * weight1;
sampleCount += weight1;
}
return sum / sampleCount;

This is golden...

2012-01-04T11:15:00.000-08:00

This video, by a Bill Dally (NVidia) about chip architectures is so great it deserves its own post: http://mediasite.colostate.edu/Mediasite/SilverlightPlayer/Default.aspx?peid=22c9d4e9c8cf474a8f887157581c458a1d#

It probably will tell you thinks you already know, but even if it does, it's worth to see it.

The three skin rendering horrors you want to avoid

2011-12-12T14:50:00.022-08:00

It's 2011 and still most games come with some truly horrific skin rendering. I don't think skin is that hard to get decent, but you have to understand what's important and how to hack it in your lighting/rendering model.

This article won't focus on the state of the art skin rendering techniques or on the state of the art acquired data for skin, these things are really important, fundamental, go ahead and google them, there's plenty to read and it's not hidden knowledge at all (start with Debevec and Jensen).

What I want to do here is just a write up of what I've learned about skin in my experience, and some of the common technical mistakes I've seen.

1- Bad tone

Fallout 3: Radiation really does some numbers on your skin...

By far, the worst offender. Nailing the right colours from the day to the nightside of the skin is the most important, and most often neglected things. And yes, it is mostly (but not only) a matter of subsurface scattering, as most of the skin diffuse lighting comes from it, but it's not about complicated techniques. It's about understanding what happens, what's important to model and how to hack it in whatever lighting model you have in your rendering.

Model: Commander Shepard @ Bioware

Model: Charlotte Free @ IMG/NY

You can get great skintones with even the simplest hacks on top of your basic lambert, and horrible tones by badly tuned texturespace or screenspace diffusion techniques. Penner's pre-integrated skin shading is the current champion of the lambert hacks, but truth is that some sort of ramp on top of your lambert is everything you need (that's to say, you can most often get decent results without even bothering to approximate the geometry with the curvature, as Eric does) if you nail the right hues! Also, remember that scattering will affect all the skin shading, you might want to "ramp" the edges of the shadowmaps (again, Penner's paper covers that) but it's actually often even more important to properly blend AO on skin, especially if you're using a normal-aware SSAO which is capable of capturing very fine details, a straight multiply will create some really questionable gray patches. Again, even simple hacks, like adding a slight constant colour to your SSAO, will go a long way towards creating more organic skin.

Debevec, Digital Emily, Acquired Diffuse

Give your artists the right tools! It's fundamental to be able to tune the skin while comparing the rendered results with some reference material (put an image viewer and color sampler in game!), it's fundamental to tune colors after fixing your tone-mapping operator, and it's fundamental to understand _all_ your sources of color: shading model, parameters, textures.

Debevec, Efficient Estimation of Spatially Varying Subsurface Scattering Parameters

Textures are particularly tricky: remember that skin specular is white (monochromatic), skin diffuse maps really are tied with the subsurface scattering model you implement in the shader, in theory the epidermis is pretty gray, all the colour comes from the blood vessels below the surface. In practice, especially in the realtime rendering world, we mix together all the skin layers and more often than not the problem with skin textures is that they are not saturated enough (consider that the white, additive specular sheen should be responsible for some of the loss of saturation in the final rendering) and have too uniform hues (skin hue shifts quite a lot in many regions, i.e. elbows, joints, hands and feet, and usually presents quite a few blemishes)

2- Bad detail

Skin detail does not come from diffuse! That's quite obvious as we said that diffuse lighting is mostly due to subsurface scattering.

Debevec, Digital Emily, Specular

Detail is all in the specular layer, ideally, you would need two different normalmaps for diffuse and specular (actually, Debevec achieves good results rendering with different normals for each of the RGB channels of the diffuse), fetching the same at different miplevels (bias) is often good enough and in a pinch, even just using the geometric normals for diffuse (or a lerp between the normalmap normal and the geometric ones) is way better than using the specular normals.

Skin pores are a really, really fine detail which is both difficult to capture and easy to get in the way of your BRDF. Most of the times, it's better to have the pore details as self-occlusion in the specular map than in the normalmaps (which being a derived measure will require higher resolution to capture the same amount of detail). Also, you either have to model skin for a given viewing distance, or you'd better consider that at all but the closest distances the pores will/should fade in your miplevels and their scattering effect should be modeled by varying the specular exponent (broadening it with the distance, you can use ddx and ddy to estimate the filtering width, our use cLEAN), there is no way to tune the specular with a constant exponent that will yield both the right amount of detail up-close and the broad sheen you see from a distance.

Peach fuzz on dry skin is another geometrical source of specular detail perturbation, you might want to model it by adding noise at the grazing angles if you need that amount of detail.

3- Bad volume

Fight Night Champion
When it works is one of the few games that captures volume right...

Skin texture luminosity is often quite uniform, and in caucasians quite light too, making skin shading more a matter of volume and shape than texture. Three elements here are important: specular, ambient and normals.

Normals are tricky because skin is often... skinned. Conventional bone skinning yields bad geometric normals. I've already blogged about that so I won't repeat myself here.

Ambient is "easy" but really important, a simple contant won't do the job, it just flattens everything, even a simple hemispherical (top/bottom, a lerp between two colors based on the geometric normal y component, boils down to a single madd) is way better. Observe and understand. Look at your references, understand what's important, understand your visual errors and their sources. The "correct hack" really depends on the context, if your environment is quite fixed, like in a sport or racing game you can model ambient with different components: a sky layer which does not depend on the position plus a ground reflection that maybe fades on the model based on the distance to the ground and so on...


Shadows, AO, Ambient model... All coming together nicely

Ambient occlusion is fundamental, ideally you'd want to have some directionality in the occlusion: bent normals, or encoding the occlusion in some base. Really "occlusion" is fundamental for volume. Each light component (ambient, diffuse, specular etc) should be occluded in a reasonable way.
Again, there are many ways, technically to achieve that, I won't delve into any detail because it depends on the context, you might want to agument SSAO to encode directional information (as I already said), or precompute occlusion at the vertices (SH or similar), cast rays in screenspace (i.e. screenspace reflection occlusion is easy, especially at the all-important fresnel angles, for rim occlusion) or even simpler hacks.
The important message here is that volume requires occlusion, look at your rendering and understand your visual defects, look for light leaks, compare with real-world references and acquired data and craft your own solution! Sometimes the actual technical answer is dead simple (an example - one of the improvements on Champion over Round 4 was a "downgrade", we went from VSM to simple PCF filtering because VSM even if on average are "nicer" were not able to capture the very important occlusions on the face due to precision issues, the nose shadows and the eye-sockets. Going back to PCF gave us "worse" shadows but way better faces!)

Phong, modified to behave better with Schlick. No lousy round highlights!

Achieving proper specular on skin is probably one of the trickiest parts as unfortunately, right now in realtime rendering we have either the choice of employing nice material models on simple analytic lights. It's well-known that the Kelemen/Szirmay-Kalos model fits the skin acquired data very well and if you can afford it it's probably the way to go, especially if you lighting is not very complex (i.e. outdoor, harshly lit scenes).

Unfortunately, in many contexts we want to use some image-based lighting approach (baking reflection cubemaps), and that restricts the BRDF filtering we can employ pretty much to Phong, and straight Phong is really, really bad on skin.

Again, it's important here to observe and understand what is important, what qualities we want to model or hack. The specular sheen, other than affecting the saturation and the skin detail, as we discussed already, is important for the perception of the shape. Human vision can't really distinguish the effects of the various lights in the specular, we can't relate well the scene lighting with the specular hightlights shapes.

What we do with the specular is to understand the surface shape and material, so what's important is more to model the general shape of the specular, than the link between the specular and the actual scene lights.

That's why we can often ignore accurate specular occlusion (and just modulate all the specular light with our single shadowmap and omnidirectional ambient occlusion... or even "worse" multiply some of the diffuse product into phong, not ideal, but decent) and we can often disregard accurate light positions and use reflection cubemaps. And that's why what's really important is the shape of the specular sheen, and you get quite some latitude in hacking that in, as we can't really "see" these hacks as long as the final shape behaves "well".

What phong get really wrong is the highlight shape, which is too uniformly circular, and the lack of frensel. You can "break" the highlights using an exponent map (which should always be present, but it's especially important with Phong), adding fresnel is not easy as Phong simply does not reflect enough light at the grazing angles, and thus even multiplying with a decent approximation, like Schlick's, does not usually yield great results, it's better to use fresnel to (also) drive other parameters in the Phong model instead, either bending the reflection normal, or lowering the exponent.

Conclusions
I probably should have included a fourth horror "bad behavior", as most CG humans (all? in videogames) really lack the proper "fleshiness" in terms of animation and behaviour, but that's a harder problem to tackle which is probably outside the realm of realtime animation, at least for most games where you have many characters on screen and you can't afford having 300 bones in each plus some soft-body system and some UV relaxation and so on...

Really what I wanted to point out are some simple things that every game should do, aspects which can most of the times be fixed to a decent degree with really cheap hacks, but do require an understanding of how the underlying physics work and how our perception works to then focus on what really matters visually.

Then the actual techniques really depend on your project, for those of you that really want to delve into some actual tech, an example of a possible solution for skin rendering is here.

Addendum: Hair, Eyes, Teeth, Ears
Hair is a huge pain, on this generation we're stuck with hair cards. Cards don't sort well, are hard to shadow and hard to shade. My usual suggestion is to avoid spending too much time on it, it's not really worth it.
Shading usually employs Kajiya-Kay for specular (card will require a tangent shifting map to avoid having uniform highlights on a card, which are ok only for very straight hair), some wrap lighting for diffuse and a way of getting good rim.
For sorting, the usual solution is to go multipass, first doing alpha testing with z-writes, and then alpha blending with z-testing. You can somewhat pre-sort the triangles from in to out, some games do the alpha blending pass in two phases, first the backface cards then the frontfacing ones. It's decent, but defects will still be there, even sorting per triangle, if you can, doesn't solve them all as cards often intersect. I honestly think a better solution is to just alpha test and rely on alpha to coverage, if you can. This is what Fight Night Champion did by the way (Round 4 did a three-pass alpha blending).

Eyes are way, way more important. Shading is not that hard, specular/reflection is done with a cubemap usually, diffuse would need SSS but again some wrap is good, on Fight Night we computed a second set of normals to match the diffuse lighting with the skin around the eyes, probably a bit overkill. Behavior is hard, and getting it right is crucial! On the shading side, one trick I used is to disable the mipmaps, and the bilinear filtering on the reflection cube, to get aliasing far away that in turns create some shimmering in the small highlights. But most of it is in the animation, and some procedural techniques are to be employed.
Ears have lots of translucency, again, it can be faked easily, I won't lose my sleep there. For teeth, the important thing is to use the right occlusion and avoid them to become too bright, shadows from the mouth and lips onto the teeth are very hard to cast so again, you'll have to do some faking.
Bottom line, for each of these elements there are some easy hacks, but it's important to consider them. Once you understand what each of them needs and what physical things are important to model for each (ear: translucency, eyes: SSS and behavior, teeth: occlusion etc...), the actual techniques are "easy" or can be easily faked to a decent standard.

Google reader share

2011-11-28T15:09:00.000-08:00

So, some of you have noticed that my google reader share is dead (not updated anymore). That's not me being lazy, but google being a bit evil and trying to shove google+ down our throats by killing the old facilities instead of integrating plus into the existing stuff. So, there is no reader share anymore, and my iPad newsreader (reeder), which was responsible of most of the share posts, does not support google+ yet. Stay tuned.

Photoshop scripting - Cleartype for images

2011-11-25T13:45:00.002-08:00

Left: bilinear, Right: bilinear with "cleartype"
note- the effect is configured for a "landscape" RGB pattern LCD

I always wanted to learn how to script Photoshop (what I learned is that it's a pain and the documentation sucks...), so yesterday I started googling and created a little script to emulate cleartype on images. Here is the source (it assumes that a rgb image is open in PS):

// 3x "cleartype" shrink script

var doc = app.activeDocument;

var docWidth = doc.width.as("px");

var docHeight = doc.height.as("px");

doc.flatten();

// let's go linear RGB

doc.bitsPerChannel = BitsPerChannelType.SIXTEEN;

doc.changeMode(ChangeMode.RGB);

// now that's a bit tricky... we have to go through an action, which has binary data... which I'm not sure it will be cross-platform

// it works on Photoshop CS3 on Win7...

function cTID(s) { return app.charIDToTypeID(s); };

function sTID(s) { return app.stringIDToTypeID(s); };

var desc6 = new ActionDescriptor();

var ref5 = new ActionReference();

ref5.putEnumerated( cTID('Dcmn'), cTID('Ordn'), cTID('Trgt') );

desc6.putReference( cTID('null'), ref5 );

desc6.putData( cTID('T '), String.fromCharCode( 0, 0, 1, 236, 65, 68, 66, 69, 2, 16, 0, 0, 109, 110, 116, 114, 82, 71, 66, 32, 88, 89, 90, 32, 7, 219, 0, 10, 0, 22, 0, 19,

0, 25, 0, 58, 97, 99, 115, 112, 65, 80, 80, 76, 0, 0, 0, 0, 110, 111, 110, 101, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,

0, 0, 0, 1, 0, 0, 246, 214, 0, 1, 0, 0, 0, 0, 211, 44, 65, 68, 66, 69, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,

0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,

0, 0, 0, 9, 99, 112, 114, 116, 0, 0, 0, 240, 0, 0, 0, 50, 100, 101, 115, 99, 0, 0, 1, 36, 0, 0, 0, 101, 119, 116, 112, 116,

0, 0, 1, 140, 0, 0, 0, 20, 114, 88, 89, 90, 0, 0, 1, 160, 0, 0, 0, 20, 103, 88, 89, 90, 0, 0, 1, 180, 0, 0, 0, 20,

98, 88, 89, 90, 0, 0, 1, 200, 0, 0, 0, 20, 114, 84, 82, 67, 0, 0, 1, 220, 0, 0, 0, 14, 103, 84, 82, 67, 0, 0, 1, 220,

0, 0, 0, 14, 98, 84, 82, 67, 0, 0, 1, 220, 0, 0, 0, 14, 116, 101, 120, 116, 0, 0, 0, 0, 67, 111, 112, 121, 114, 105, 103, 104,

116, 32, 50, 48, 49, 49, 32, 65, 100, 111, 98, 101, 32, 83, 121, 115, 116, 101, 109, 115, 32, 73, 110, 99, 111, 114, 112, 111, 114, 97, 116, 101,

100, 0, 0, 0, 100, 101, 115, 99, 0, 0, 0, 0, 0, 0, 0, 11, 67, 117, 115, 116, 111, 109, 32, 82, 71, 66, 0, 0, 0, 0, 0, 0,

0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,

0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 88, 89, 90, 32, 0, 0, 0, 0, 0, 0, 235, 194, 0, 1, 0, 0, 0, 1, 65, 50,

88, 89, 90, 32, 0, 0, 0, 0, 0, 0, 97, 15, 0, 0, 36, 77, 255, 255, 255, 232, 88, 89, 90, 32, 0, 0, 0, 0, 0, 0, 103, 37,

0, 0, 220, 208, 0, 0, 5, 29, 88, 89, 90, 32, 0, 0, 0, 0, 0, 0, 46, 162, 255, 255, 254, 227, 0, 0, 206, 39, 99, 117, 114, 118,

0, 0, 0, 0, 0, 0, 0, 1, 2, 0, 0, 0 ) );

desc6.putEnumerated( cTID('Inte'), cTID('Inte'), cTID('Clrm') );

desc6.putBoolean( cTID('MpBl'), true );

desc6.putBoolean( cTID('Dthr'), false );

desc6.putInteger( cTID('sdwM'), 2 );

executeAction( sTID('convertToProfile'), desc6, DialogModes.NO );

doc.backgroundLayer.applyGaussianBlur(0.75); // limit the frequency a bit to avoid too many fringes

doc.resizeImage(

UnitValue(docWidth, "px"),

UnitValue(docHeight / 3,"px"),

null,

ResampleMethod.BILINEAR // To-do: box filter (mosaic + nearest)

);

var unitValue = UnitValue(1, "px");

// RGB pattern, note that the nearest resize will take the center pixel, that's why red shifts by one and not zero

var redLayer = doc.backgroundLayer.duplicate();

redLayer.applyOffset(unitValue, 0, OffsetUndefinedAreas.WRAPAROUND);

var greenLayer = doc.backgroundLayer.duplicate();

greenLayer.applyOffset(-unitValue, 0, OffsetUndefinedAreas.WRAPAROUND);

var blueLayer = doc.backgroundLayer.duplicate();

blueLayer.applyOffset(-unitValue*2, 0, OffsetUndefinedAreas.WRAPAROUND);

doc.resizeImage( // Resize to "select" the RGB columns in the various layers

UnitValue(docWidth / 3, "px"),

UnitValue(docHeight / 3,"px"),

null,

ResampleMethod.NEARESTNEIGHBOR

);

//var col = new SolidColor(); col.rgb.hexValue = "FF0000"; redLayer.photoFilter(col, 100, false);

redLayer.mixChannels ([[100,0,0,0],[0,0,0,0],[0,0,0,0]], false);

greenLayer.mixChannels ([[0,0,0,0],[0,100,0,0],[0,0,0,0]], false);

blueLayer.mixChannels ([[0,0,0,0],[0,0,0,0],[0,0,100,0]], false);

redLayer.blendMode = BlendMode.LINEARDODGE; // add!

greenLayer.blendMode = BlendMode.LINEARDODGE; // add!

// blue is the base layer

doc.flatten();

// let's go to 8bit sRGB

doc.convertProfile ("sRGB IEC61966-2.1", Intent.PERCEPTUAL, true, true);

doc.bitsPerChannel = BitsPerChannelType.EIGHT;

Silvio Berlusconi

2011-11-08T15:30:00.005-08:00

This is obviously going to be off-topic with the rest of the blog... If you landed here for the first time, this is a rendering related blog and this article is an exception to the rule.

As the Europe's and Italy's financial crisis deepens, news crossed the wire today that prime minister Berlusconi vowed to resign. I see many people asking around the world how it was possible that this happens only now, how did Berlusconi manage to be in power for seventeen years even after countless scandals and accusations. While in general I think it's not surprising, and the eight years of Bush administration could be served as an example, I'd like to try to explain what's peculiar about Italy's situation (of course, in my point of view).

Also, of course, I will make generalizations in the following. In no way I want to express that this is applies to everyone and everything, that should be pretty clear.

Survivalists

One way or another, we keep going on. This is by far what I believe to be the deepest of our problems. We don't care much about our society, we avert our eyes and keep going on, everyone trying to find a hole in which to live their lives.

We are masters in bending (if there is a profit to be made) or ignoring laws. Even our image outside the country is that of creative, chaotic individuals (at best), known for being obsessed about family and our own small individualities.

We are not socialists not liberals, we are just driven towards what can get us a gain tomorrow morning. Mind you, to a degree this happens everywhere, but it's not a defining quality of a population quite as it happens in Italy, where is deeply buried everywhere, from how people live their lives to how companies make business.

Even our economy, made mostly of small or family owned companies, with our comparatively large private savings and our huge public debt, is a testimony of this mentality.

I can't tell why this is the case, we are a young republic and unification was not a smooth deal, but we don't believe in society. Berlusconi is the embodiment of all this, and I don't know how much he was just "born this way" or how much he knowingly acts to please, to be popular, but he is certainly great in leveraging such sentiments. His political message, either explicitly or implicitly has always been "vote for me and I'll let you live your lives without control", "you don't need to be responsible for your actions", "I was successful, don't ask me how, you want to be like me, I won't ask you how"...
Berlusconi was never a left or right wing politician, he is obsessed about communism and certainly sees Italy's leftists as pure evil, but his actions are not the ones of a liberal. Among other things, he's remembered for saying openly at an entrepreneurs' convention that companies without off-shore operations were not led smartly, and that evading taxes (one of Italy's chief problems) was morally sound in a country like ours (in which taxes are too high). He didn't them proceed to lower the taxes and impose strict controls to have everyone paying the right amount, or to incentive competition and freedom of enterprise.

In the first months of his government he proceeded in the opposite direction, abrogating liberalization laws that were passed by the previous government, loosening controls over financial transactions and not reducing a single tax. Not touching established interests, not reducing bureaucracy, but just allowing people to just screw each other more freely.

Shameless

Berlusconi is probably not the worst individual in Italy's history. Corruption and misgovernment were always there and can be even tracked to the same underlying sentiment. The first republic created a massive debt because political parties were quite literally buying votes by flushing enormous amounts of public money into all kinds of public ventures, creating hundreds of thousands of "fake" jobs, public employers who were pretty much useless. But it came down onto his knees when it was found that politicians also used public money to fund their own parties. There was corruption, but there is was still a sense of shame.

Berlusconi took inspiration from this and pushed it one step further, he was proud of his tricks, every trial he escaped by passing laws in his own favor, every lie and joke he said made him look "smarter", more successful. It's not that the vast majority of Italians do not know he was a thief, that's also why many of his voters were even shy of saying so, especially if singled out.

It's that deep down, they knew, but they admired his skill, they wanted to live the same dream, to take the easy way and just not have to care. That's also why for years even after all the sex scandals he managed to keep a mostly Catholic country under control. That's why he still now has a huge following...

Media and Opposition

These two aspects were also important and I'm sure there is a lot more to be said, but I think they don't contribute to explain the Berlusconi phenomenon quite as understanding how much deeply he connects with some Italian sentiments.

Berlusconi owns most of Italy's media and he is renown to be a great communicator. He lies, but his lies are so constant and so fiercely defended by so many, that they slowly become truths. Words slowly lose their meaning and the public becomes divided into factions who do not reason but just mindlessly cheer for one or the other party.

Again, that's partially a "quality" of Italians, being more emotional than rational, being hot headed and profoundly divided. But he managed to exploit that incredibly well.

His power does not extend over only media of course, most if not everyone in his party is strongly tied to him, he choose men with little political past and respectability of their own, people who depended on him to be elected. Berlusconi IS his party, and everyone sings the song he sings. And he made pretty clear that was the way from the start, his party always had direct references to him in the logo, in the hymns, everywhere. Everyone laughs at his jokes. Everyone follows the same rules, tells the sames words, uses the same dialectic tricks. I'm not sure if it's imitation or doctrine but it's powerful.

On the other hand the opposition is fragmented and largely seen as made of intellectuals and professional politicians who do not have any connection with the people. (which to a degree can be even very true). They were always bad communicators so it was easy for Berlusconi to play them, routinely saying that there was no better alternative than him, that the left-wing was made of communists that had no real plan other than raising the taxes (even if financial pressure increases or decreases have not really been strongly linked to any particular government). Furthermore they showed no cohesion, being unable to claim even the huge victories they sometimes achieved (Italy was able to enter the Euro as one of the founding parters due to the work of a left-wing government for example) and not being able to look past their divisions.

Open questions - my two rules

2011-10-28T14:28:00.002-07:00

As I wrote here, there are some fundamental questions in realtime rendering that I wish I knew more about. I do have though two rules I apply when thinking about rendering techniques

Reduce variance: It's better to be consistent at a lower quality than have glitches/flickering artifacts at a higher quality.

Postulate: all graphical effects should be reviewed in motion, crossing quality boundaries

Less is more: It's better to not have a given effect than have it at a too low quality level.

As an example we can analyze shadows. The first rule tells us that it's better to have stable cascades at a lower resolution than having perspective shadowmaps at high resolution.

The second rule tells us that it's better to have lower filtering and cull shadows from some objects or limit the shadowing maximum distance, than having bad shadows everywhere.

Unrelated, I just saw this as a job opening at Valve... Smart guys!

Psychologist

We believe that the more we know about human behavior, about how and why people do what they do, the better our products will be. All game designers are, in a sense, experimental psychologists. That’s why we’re looking for a experimental psychologist to apply knowledge and methodologies from psychology to game design and all aspects of Valve’s operations. We want to exploit your experience with experimental design, research methods, statistics, and human behavior to help craft even more compelling gameplay experiences for future Valve titles. We’d also expect you to research and weigh in on any and all topics that are relevant to improving the experiences of our customers, partners, and employees.

Duties:

Provide relevant insight into human behavior in order to shape gameplay and customer experience.
Perform statistical analyses on all aspects of Valve’s operations: gameplay, financial, and company data.
Research compelling new hardware technologies.
Design experiments to evaluate various gameplay hypotheses and design choices.
Improve existing playtesting methodologies while incorporating novel techniques to improve best practices.
Develop innovative ways of acquiring relevant data to answer open questions about all aspects of Valve’s products and business practices.

Requirements:

Graduate degree in Psychology (or equivalent) field
Advanced knowledge of statistics
Familiarity with one or more of the following pieces of data analysis software: SPSS, Systat, Matlab, R, (or equivalent)
Four years experience with:
- Experimental design/research methods
- Relevant research in cognitive, social, human factors, and related disciplines in psychology

Recommended:

Proficiency in one or more of the following programming languages: C++, SQL, PHP, (or equivalent)

Open questions

2011-10-26T21:17:00.017-07:00

From the series "End of the virtual world" by Robert Overweg

Battlefield, Mass Effect, Modern Warfare, Red Dead, Crysis, Rage, Forza, GT, Alan Wake...

What makes the graphics work? I've played the last three call of duty games on 360, with a projector and a 5.1 system. I found them to be amazing. Then I downloaded MW2 on steam, and the graphics looked mediocre.

Conversely, Mass Effect 2 seemed decent on 360 but way more awesome on PC.

Red Dead pushes less polygons and has more visual defects than Crysis but awed me in a way no other game of this generation did.

Battlefield 3 is a technical jewel but MW seemed to me to have a better atmosphere.

Why? Is my subjective judgment shared by others? Is my memory failing me and have my expectations shifted? Or there is also something technical behind these impressions?

MW textures on PC have the low-res quality of a bad port. Is this, coupled with the increase in resolution, the reason the game looks worse to me on PC, makes my brain "see" the polygons more? Or is it because its atmosphere is better suited for a projector and a couch? Or maybe it's because playing the game I was more immersed into it than when I just looked at the graphics on the PC.

Why was it the opposite for ME2? Is its art style, less reliant on gritty texture detail, making better use of the high resolution and antialiasing of the PC?

Why I enjoyed more ME2 single player graphics than BF3 ones? Are the graphics enhancing the gameplay, or is also the opposite true, game and story do affect the perception of the graphics?

Is it the aliasing that is killing BF? Or is the inability of deferred lighting to express the subtleness of light transport and materials the issue? Is our industry jumping on the deferred lighting approach too fast, without really understanding what it's losing form precomputed lighting?

What about the heavy bloom and flares that BF3 and Crysis2 use? How are they working? How do they alter the perception of the image?

From my experience I observed some patterns, but I don't really know much, I've also found very little research...

Aliasing and other high frequency artifacts quickly tell your brain that it's looking at CG, they are very disturbing. Motionblur at 30fps looks more cinematic and packs more punch than 60fps without blur. We tolerate framerate problems way more if the game looks busy (i.e. A huge explosion) than if they are not connected with game actions. Colour is hard to get right, and ambient lighting and proper occlusion of lighting terms are important to represent volume, rendering the air (haze, fog, scattering, desaturation tricks etc) helps with scale. Crowd variety is achieved more with colour and behavior variation than texture and model. Specular lobes are everywhere, have always fresnel, and we can't recognize errors in the light directions in the specular but we use high-gloss highlights to evaluate shape. Bleeding dark edges (i.e. when dealing with subsampled effects) looks less questionable than having bright halos.

There, I think I didn't miss anything, that's pretty much all I know, it fills a few lines of text. I think that's a big challenge for us, there are more studios that know how to do deferred lighting right and fast than there are studios that know what's important to create an immersive, beautiful game. We don't know what to focus on where, what devices are used for what, which artifacts are tolerable and which are disturbing.

We know the (sometimes) the physics but we have very little math to model the psychology. Yes, KSK fits skin specular well. But what parts of it are important? When does it break and breaks the perception of skin being skin? We need to make hacks, and physically reasonable hacks are fundamental in our line of business, physics are fundamental, but physiologically motivated hacks would be way better!
This also affects all the "tuning" decisions, i.e. more LOD switching but with better detail near the camera, or vice-versa? Geometry or textures? Bigger SSAO radius and more noise or the opposite and so on.
Moreover linking perception (vision) and psychology with rendering would give us more objective tools for art-direction, like what device is the best to convey in the general audience a given intent, what makes a sense of "scale" or of "fear" and so on...

Rendering without knowing about perception is crazy, it's like if musicians knew more about sound waves and instruments than harmony and melody. And yet we often chose rendering techniques based on really faint leads about what is needed to look good. There is little research, and the little there is focuses on issues that seldom are directly applicable to modern videogame rendering techniques.
Even worse, we are just not starting to understand the basics, like color and normals, and not only in the industry but even in most publications you will find little regard towards even basic visual perception metrics.

Videogame rendering today at its best it's a work of iteration, the more you can try and the better feedback you get, the more you inch towards this ill-defined target of visual splendor. But even artists and art-directors with a great eye for light and colour seldom have much experience about realtime CG artifacts and their impact on perception.

Shifting from art to science has to happen in order for our profession to evolve, we can't rely on art direction for technical problems, it's not only too error-prone but also very inefficient. Scientific studies can be shared and described exactly, while art direction remains subjective and does not result in a shared progress.

It of course not something that happens only in rendering (or presentation in general, animation, audio), we even make games with very faint leads about what is needed to make fun. And that's why often you have studios investing big amounts of money, and staffed with great talents, producing results that are impressive but still fail, while only very few games really know how to be fun, and really know how to immerse players in their art...

More to come... Meanwhile you might want to read something (other than the links I've already scattered in the post): Some interesting reads here. Also Holly Rushmeier's work (some is linked in the post - from the website the EG2001 and EG2003 presentations are very neat)
Please comment if you know more resources on the topic. A question has been posted on Quora here...

Rendering 102

2011-10-18T17:52:00.005-07:00

Follow-up to http://c0de517e.blogspot.com/2011/09/rendering-101.html: a simplified and hopefully clearer version of the "how the gpu works" posts. Covers GPU computational model and GPU graphics pipeline.

Check it out here:
http://www.scribd.com/doc/69372569/videogame-Rendering-102

As usual, let me know in the comments about mistakes and possible improvements!

Spent

2011-10-04T14:47:00.001-07:00

http://www.playspent.org/

Don't lie.

2011-10-03T12:12:00.003-07:00

A task is not done when its code gets checked-in. In production, "done" is:

Checked-In
Stable (passes automated tests)
In-budget (memory and performance)
Usable (tools, parameters)
Verified (art-directior or designer or lead programmer)

If you ignore the last four, you're living in a dream. A dream in which you will ship the game. Instead, you'll die under the pressure of crunch time and ship a flawed product that does not match the quality standards it should.

That also means that in production we start with a stable, in-budget product. And that we do have means of verifying that this is true for the entire production (tests).

Yes, it will take twice as much time to finish a task. Yes, it will mean that some tasks can't be declared done until you make more space somewhere else, to fit them in.

Of course you can avoid all this. All it takes, is to lie.

We are a creative industry. We have to deal with change, we don't plan things up ahead and then waterfall until they are all done (not _most_ of the times at least, there are situations in which that applies).

We don't craft a product by following a plan, we make drawings and sketches (prototypes) and then take a canvas and start painting.
You don't have a person detailing a finger, another one refining an eye, another working on the nose and then hope that everything will stick together just right. Or hope that you will have all the parts done by a given date! And what if it then misses one of the ears? What do you do, take ten artists working on that ear till midnight every day near the deadline?

Only an idiot would do that, and still many game companies work like that, they don't consider that everything has to fit just right, and that change is not local, every change has to fit with the entire context, every brush stroke has to make a sense in the entire painting.
You start with a painting, a rough one, and then refine it, and at all time the painting is a painting, it's not a collection of unrelated pieces. You can stop at any moment and it will be still a painting, maybe not that refined, maybe not as intricate and detailed as you wanted, but it's a painting.

A game is such only if it runs. If it crashes or goes out of memory on the target platform, it's not a game, it's some binary that crashes. If it does not hold its performance, it's not a game, we can't burn it to a disc and call it a game, it won't be shipped, it won't pass certification. Iteration should not break this quality, it should go from a "shippable" game to a "shippable" game. Especially during production.

Rendering 101

2011-09-30T16:02:00.000-07:00

This is how I explain (videogame) rendering to non-rendering (engineers... even if hopefully it's still understandable for some non-programmers as well).

If you have suggestions on how to make it more clear, post a comment!

Fight Night Champion Rendering @ GDC

2011-09-28T14:30:00.007-07:00

I wrote so much for Fight Night Champion over the years, but not much can be found on the internet. I crafted at least five presentations for it, three mostly internal for the team on Fight Night 4 analysis and pre-production, one on the various prototypes and ideas done to exit pre-production, to share ideas with the other EA teams, and one for GDC after the game was done, which was refined and presented by my (awesome!) colleague Vicky Ferguson when I left EA (you should see the recording of her presentation, and she also presented for FN4 at the previous GDC which was very cool too).

Unfortunately the slides of this last one were not published (a couple could be found here, but they are pretty useless)... until now.
At the beginning I thought about writing more detalied blog articles about many of the experiments that were done, but many of them do not really need that much space, some others were rediscovered and published by others (i.e. pre-blurred SSS, bent normals SSAO), and I think FNC was more about the process than the final tech (in fact, the lengthy presentation never goes into the detail of what at the end were the final FNC shaders!), so...

This is a "director's cut" of the GDC slides, it's way more verbose (not meant for presentation) and with much more material. Enjoy.

Bad ideas don't require much explanation: Caching Stable Cascaded Shadowmaps

2011-09-27T20:22:00.003-07:00

Note:

Mathematica and Spherical Harmonics

2011-09-20T15:16:00.001-07:00

As my previous post about Mathematica seemed to be well-received, I've decided to dig some old code, add some comments and post it here. Unfortunately it's littered with \[symbol] tags as in Mathematica I used some symbols for variables and shortcuts (which you can enter either in that form or as esc-symbol-esc). You can also see a PDF version of the notebook here, with proper formatting. Enjoy!

A function and its SH approximation

(* Normalization part of spherical harmonics *)
shNormalizationCoeffs[l_, m_] := Sqrt[((2*l + 1)*(l - m)!)/(4*Pi*(l + m)!)]

(* Evaluates to a function of \[Theta],\[Phi] for a given degree l and order m, it's defined as three different cases for m=0, m<0, and m>0*)
shGetFn[l_, m_] := Simplify[Piecewise[{{shNormalizationCoeffs[l, 0]*LegendreP[l, 0, Cos[\[Theta]]], m == 0}, {Sqrt[2]*shNormalizationCoeffs[l, m]*Cos[m*\[Phi]]*LegendreP[l, m, Cos[\[Theta]]], m > 0}, {Sqrt[2]*shNormalizationCoeffs[l, -m]*Sin[-m*\[Phi]]*LegendreP[l, -m, Cos[\[Theta]]], m < 0}}]]

(* Indices for a SH of a given degree, applies a function which creates a range from -x to x to every element of the range 0...l, return a list of lists. Note that body& is one of Mathematica's way to express pure function, with parameters #1,#2... fn/@list is the shorthand for Map[fn,list] *)
shIndices[l_] := (Range[-#1, #1] &) /@ Range[0, l]

(* For each element of the shIndices list, it replaces the corresponding shGetFn *)
(* This is tricky. MapIndexed takes a function of two parameters: element of the list and index in the list. Our function is itself a function applied to a list, as our elements are lists (shIndices is a list of lists) *)
shFunctions[l_] := MapIndexed[{list, currLevel} \[Function] ((m \[Function] shGetFn[currLevel - 1, m]) /@ list), shIndices[l]]

(* Generates SH coefficients of a given function fn of \[Theta],\[Phi], it needs a list of SH bases obtained from shFunctions,it will perform spherical integration between fn and each of the SH functions *)
shGenCoeffs[shfns_, fn_] := Map[Integrate[#1*fn[\[Theta], \[Phi]]*Sin[\[Theta]], {\[Theta], 0, Pi}, {\[Phi], 0, 2*Pi}] &, shfns]

(* From SH coefficients and shFunctions it will generate a function of \[Theta],\[Phi] which is the SH representation of the given coefficients. Note the use of assumptions over \[Theta] and \[Phi] passed as options to Simplify to be able to reduce the function correctly, @@ is the shorthand of Apply[fn,params] *)
angleVarsDomain = {Element[\[Theta], Reals], Element[\[Phi], Reals], \[Theta] >= 0, \[Phi] >= 0, \[Theta] <=
Pi, \[Phi] >= 2*Pi};
shReconstruct[shfns_, shcoeffs_] := Simplify[Plus @@ (Flatten[shcoeffs]*Flatten[shfns]), Assumptions -> angleVarsDomain]

(* Let's test what we have so far *)
testNumLevels = 2;
shfns = shFunctions[testNumLevels]
testFn[\[Theta]_, \[Phi]_] := Cos[\[Theta]]^10*UnitStep[Cos[\[Theta]]] (* Simple, symmetric around the z-axis *)

(* generate coefficients and reconstructed SH function *)
testFnCoeffs = shGenCoeffs[shfns, testFn]
testFnSH = {\[Theta], \[Phi]} \[Function]Evaluate[shReconstruct[shfns, testFnCoeffs]]

(* plot original and reconstruction *)
SphericalPlot3D[{testFn[\[Theta], \[Phi]], testFnSH[\[Theta], \[Phi]]}, {\[Theta], 0,
Pi}, {\[Phi], 0, 2*Pi}, Mesh -> False, PlotRange -> Full]

(* Checks if a given set of coefficients corresponds to zonal harmonics *)
shIsZonal[shcoeffs_, l_] := Plus @@ (Flaten[shIndices[l]]*Flatten[shcoeffs]) == 0

(* Some utility functions *)
shSymConvolveNormCoeffs[l_] := MapIndexed[{list, currLevel} \[Function] Table[Sqrt[4*Pi/(2*currLevel + 1)], {Length[list]}], shIndices[l]]
shExtractSymCoeffs[shcoeffs_] := Table[#1[[Ceiling[Length[#1]/2]]], {Length[#1]}] & /@ shcoeffs

(* Convolution with a kernel expressed via zonal harmonics, symmetric around the z-axis *)
shSymConvolution[shcoeffs_, shsymkerncoeffs_, l_] := (Check[shIsZonal[shsymkerncoeffs], err];
shSymConvolveNormCoeffs[l]*shcoeffs*shExtractSymCoeffs[shsymkerncoeffs])

(* Another test *)
testFn2[\[Theta]_, \[Phi]_] := UnitStep[Cos[\[Theta]]*Sin[\[Phi]]] (* asymmetric *)
testFn2Coeffs = shGenCoeffs[shfns, testFn2]
testFn2SH = {\[Theta], \[Phi]} \[Function]Evaluate[shReconstruct[shfns, testFn2Coeffs]]
plotFn2 = SphericalPlot3D[testFn2[\[Theta], \[Phi]], {\[Theta], 0, Pi}, {\[Phi], 0, 2*Pi}, Mesh -> False, PlotRange -> Full]
plotFn2SH = SphericalPlot3D[testFn2SH[\[Theta], \[Phi]], {\[Theta], 0, Pi}, {\[Phi], 0, 2*Pi}, Mesh -> False, PlotRange -> Full]
Show[plotFn2, plotFn2SH]

(* Test convolution *)
shIsZonal[testFnCoeffs, testNumLevels]
testConvolvedCoeffs = shSymConvolution[testFn2Coeffs, testFnCoeffs, testNumLevels]
testFnConvolvedSH = {\[Theta], \[Phi]} \[Function] Evaluate[shReconstruct[shfns, testConvolvedCoeffs]]
plotConvolvedSH = SphericalPlot3D[testFnConvolvedSH[\[Theta], \[Phi]], {\[Theta], 0, Pi}, {\[Phi], 0, 2*Pi}, Mesh -> False, PlotRange -> Full]

Raytracing Myths

2011-09-19T16:58:00.008-07:00

I've already blogged about raytracing and rasterization, a couple of times, and I guess I should stop. Still, duty calls when I read statements such as "rasterization cannot be parallelized nearly as effectively as ray tracing".

Let's go!

Algorithms

Let's sketch both algorithms in some sort of pseudo-accurate pseudo-code:

Raytracing (first-hit)

for_each pixel px on screen { for_each primitive pr in scene { intersect ray(px) with pr }}

Rasterization

for_each primitive pr in scene { for_each pixel px covered by pr { mark px on pr }}

Pretty similar right? The inner loops are different, one intersects a ray equation with the analytic description of a primitive, the other in some way walks on the primitive marking pixels covered by the primitive (i.e. for triangles, by scanline interpolation or by walking on the pixels of the bounding box checking the edge equations).

Visible Pixels: there is a fundamental difference here that we will disregard for the future, as it's written raytracing has to walk on all the pixels of the screen, while rasterization walks all the pixels covered by primitives, thus is only part of the screen is covered raytracing has a disadvantage. The same could be said of rasterization if primitives have large parts that off-screen. This is true but not interesting, rasterization could solve the problem by clipping (moving the complexity from pixels to objects at least) and raytracing could subdivide the screen and identify empty tiles, anyhow this does not change the problem in general and does not change the complexity.

Also, some of you might have noticed that the description I've made is overly simplistic. What I've sketched does not deal in any way with depth hiding. Both algorithms need to keep track of which primitive is the closest for every pixel. Fundamentally it's the same operation, but it has a different memory cost due to the ordering of the operations: raytracing can keep a single depth while scanning every primitive of a pixel and understand which primitive is the closest to the camera, rasterization needs to keep a value for each pixel as we will know which is the closest only when the outer loop is done (z-buffer!).

Raytracing (first-hit)

for_each pixel px on screen { for_each primitive pr in scene { intersect ray(px) with pr, keep nearest depth }}

Rasterization

zbuffer = list of depths

for_each primitive pr in scene { for_each pixel px covered by pr { mark px on pr, store nearest in zbuffer }}

Complexity

So memory-wise rasterization is worse than raytracing! We have to keep a z-buffer! Not so fast... Let's assume that we have a huge scene to render. Maybe we want even to stream stuff. What could we do with both algorithms? It looks like we would need to have around, in memory, the contents of the inner loop, while we could stream over what's in the outer loop. It's another interesting difference that originates from the same fundamental fact, the two loops are inverted. The z-buffer can be nifty because we can stream primitives through it, no matter how huge the scene is, we can go primitive by primitive and keeping only the z-buffer around we can render a scene. Raytracing on the other hand, has to keep primitives around.

Now we can of course imagine variants for both to be more efficient, z-buffer can be subdivided in tiles (or, for other reasons that are thought still about memory efficiency, we can use a hierarchy), primitives in raytracing can similarly partitioned and aggregated using bounding volumes and so on. It does not really change that we don't have a winner, we have the same computation expressed in a different order, which leds to different trade-offs.

What about the compute time? It's evident that from what I've sketched, it's identical. But raytracing is O(log(primitives)) right? At least using spatial subdivision structures and disregarding the complexity of building them (which is often more than linear, thus making from the beginning complexity arguments rather moot)...It might not be obvious how that is not true, so let's make a couple of examples, hopefully these will be enough to generalize to any subdivision structure.

A scene with two group of primitives, one on screen, one off screen. These are bound by a box each, and the two boxes are bound with another box, thus creating a box hierarchy.

What would the raytracer do? It would for every pixel on screen check the outer box, for each pixel intersecting it it would check then inner boxes, one will fail always and not trigger the check with the inner primitives, the other will not and thus we will end up checking only one of the groups.

Rasterization? We go through all the pixels of the outer box, we see that something is on screen so we go through all the pixels of the inner boxes, one will be on screen, the other won't and we will render only one group of inner primitives.

Other than the tradeoff between "for every pixel on screen" and "for every pixel covered by primitive" - the "visible pixels" problem that I already said that we would disregard, it's the same thing.

But that is about screen coverage in two of the dimensions. What about depth complexity?

A sparial subdivision structure with a raytracer provides a way of having guarantees on the depth ordering. For each ray, if we find an intersection in a given depth of the subdivision, we don't have to proceed any further, we know that all the primitives in lower depths are hidden for that ray. Or to put it in another way, for all the nodes that are fully covered by previous node's primitives, we don't do any work.

Doesn't that happen also for rasterization? Let's persuade ourselves of this fact. Assume that we have a depth-only hierarchy, our scene is split into different "chunks" along the depth, each chunk is defined by a splitting plane and contains a primitive. Now, let's say that the primitive of the first chunk covers the entire screen. Obviously in a raytracer we would trace that and stop, considering once all the pixels of the first plane, then once over all the pixels of the primitive, and then ending. In a rasterizer, we would rasterize the plane, then the primitive, then the following plane, find it all occluded and end. A bit more work, but constant, no matter how deep the chunk with the all-covering primitive is we stop always analyzing one more plane, so it's not a concern complexity wise.

What if the primitive does not cover the entire screen though? What if it leaves exactly one pixel not covered? The raytracer would then go and check the following chunks against that single pixel only until a primitive covers it. The rasterizer would rasterize all the pixels in the planes every time it need to check, until the primitive covers it. It seems that we have a fundamental difference here, we could invoke again the "visible pixels" principle, but we would like a way to make the occlusion test resolution independent for the rasterizer to have the same cost.

To be clear, I'm not advocating the use of BVHs and z-buffer pyramids in your game engine today. I'm just saying you could, so the big algorithmic complexity argument about the doom day when rasterization will not be viable anymore are wrong. By the way, it won't even be a very impractical method good only on the paper (like the constant-time raytracing stufff), it would probably be a good idea in some heavily occluded scenes...

Luckily depth complexity in practice is not even a huge deal for rasterizers, for example on a GPU. How comes? Overdraw should kill us, how do we avoid it? We can because a GPU does not need to rasterize fully covered primitives, if we have a front-to back ordering we can save most of the cost by checking our primitives against a coarse z-buffer, before shading. In a GPU shading is the big cost, and so we just do an early depth test and live with that, but in theory this works in general, we can make the occlusion cost logarithmic on the pixels by subdividing the z-buffer grid in a hierarchy and in fact if we're interested in occlusion culling and not on shading, this is a good idea (see HOMs).

In the real world, if we want to talk outside the theory and useless statements about big-O complexity, both techniques need to deal with primitives and occlusions and they are if not equally good, very close, with both being able to handle scenes of immense complexity and out-of-core rendering. Both can be linear, logN or whatever, it's only that some techniques make more sense in some situations, so in practice most raytracers are logN with N^2 or more subdivision structure build times, rasterizers are mostly linear, both live happily with that on equally complex scenes.

Ironically in some way, raytracing which is said to be less dependent on primitive counts, absolutely needs accelleration structures for all but the simplest scenes, no raytracer goes without one, that's because you need them not only for depth complexity, but also to deal with the screen-space one (I guess you could cull primitives, LOD and tile, but then it would be so very close to a rasterizer not to be really smart, so really no one does that).
Rasterization on the other hand performs great on a wider range of scenes and needs z-buffer hierarchies, level of detail and other complex occlusion strategies only when the shading costs are negligible and the scenes have a lot of depth complexity. Both can do amazing things.

Parallelism and Concurrency

Enter the second big topic, parallelism and concurrency. Raytracing is easier to parallize says the myth - because each ray is independent. Is that true?

I can honestly see where this myth comes from, it even has a bit of truth in it. I picture someone in a university, writing a raytracer for fun or for a course and finding that's easy to fork at the ray-generation lever or a parallel_for over some worker threads and it works, because the inner loop is indeed independent, does not rely on shared state. A rasterizer does, for the z-buffer, so it would require some form of syncronization. Not the end of the world, you could have a z-buffer per thread and merge, you could have some a lockless stuff per pixel, tiles or whatever, but still it requires some changes.

So is it easier? In that sense yes. But let's reason as a more experienced engineer (or scientist) and not as a freshmen. We see two nested loops, we are on a modern CPU architecture, what do we do? Instinctively (at least, I hope) we unroll and parallelize the inner (instruction parallel), and try to make concurrent the outer (data parallel). In this respect, how do the two algorithms fare?

These two, powerful forms of parallelism are absolutely fundamental to achieve great throughput on modern architectures as it provides them with a lot of coherent computation. Modern CPU execution times are dominated by latencies. Instruction latencies and dependencies and even more nowadays, memory latencies. And this won't change! Latency is hidden by unrolling, by having enough "predictable" computation (and memory access). If we have an operation A1 and a subsequent B1, with B1 dependent on A1 and A1 taking 10 cycles of latency, if we can find nine other instructions between A1 and B1 we're safe, we "hid" the latency. If we can unroll a cycle ten times over data elements 1...10, then we can trivially do A1,A2,A3,... then B1,B2... and we're done.

Now in theory, they are both good. A raytracer inner loop intersect a ray with a number of primitive, a rasterizer walks on a primitive looking for pixels. If the primitives are uniform, both are easy to unroll and play well with SIMD, we can intersect a ray with i.e. four primitives at a time, or in a similar way rasterize four pixels of a scanline (or a 2x2 block...) together.

The outer loop follows the same reasoning we did before, and indeed it's a bit easier on a raytracer. So again a win? Yes if we limit ourselves to a case that does not (as I wrote before) exist: a raytracer without a spatial acceleration structure and shading. And if we don't consider that accessing primitives in the inner loop is way more memory intensive than walking on pixels.

If we write things in "theory", or rather, with a theory based on an over-simplistic model raytracing wins... If we take a realistic renderer on the other hand, things change.

Spatial acceleration breaks uniformity. Shading of different primitives at the same time leads to incoherent memory access. Same for neighboring rays intersecting different groups of primitives. We can have a ray at level zero of the hierarchy and the neighboring one at level ten. A ray might need shading, while the next one might be still traversing. It's a mess!

And not an easy one to fight against at all, how do we deal with that? Well it turns out, it's not that simple, and not fully solved yet.

One of the first approaches was to collect rays into the spatial structure cells and then be parallel to all the rays in a given cell. It works if you can generate enough rays, if you have still some coherency, i.e. for a distributed raytracer, less well for a path tracer. And still it deals only with part of the problem.

For a while the solution seemed to be to organize rays in "packets" that could walk the structures together, hoping they won't diverge much or trying to reconstruct some coherence by splitting and merging them. Then we found out that this works again for the kind of coherent loads that (we will see) are not really _that_ interesting for a raytracer, it will give you decently fast first-hit, shadows, the sort of stuff a raytracer is still good at. You can get some perfect mirrors on a sphere or so, how cool is that.

More recently, the first approach evolved in more general ray reordering strategies, while the second was turned sideways and experimented with having parallelism on a single ray by growing the width of the acceleration structures.

All this while on GPUs rasterizers have been executing stuff in parallel, where it matters, over tens and hundreds of computational units, for years now (and Reyes have been shading in parallel since the '82).

Ease of use

Simple algorithms are easy to express in both, and writing a bare-bones triangle rasterizer or sphere raytracer (using the primitives that are easier to map in both) is trivial. Maybe the raytracer will take a few lines less, but hardly anything that we care about.

For more complex effects, raytracer starts to win. Adding shadows, reflections. An univeristy student can have a lot of fun with a few lines of code, feeling happy. And from here on, if we never consider resource usage, raytracing always wins. You can solve the entire rendering equation (or so) with a path tracer in a hundred lines of code.

But when we make programs, in practice and not on paper, no matter what we have to deal with resources, they enter the equation. Then it becomes a matter of performance and how much do we care about it.

When I started programming, on a commodore 64, I knew only Basic and Assembly, and that was all I knew even when I moved to the Amiga (amos) and the PC (QuickBasic, PowerBasic), until one of my older cousing started university and I grabbed his K&R book. I loved to write graphical effects, and I had to "optimize" code by either tweaking the code to suit better the (to me unknown) logic of the underlying interpreter, or add inline assembly directly via machine code. Hardly easy, even if you might argue that for some things Basic was easier to learn and use than C (and I really liked the QuickBasic PDS IDE at the time...)

I think the complexity argument is rather dumb. Raytracing is less coherent and struggles with performance. Rasterization is more coherent and struggles with algorithms that need more flexibility in the visibility queries. Raytracing needs spatial accelleration structures and complex strategies to gain ray coherency. Rasterization needs fancy algorithms and approximations when it has to deal with global effects and more general visibility queries.

Conclusion

In practice if you've shipped a game, you already knew this. What do GPU use to render all these incredibly complex scenes over thousands of processors? What do we do on the other hand if we have to execute random visibility queries, for example for A.I.? Sometimes there are reasons why things are the way they are. The biggest trade-off between the two is surely that raytracing starts with incoherent visibility queries and research is trying to "find" patterns in them in order to gain performance, rasterization starts with coherent primitive walking and research is trying to find algorithms to compute global, complex illumination out of these fast massive queries we can issue (this is a nice example).

I liken it to programming languages. Dynamic versus Static, Pure versus Imperative. They are all Turing-complete and in theory equally capable at expressing computation. In practice though they start from different endpoints and try to reach the other, pure languages start safe and strive to express side-effects and program order, imperative start with side effects and strive to achieve safety, and so on.
And if there is something to learn from that (and hopefully there is not :D) is that performance in computer science has been always a stronger drive in the success of a technology than other considerations (we program, unfortunately, in C++ not in Haskell), probably because it's more important to achieve a given amount of quality in practice, than having something that in theory produces something with a higher quality but in practice does not deliver.

Going back to the article that "spawned" all this. Is raytracing the future? Sure it is, it's surely "in" the future of rendering. But not because it's easier, or because it has a lower algorithmic complexity, or because it parallelizes better. None of these big global adjectives apply, that's all bullshit. It's in the future because it's a technique which has a set of trade-offs that are orthogonal to rasterization. It's useful, having more options, techniques, points of view is always useful. There will be areas in which rasterization will always make much more sense, and areas in which raytracing will, while most else will be happy to be able to pick and choose depending on the kind of problem they're facing.

I think you can always peek into the future by looking around, looking what other people in other fields with similar problems are doing... In these years, for CG, raytracing and rasterization are influencing each other more and more.

Offline renderers are mixing techniques from both realms, just see what Pixar's Photorealistic Renderman is capable of doing. It's a Reyes rasterizer, for the first hit. It's parallel, concurrent and distributed. It can do shadowmaps and raytracing. It can cache point clouds and do some sort of screen-space ambient occlusion and so on. Still we lived with rasterization for a long time even in the offline realm especially in some fields that are less concerned about absolute lighting precision.

Only recently we started to see GI and raytracing as useful and productive replacements for some algorithms in the offline realm. I expect everything to be more and more hybrid. I don't think rasterization will ever die. Also, from what I know and I see of the research around raytracing I don't think that we'll see it in realtime, in games for quite some time yet.

Mathematica and Skin Rendering

2011-09-08T18:54:00.007-07:00

Introduction

Now, if you've been following my blog for a while, you might have noticed that I hate C++ almost as much as I love rapid prototyping, live-editing languages. Wolfram's Mathematica is one of the finest of such tools I've found so far.

It might sound surprising as Mathematica's mostly known as a computer algebra system, performing symbolic manipulations that are arguably not the bread and butter of the realtime rendering engineer, but deep down everything boils down to a Lisp-like language with a twist, as it allows different representations of the parse tree (i.e. plain code, mathematical notation, editable widgets and even inline graphics). All of this is coupled with one of the most powerful mathematical and numerical libraries available, with the symbolical math functionality "helping" the numerical one (automatic algorithm selection, generation of derivatives and so on), and topped with a very good visualization system and amazing documentation.

Sounds cool right? Let's start using it then!

Preintegrated Skin Shading

I've been following Eric Penner's Preintegrated Skin Shading for more than a year now. We were colleagues at EA Canada, but we never ended up on the same project and being both interested in graphics research, we ended up having independently discovering a few of the same ideas while working at our project.

I was pretty amazed when he showed me his "Shader Amortization using Pixel Quad Message Passing" (Now in GPUPro 2) as I had a similar idea in the back of my mind as well, and I guess he was pretty surprised as well when I presented internally to EA a prototype for Fight Night that sported "preblurred" subsurface scattering (at that time FN still did diffuse lighting with cubemaps, so the prototype was about cubemap biasing, normal preblurring and different normal fetches for each of the RGB channels as in "Rapid Acquisition of Specular and Diffuse Normal Maps from "). He combined that same intuition with gradient estimation (as in "Curvature-Dependent Local Illumination Approximation for Translucent Materials", "Curvature-Dependent Reflectance Function for Interactive Rendering of Subsurface Scattering", by Kubo et al and "Curvature-Based Shading of Translucent Materails, such as Human Skin” by Kolchin) and precise integration of Jensen's SSS model to create something fairly amazing, that he recently presented at Siggraph 2011.

While Eric was preparing his talk I was trying to help by finding a good analytic approximation for his integral. Unfortunately this work came too late and couldn't be included in the Siggraph talk (Eric lists it as "future development" possibility) so I'm writing this now to complement his work and as a primer to show Mathematica's usefulness in our line of business.

Mathematica's crash course

When you start up Mathematica's interface you'll get prompted to create a new notebook. A notebook is a handy document where you can mix Mathematica's expressions with normal text and graphics, a sort of "active" wordprocessor.

Everything you enter will go in what Mathematica calls a "cell", by default cells are of "input" type, that means ready to accept expressions. Other kind of cells are not "active" and won't be evaluated when asked to process the entire notebook.

When you want to evaluate the contents of a cell you can press shift-enter in it. That will cause its text to go through the Mathematica kernel, and any output generated by it will be shown underneath, in some output cells grouped together with the input one. Expressions terminated with a semicolon won't generate output (it will be suppressed) which is useful to avoid to spam the notebook with intermediate results which don't require checking.

A nifty thing is that expressions can be entered in various formats, Mathematica supports different notations for the same syntax trees. The cool thing is that you can even ask the interface to convert a given cell in one of the formats, so you can get all your expression in a "programmer's" notation and then have them converted into mathematical formulas ("TraditionalForm") without needing to use complex shortcuts to write everything in the mathematical notation.

Ok let's get started now. We'll need to use the skin diffusion profile of d'Eon and Luebke that is expressed as a sum of gaussians with some given coefficients. We start by defining a couple of lists with these coefficients. List literals in Mathematica are entered using curly braces, so we can write and evaluate the following:

ScatterCoeffs = {0.0064, 0.0484, 0.187, 0.567, 1.99, 7.41}*Sqrt[2]

ScatterColors = {{0.233, 0.455, 0.649}, {0.1, 0.336, 0.344}, {0.118, 0.198, 0}, {0.113, 0.007, 0.007}, {0.358, 0.004, 0}, {0.078, 0, 0}}

Note how the output of the first assignment should already have performed the multiply by 1.414. Using the equal operator in Mathematica forces the evaluation of the expression on the right before assigning it to the variable.

If you squinted you should have noticed already the Sqrt[2] in the previous expression. Function arguments in Mathematica are passed in square brackets, and all Mathematica's built-in functions start with an upper case.

As an exercise, let's now define a Gaussian function manually, instead of relying on Mathematica's internal NormalDistribution:

GaussianFn[var_, x_] := 1/Sqrt[2*Pi*var]*Exp[-(x*x)/(2*var)]

Now, as you evaluate this, you should notice a few things. First of all, it generates no output, the := operator is a delayed assignment, it will evaluate the expression only when needed, and not beforehand (similar to a list quote). Second, that function's input parameters are post-pended with a underscore.

In Mathematica, functions are fundamentally expressions with delayed evaluation, when they are invoked the parameter values are substituted into the expression and then the result is evaluated.

This concept is important, also because being a computer algebra system, Mathematica is all about expressions (see "FullForm" in the documentation), and in fact we could have literally defined the same GaussianFn by performing a delayed replacement (using ReplaceAll, or its shortcut /. note that in Mathematica most basic functional operators infix notations as well) into an expression, like this:

GaussianExpr = 1/Sqrt[2*Pi*var]*Exp[-(x*x)/(2*var)]

GaussianFn[p1_,p2_] := GaussianExpr /. {x->p1, var->p2}

GaussianFn[1,0.5] (* evaluate the function, to prove it works *)

Or as a function object, which won't require a delayed assignment as the function object itself will delay the evaluation of its expression:

GaussianFn[p1_,p2_] = Function[{p1,p2},GaussianExpr/.{x->p1, var->p2}]

GaussianFn[1,0.5]

Got that? Probably not, but all should be clear if you as we go, also try things out and have a look at the excellent online documentation that ships with the program.

Let's define now our last helper expression, and then we will delve into Eric's model and how to fit a simplified function to it.

This is the scattering function, it takes a variable between zero and one and as we don't want to deal with computations in the RGB space which would make visualization of all the functions quite more complex, we pass an index between one and three (lists are one-based in Mathematica) that will decide which channel we're using. We will then need to repeat the fitting for every channel. As I wrote, Mathematica is a lisp-like functional programming environment, so we'll often work with lists and list operations:

ScatterFn[x_, rgb_] := Apply[Plus, Table[GaussianFn[var, x], {var, ScatterCoeffs}]*ScatterColors][[rgb]]

In this case the Table function evaluates GaussianFn multiple times generating a list. Each time it will replace the variable "var" with one of the entries in the "ScatterCoeffs" list (note that the other variable, x, will be replaced as it's one of the arguments of ScatterFn, so no "free variable" is left and we should expect to get a list of numbers out of the evaluation). It then multiplies element-wise the list with the ScatterColors one and it applies to the list the "Plus" operator (note that we could have used the infix version "fn @@ expr" as well), thus summing all the elements. We'll then have a single, three-component result (as ScatterColors is a list of three-elements vectors), and we take the "rgb" indexed one out of it (double square brakets is the list indexing operator).

Integrating Skin Scattering

The key idea in Eric's work is that the subsurface scattering in the skin depends on a neighborhood of the shaded point, if we don't account for lighting discontinuities caused by shadows (which are handled separately). We can then approximate this neighborhood using the surface curvature, and pre-integrate the scattering on a sphere (circle, really) of the same curvature. It then generate a two-dimensional texture based on the curvature and normal dot light with the results of this precomputation. We are going to try to fit a simple expression that we can compute in realtime to avoid having to use the texture.

Let's now express this two-dimensional function in Mathematica, following the expression that you can find in Eric's paper:

(* This is the function defined with integrals, as in the paper *)

DiffuseLight[\[Theta]_,r_,rgb_]:=NIntegrate[Clip[Cos[\[Theta]+x], 0,1}]*ScatterFn[Abs[2*r*Sin[x/2]],rgb],{x,-Pi,Pi}]

DiffuseWeight[r_,rgb_]:=NIntegrate[ScatterFn[Abs[2*r*Sin[x/2]],rgb],{x,-Pi,Pi}]

DiffuseIntegral[\[Theta]_,r_,rgb_]:=DiffuseLight[\[Theta],r,rgb]/DiffuseWeight[r,rgb]

This is pretty simple stuff. We're going to generate samples out of these functions (like Eric's precomputed texture) to then compute a fit, so I directly used the numerical integration (NIntegrate) instead of entering it as a mathematical expression and then asking the numerical approximation of it (N[expr] in Mathematica).
Notice the \[Theta], these were entered as "esc" theta "esc" which is one of the ways of adding symbols in Mathematica's expressions (in this case, the Greek letter theta). These expressions are more readable if pasted in Mathematica and converted into StandardForm.
If you are on a slower computer you can tweak the precision parameters of the NIntegrate or replace it with a NSum, as Eric does in his texture-generator shader:

(* This is a bit faster to evaluate, it's a translation of the code used to generate the lookup table, using an evenly spaced sum instead of NIntegrate *)

DiffuseWeight2[r_,rgb_]:=NSum[ScatterFn[Abs[2*r*Sin[x/2]],rgb],{x,-Pi,Pi,2*Pi/20}]

DiffuseLight2[\[Theta]_,r_,rgb_]:=NSum[Clip[Cos[\[Theta]+x],{0,1}]*ScatterFn[Abs[2*r*Sin[x/2]],rgb],{x,-Pi,Pi,2*Pi/20}]

DiffuseIntegral2[\[Theta]_,r_,rgb_]:=DiffuseLight2[\[Theta],r,rgb]/DiffuseWeight2[r,rgb]

Sampling the function

Let's generate now a table of data to be used for the numerical fitting. That has to be in the form { {input0, input1...., output}, ... }. We will use the Table function again for that, the expression that will be evaluated at each element of the table is the list literal {angle, radius, DiffuseIntegral[...]} (we could have used the List[...] function as well, instead of the literal) but this time we'll need two ranges, one for the radius and one for the angle.
The Flatten call is needed because Table, when used with two ranges, generates a list of lists (a matrix), while we want a flat list of three-dimensional vectors.

channel = 1; (* 1=red 2=green 3=blur *)

dataThetaRVal = Flatten[Table[{\[Theta], r, DiffuseIntegral[\[Theta], r, channel]}, {\[Theta], 0, Pi, 2*Pi/20}, {r, 0.25, 6, 1/20}], 1];

Finding a fitting function

For the fitting function I also want to check how well it extrapolates. So I will compute the fit over the original sample range, but then plot it over an extended interval to verify how the function looks like outside it.

To do this we could use Mathematica's function plotting operators (and it might even be a smarter choice) but instead let's still work with a plots that operate on lists of samples, ListPlot3D.

To plot over an extended range we define an interval table, dataIntervalExt, that's because I want to do these extended plots various times so I "save" the sample inputs and then apply them to different functions.

The first time I use this is to generate a graph of how the lambert lighting model would look like. Using Mathematica's functional programming primitives I define an inline function that maps from x (that will be an element of dataIntervalExt, so a two-dimensional input vector) to a list {first of x, second of x, LambertFn[first of x - that is, the angle]} using the function symbol (in Mathematica it will look like "|->"), then I map this function to every element of dataIntervalExt using the inline /@ map.
Notice how the cosGraph plot is hidden by closing the expression with a semicolon, then the Show[...] function will combine the two plots into a single output.

dataIntervalExt = Flatten[Table[{\[Theta], r}, {\[Theta], 0, Pi, 2*Pi/20}, {r, 0, 12, 1/20}], 1];

dataGraph = ListPlot3D[dataThetaRVal, InterpolationOrder -> 3]

cosGraph = ListPlot3D[(x \[Function] {First[x], Last[x], Clip[Cos[First[x]], {0, 1}]}) /@ dataIntervalExt];

Show[dataGraph, cosGraph]

dataGraph and the combined 2d plots

In order to find a suitable fitting function, I start looking at a graph of some slices slice, fixing a reasonable radius. If the first argument of Plot is a list of functions instead of a single one, it will overlay the plots of all the functions in the list on the same graph. Show does something similar but with any unrelated plots.

plotIn2d = Plot[DiffuseIntegral[x, 3, 1], {x, 0, Pi}]
(* This will generate a graph with three slices and the Lambert's clamped cos *)
Plot[{DiffuseIntegral[x,1,1], DiffuseIntegral[x,3,1], DiffuseIntegral[x,6,1], Max[Cos[x],0]}, {x, 0, Pi}]

I then make some hypothesis and plot a test function against the slice to visually see if it might fit:

(* We have to find a function of cos\[Theta] which could look similar to the DiffuseIntegral*)
Show[Plot[{Max[Cos[x], 0], Max[Cos[x]*0.5 + 0.5, 0]^3}, {x, 0, Pi}], plotIn2d]

Optimizing fitting parameters
Now we can define a function with a number of free parameters, and ask Mathematica to find a fit, a set of parameters that minimizes the distance between the function and the sample set. NonlinearModelFit is used here, that is a slightly more controllable variant of FindFit. This will take quite some time as the function to fit is quite non linear (due to the Max and Clamps).

A word on the function I've selected here and why. I'm sure something better can be done, this is not much more than a simple test, but my reasoning here, after visually inspecting the function to approximate, is that the dependency on the radius is rather simple, while the shape that is dependent on the angle is quite complex.
That's why I started plotting the function in two dimensional graphs that are slices with a fixed radius. Once I've found a possible function that could approximate that with some free parameters, I've modeled the dependency on the radius of these parameters as simple linear equations.
Also I wanted the function to become Lambertian for large radiuses, and I did that by literally having part of the fitting function be a linear interpolation towards the cosine. This constraint is questionable probably for many applications you could just clamp the radius that you will use in a given range, and care only about finding a good fit in that range.

(* Let's find the fit now, first we have to define a parametric function, the fitting process will optimize the parameters to minimize the error *)

(* piecewise linear models... *)
(* fitFunction[\[Theta]_,r_] := Max[Cos[\[Theta]] * Min[(a0*r+a1),1] + Max[(a2*r+a3),0],0] *)
(* fitFunction[\[Theta]_,r_] := Max[Max[Cos[\[Theta]] * Min[(a0*r+a1),1] + Max[(a2*r+a3),0],0], Max[Cos[\[Theta]] * Min[(a4*r+a5),1] + Max[(a6*r+a7),0],0]] *)

(* nonlinear models... *)
(* fitFunction[\[Theta]_,r_] := (Cos[\[Theta]]*(a0*r+a1) + (a2*r+a3))^3 *)

fitFunction[\[Theta]_, r_] := (Cos[\[Theta]]*(a0*r + a1) + (a2*r + a3))^3 * Clip[a4*r + a5, {0, 1}] + Max[Cos[\[Theta]], 0] * (1 - Clip[a4*r + a5, {0, 1}])

fitting = NonlinearModelFit[dataThetaRVal, fitFunction[\[Theta], r], {a0, a1, a2, a3, a4, a5, a6, a7}, {\[Theta], r}, Method -> NMinimize (* We need to use NMinimize for non-smooth models *)]
fitParams = fitting["BestFitParameters"]
Plus @@ Abs /@ fitting["FitResiduals"] (* Print the sum of residuals *)

Original function and fitted one, plotted at an extended range to show extrapolation to Cos(theta)

We can now plot the fitted function and overlay it with the original one for a visual comparison. In order to use the fit we've found, we just replace the parameters we got from the NonlinearModelFit into the original fitFunction using the /. operator.

(* We plot the fit function over dataIntervalExt to check that it extrapolates well *)

dataFit = (x \[Function] {First[x], Last[x], fitFunction[First[x], Last[x]] /. fitParams}) /@ dataIntervalExt;

fitGraph = ListPlot3D[dataFit, InterpolationOrder -> 3, ColorFunction -> "SouthwestColors", Mesh -> None];
Show[fitGraph, cosGraph]
Show[fitGraph, dataGraph]

We can do much more, but this should be a good starting point. I've uploaded a PDF of the whole Mathematica notebook on ScribD here. Enjoy!

P.S. I'd bet everything here could be made ten times faster by applying these suggestions... Left to the reader :)

P.S. You might want to find some better example of Mathematica's goodness than my lame test code. You could start here or here. Also, for Italians only, I've found on Issuu an entire collection of McMicrocomputer magazines, by ADP! Incredible not only for its nostalgia value but also for all the Mathematica's articles by Francesco Romani that showed me more than ten years ago the power and beauty of this software and of maths in general.

What I've learned from shipping a deferred lighting renderer

2011-09-08T11:52:00.007-07:00

You're going to draw every mesh three times on average (considering shadows), if you don't fuck up the GPU badly, chances are that you're going to be CPU bound.

Cache-misses in your rendering pipeline are going to kill you.

Use indirections VERY sparingly. Control where resources can be created and destroyed to be able to avoid to have to reference count everything. Prefer "patching" pointers in place to keeping references around.
Don't iterate multiple times on "drawable" objects collections to isolate which ones to draw in each pass. Store only the draw data needed for a given pass. Scene trees are dumb but by now everyone should be well aware of that.

Parallel rendering is really important.

Especially if you don't double buffer and add frames of latecy here and there (bad for gameplay and memory), you won't find much to do later in the command buffer generation to keep the cores busy otherwise.

Batching, "sort-by-material" is needed.
My best bet on how to handle all this is still the old design I've blogged about of having every "drawable" object correspond to a drawcall and represent it as fixed length sort-key whose bits are an encoding of all the state needed for the draw.

You'll either need to have a robust geometrical pipeline to split/aggregate meshes and generate LODs and occluders...

...or you'll have to schedule a sizable chunk of your artist's time for that. And you'd better not think that you can do these optimizations at the very last...
On the upside, software occlusion culling is a big win and not that hard!

Carmack's megatextures (clipmaps) are not (only) attractive to achieve a given art-style, but they have in a deferred setting the big plus of requiring very few material shaders (as you don't need to combine textures, which is most of what material shaders do in a deferred lighting renderer) and less objects (no need of static decals), thus requiring less work on the CPU and making easier to split/merge meshes.
A tiled lighting stage (as opposed to the stencil marked volumes) is a good idea

...especially if you need shader variations in the lighting stage to render some decent materials, consoles have a one bit hi-stencil that won't help you

Point, spot and directional lights are not enough. You'll need some kind of ambient lighting devices to create volume.

If your game can stream in chunks (regions), don't think about continuous streaming.
Edge-filtering AA can work pretty decently and extremely fast. Not limited only to the final framebuffer...

PS3 early-z is a bitch, PC Dx9 tools are non-existent

Back from vacations, Parameter Databases poll results.

2011-09-07T17:58:00.001-07:00

Back after Siggraph and some well-deserved vacations, I have many posts to prepare. For now, here there are the results from my last poll.

Coast near my hometown

The poll was: "Parameter Database systems should use..." and here are the final results. Didn't get too many votes this time (32), maybe people were in vacation as well, or maybe the subject was boring, anyhow:

Reflection (53%)
Code-Generation (28%)
Handles/Dynamic Types (18%)
Immutable Data (in-game) (18%)
Data Inheritance (18%)
Schema Inheritance (18%)
Relational DB (15%)
Live-editing via tool (78%)
Live-editing in-game (40%)
Serialization via tool (53%)
Serialization in-game (34%)

Predictably I'd say, people prefer to keep editing and serialization in a tool instead of in-game, having the in-game client only to load/listen to the data changes.

As for the database type, relational systems do not seem to be favored, I guess most parameter systems are still key-values. Low scores are also assigned to dynamic typing, data and schema (class) inheritance.

What is maybe surprising that reflection wins over code-generation, even if in C++ there is no easy way to implement the former.

I've noticed how often we, as engineers, have a perverse aesthetic sense for which a C++ disgusting macro based reflection system is considered "prettier" than mixing different languages or crafting languages and tools.

To be fair, reflection could also be done more simply by parsing PDBs or using C++ parsers, but I doubt these are really widespread.

The good thing about reflection though is that code changes can "break" data, or require data changes, but the opposite is not true. With code-generation data changes can "break" code which is arguably worse.

If you're asking for crazy overtime, apologize first

2011-07-30T17:10:00.007-07:00

Here I'm going to use Team Bondi's emails as they appear in an article on eurogamer. But if you're been working in this industry for a while chances are that you've seen something very similar at least once. I'm using L.A.Noire developers as an example just because I found these emails todays but we all know excessive overtime is common, other recent examples that went public are the story of Kaos Studios, the letter from the spouses of Rockstar San Diego and of course the one that "started" them all, the EA_Spouse blog.

Extract from an email to the team:

[...]This is an amazing result for 4 hard years and I'm proud of what we've achieved this far. The game is huge in size and scope and will be a real breakthrough. We have almost re-invented the adventure game whilst including the action elements that people expect in a modern game. Its these action elements that we really need to tighten up.

That said, anyone who has worked on a game or film before knows that to make a AAA title is going to take a big push at the end to get it complete. This is not uncommon within our industry and while it's not ideal, it is what we need to do to get a polished result to the standard of the competition. To achieve this result we're introducing two new working practices, effective immediately:[...] The hours on Saturday will be compensated through the weekend working scheme, giving everyone the opportunity to take payment at the end of the project, or an extended holiday period. As I said this isn't ideal, but it is typical of what it takes to get a game finished.

Another example:

[...]That means that everyone is required to keep going until the milestone ships or your lead informs you that you have done all that you can for N10 and sub-alpha. Specifically this means in the last two weeks of the milestone you can expect pretty long days. It's "one in all in" until we get the Milestone shipped and get the game ready for testing. We need teamwork to get the game finished to the quality that we are after and that means being here to help a tester, a designer, an artist or programmer who needs your support to get their work finished. You are not required to work round the clock everyday up until the milestone ships but for the next six months we will need more from you than we ever have asked from you in the past. That's the nature of getting a AAA title out the door and into the hands of the playing public. Getting a result with this game means that the public finally get to enjoy the fruits of your hard work. It also means that we get to take a good break later this year and come back refreshed to work on some exciting new ideas for future projects.[...]

Again:

[...]To complete the project at this time, we require an extension of the ordinary hours of duty and we are asking people to give more hours. Putting a product to market of this size, scale and quality is going to require extra effort from everyone and while we are asking for it, and not saying it's easy, the company is perfectly happy to be flexible of commitments you have outside of the organization[...]

So there is a pattern to this. All these emails basically say:

We're making a great product
We should all be very proud, we're all on the same boat, let's do this
We need you to work harder
It sucks but this is how AAA games are made.

Now of course in many cases, managers who say this don't actually believe it's true. They know the project is not going in the right direction and they try keep the morale up by minimizing the problems (it's the way AAA games are made) and trying to gather the last energies of the team around the positive aspects of the product (this is a great thing, you've done an amazing job).

Now, while this can resonate well with some employees, unfortunately it's far from ideal for everyone. Remember that in a game company your audience is very varied, and when asking for sacrifices you want to motivate, but also not to piss anyone, especially not to piss your best talents! The problem here is that with this kind of communication you leave the more inquisitive types wondering if this is just a mediocre, stereotyped way of keeping the morale up or if the company does really lack the vision and the ability to make AAA titles in general.

There are plenty of studies about overtime and its risks I won't get into that discussion so let's even assume that you're not an idiot and you're not pushing your employees past the limit of diminishing returns thus actually hurting the project by asking to work more.

You need overtime, you believe that it's going to be positive and you need to communicate this decision to the team. You have to remember that you're asking for a sacrifice, even if you're in one of the few countries or companies in the industry that pay for it. You don't own your employees, and surely you don't want to lose your smarter talents

One of the worst things to say to your inquisitive audience (especially I might say, engineers) is that "it's not uncommon" for AAA games to require such rough stretches, because you're dealing with people that will not simply accept that fact (that is indeed, and sadly, true today), but ask themselves a few follow-up questions faster than a journalist.

What else is not uncommon in this industry? Companies that do not generate revenue? That fail to meet their targerts? Massive layoffs and studio closures? And even if crunch is common, how does that relate to good products? Are the best studios, the ones generating most revenue and great games, using cruch? Maybe I should polish my resumee, I hear that [California, Uk, Canada, Germany...] is wonderful this time of the year...

The fact is, we as an industry have to grow a lot. Crunch is common, but it's a mistake, and many people know this, especially some of the more experienced talents will know that's not the way you make great games.
Personally I've seen great games with zero overtime, bad ones with lots of overtime but also great games done with overtime but more importantly, bad games that made money and great games that didn't... We still have a lot to learn, as an industry, on how to make good games and how to make them in time and how to make money with them...
Every time you ask for crunch, you as a manager have failed. It can happen and it might be something that you'll need to ask for, but at the very least, you should apologize.

Let me suggest a different, more humble and realistic, communication style:

This project was hard, we had to face many hurdles. We should have been better prepared.
Despite the good work done by the team, we're not were we predicted we would be at this time.
We are apologies for not being able to create a schedule that avoided this, and we will definitely do better next time, we have learned some lessons from this.
We believe in the product, and this studio needs to deliver a great game in order to go forward.
Let's all help ship this product and make it a huge success. We'll need to work harder, and we'll try our best to compensate you for this extra effort after the end of the project.

Even better, provide example of what went wrong, of what are you going to do to make things better next time, of why you still strongly believe in the product and you think that this effort will indeed achieve the objective the company has set for it.

People need to believe that the future will be better, that there are some problems and not everything went as good as you wanted it to be but you know how to fix things. That there is a reason to stay and work with you! Just saying "this is great", "we're making something amazing", "let's go team" can be meaningless if not counterproductive if you present no evidence.

With this in mind, let's see how the first email could have been worded better:

This is an amazing result for 4 hard years and I'm proud of what we've achieved this far! We wanted to create something amazing and accepted the many risks involved in making an innovative product. The game is huge in size and scope and will be a real breakthrough. We have almost re-invented the adventure game whilst including the action elements that people expect in a modern game.

Unfortunately, even if the product looks good, we failed to account for everything that creating such an ambitious project entailed, we have learned a lot about how to make such a game during these years. Now, as the deadline for this game is coming, we still have some issues in our action elements that we really need to tighten up.

We know that crunching is bad for everyone and we'll offer some bonus holiday time at the end of the project as a way of partially compensating for this extra effort we're asking you. The studio needs to do to get a polished result to the standard of the competition in order to grow and move forward. To achieve this result we're introducing two new working practices, effective immediately:[...] The hours on Saturday will be compensated through the weekend working scheme, giving everyone the opportunity to take payment at the end of the project, or an extended holiday period.

Everyday Carry

2011-07-18T19:49:00.005-07:00

A few weeks ago I was looking around to shop a small utility knife. Went on youtube, searched for some reviews, there are really lots of resources.

In the end, I didn't buy anything, I went to a local shop even but nothing I saw looked like something a normal, decent human person would really carry without shame. And maybe that's for the better, because I spent already enough money in my past collecting fountain pens and cameras, I don't really need another compulsion.

In all this search though, I learned quite a bit about knives and steels and blade shapes. And about the average knife-carrying person. One thing that seems to be popular is to create an "everyday carry" or EDC "system". A collection of things that a person comfortably carries with him everyday.

Now on the internet most of these are about guns (usually a pair, just to be "safe") and knives (a pair there too) and other "tactical" stuff, I guess these persons really need to compensate for something (Intellect? I bet they also carry a very small penis).

But I also found that this EDC stuff is interesting, because when we carry something everyday it really tells a bit about us. It tells who we are and what we need, and who we would like to be and what we think we need (but we don't).

Now I'm fully aware that no one will car, but I started this blog for myself and I still keep it that way, so I thought it would be cool to have a snapshot of this aspect of me at thirty. Here it is, my current EDC :)

I usually wear a nice jacket (if the weather permits) and a pair of running shoes. In the picture the jacket is from Armani Collezioni and the shoes are Reebok Realflex.

Why? I've always liked jackets and casual look, and I love House :D

Eyeglasses. My current red "Harry Potter" looking ones are from Armani and they are photocromic.

Why? Because I need them. My eyesight it not bad and I still play sports without glasses. I never tried contacts and I don't care to try them. My father bought me this pair last time I went to Italy. They look a bit silly but I like them, and the photocromic stuff works really well (unlike the early attempts at that tech).

A cloth bag.

Why? Because under my sink I have too many plastic ones, so I always try not to carry home more of these.

An iPad (first gen).

Why? It's the single most important gadget I've ever owned. It replaced the big and heavy mess of printed paper and magazines I used to carry around everyday. I use iAnnotatePDF and Reeder everyday.

A Rhodia notebook.

Why? Writing on the iPad is a pain and I like scribbling and drawing. I also carry another less fancy notepad from some company, I found it in my mail. The Rhodia is cheaper and better (quality paper) than a Moleskine.

Pens and pencils. Currently: a red Sharpie, a red medium point Pilot liquid ink and a ~~Bic pencil that I stole from EA~~ pink Uni KuruToga pencil a very fine red japanese pen I got from my girlfriend, a big Faber-Castell eraser and a Griffin iPad stylus.

Why? Because I collect them, and I love writing and drawing. I don't really use them often (as I don't use often the pad) because at home and at work I have more writing instruments and pads and sticky notes and so on. The iPad stylus works decently but it does not make writing on the iPad really enjoyable and even drawing does not feel great. I used to draw a lot on my old winCe phone with a small stylus and a very simple drawing application, somehow the iPad feels worse.

A tin box with various cables.

Why? The battery on my phone does not last long so I carry its USB cable, I also carry some decent Plantronics phone earbuds that I don't use because I don't really talk much on the phone when I'm around, and at work I have better open can earphones for music and so on.

An Axiom seat bag for my bike.

Why? It fits my EDC camera well and it was cheap :D

A super-cheap ParkTool hex wrench set.

Why? You might always need to tune something.

My 3DS

Why? Because I bought it. Because it had 3d and it was cool. So far I never played with it. The best game there is is Zelda and I hate it, I hate Nintendo for making a cheap port and cashing all the money with zero effort. I'll probably sell this sucker.

A Lacie USB key, 64gb. Similar to this one.

Why? I always need to transfer stuff around and I like to carry some programs and pictures always with me. This USB key is terribly slow so I recently purchased a small pocket harddrive for when I need to transfer bigger amounts of data. The nice thing is that it's all metal, it does not have an usb connector soldered to a board that can easily bent (happens to me all the time with conventional usb keys)

My "small" digital camera. A Panasonic G1 with a Leica 20mm 1.7.

Why? I love photography and this camera is really great (especially paired with that lens, with the standard zoom one is half as good). I have an adapter to mount all my older Leica lenses with it. Unfortunately it's too big to fit my bag so I carry it with my bike, which is bad as really the best camera is the one you always have with you!

A Samsung Galaxy S t959 Android phone.

Why? I don't really mind phones much. I used to when I was a teen but when I relocated to Canada I just bought a very cheap (40$) prepaid one. The problem with that is that prepaid plans are stupidly expensive here, so when Wind came with an incredibly cheap $40 all-unlimited plan I switched. And now I had unlimited internet, so I needed a decent phone to use that. This is a T-Mobile (US) phone that works with the frequencies Wind use, and I found that Android is really better for phones than iOS. I had to work on it quite a bit but now I really like it!

A Leatherman Squirt P4 pocket tool

Why? It's very cool, with a small knife and pliers it's really the most useful tool of that size I've found.

A Tucano bag.

Why? After I bought my iPad I started carrying way less stuff with me (magazines, papers...) so I bought this bag to replace the small backpack I used to carry before!

And of course my wallet, home and bike keys.

Querying PDBs

2011-07-14T16:53:00.006-07:00

We're in the final stages of our game and this means that almost everyone is chasing and fixing crashes, often debugging retail builds from core dumps, having to deal with nasty problems most times without much aid from the debugger.

Sometimes you're just looking at the memory, trying to identify structures: executable regions, virtual tables, floats and so on. From there you might hope to recover the type of the variable you're looking in memory and today we got an email from people trying to do exactly that, chasing a structure from some sparse hints.

So I thought, how cool would it be if we could execute queries on the debug symbols to find such things!
Well it turns out it's really, really easy. One great tool that does something similar is SymbolSort, and it's written in C#, and it comes with sourcecode! Cool!

SymbolSort queries the PDB for global data symbols, here we are interested in global user defined types and their subtypes, it's a pretty similar thing. Also, Microsoft provides a Debugging Interface wrapped in a COM dll that does pretty much all you need, and it's trivial to call from C# or similar.

Of course debugging is only a small fraction of what you can do with PDBs, so this is really just an example to show how easy it is, from here you can do many nifty things like code-generating reflection, cross-referencing with profile captures to do coverage analysis or serialization modules and so on.

Disclaimer: I wrote this in half an hour. It's probably wrong and surely ugly. Play with it but don't trust it! It's just meant as an example of how easy it is to query PDBs via msdia.
In fact the test program I did is a bit more complex and complete than the one I posted here, that is a stripped down version that fits the blog better and IMHO is a better starting point. Also if you really plan to chase structures with this, keep in mind that this version does not recursively search into member structures and inherited stuff.

P.S. It turned out that this particular bug was caused by bad memory (the actual memory in the hardware - it happens quite often) so this exercise was ultimately useless :)

using System;

using System.Collections.Generic;

using Dia2Lib; // we need a reference to msdia90.dll in the project

namespace Test

{

class Program

{

private static void GetSymbols(IDiaSymbol root, List< IDiaSymbol> symbols, SymTagEnum symTag)

{

IDiaEnumSymbols enumSymbols;

root.findChildren(symTag, null, 0, out enumSymbols);

uint numSymbols = (uint)enumSymbols.count;

for (;;)

{

uint numFetched = 1;

IDiaSymbol diaSymbol;

enumSymbols.Next(numFetched, out diaSymbol, out numFetched);

if (diaSymbol == null || numFetched < 1)

break;

symbols.Add(diaSymbol);

}

private static bool IsMemberPointer(IDiaSymbol s) // Quick'n'dirty based on observation of 1 (one) pointer, I'm sure there are better ways

{

return ((s.type != null) &&

(s.type.name == null) &&

(s.type.type != null) &&

(s.type.type.name != null)

);

}

private static bool IsMemberPrimitive(IDiaSymbol s, ulong length) // I'm not entirely sure about this one too :)

{

return ((s.type == null) && s.length == length);

}

private static bool SymbolPredicate(IDiaSymbol s)

{ // see the IDiaSymbol documentation here: http://msdn.microsoft.com/en-us/library/w0edf0x4.aspx

// It's around this size...

if (!((s.length > 62) && (s.length < 67)))

return false;

List< IDiaSymbol> childSymbols = new List< IDiaSymbol>(); // Note: from what I've seen the symbols are arranges in the order they appear in the class/structure

GetSymbols(s, childSymbols, Dia2Lib.SymTagEnum.SymTagData); // TagData will get us all the member variables, TagNull will get us everything

// It has to have sub-symbols (fields)

if (childSymbols.Count == 0)

return false;

// One has to be a matrix

bool hasMatrix = false;

foreach (IDiaSymbol subS in childSymbols)

if ((subS.offset < 8) && // It should be one of the first members in memory

(subS.type != null) && // It's not a primitive type, so its type has to be a symbol

(subS.type.name != null) &&

(subS.type.name.ToLower().Contains("matrix4"))

)

hasMatrix = true;

if (!hasMatrix) return false;

// Another one is a pointer to a matrix...

bool hasPointer = false;

for (Int32 i = 0; i < childSymbols.Count; i++)

{

IDiaSymbol subS = childSymbols[i];

if (IsMemberPointer(subS)

&& (subS.type.type.name.ToLower().Contains("matrix4"))

)

{ //...followed by a 4 bytes integer

if (i < childSymbols.Count - 2)

if (IsMemberPrimitive(childSymbols[i + 1], 4))

hasPointer = true;

}

if (!hasPointer) return false;

return true;

}

static void Main(string[] args)

{

DiaSourceClass diaSource = new DiaSourceClass();

diaSource.loadDataFromPdb("game360_release.pdb");

//diaSource.loadDataForExe(filename, searchPath, null);

IDiaSession diaSession;

diaSource.openSession(out diaSession);

IDiaSymbol globalScope = diaSession.globalScope;

List< IDiaSymbol> globalSymbols = new List< IDiaSymbol>();

GetSymbols(globalScope, globalSymbols, Dia2Lib.SymTagEnum.SymTagUDT /* user defined type! */);

List< IDiaSymbol> matchingSymbols = globalSymbols.FindAll(SymbolPredicate);

foreach (IDiaSymbol s in matchingSymbols)

{

if(s.name!=null)

System.Console.WriteLine(s.name);

}