Da Krunch

The Road to Substance Modeler: VR Roots, Desktop Reinvention

2026-01-21T15:13:00.000-08:00

I volunteered to join the Substance Modeler team less than a year after it came to Adobe, from Oculus. The team was really strong, and it had hired a couple of top tier engineers to beef up the roster. On my end, I was a confident "toolsmith" and I really wanted them to succeed in their push to have a usable Beta in a very tight schedule. A few months later they surprised me by asking me to stay as a tech lead. While I am usually quite confident in my skills, it sure felt intimidating to be the tech lead of such a team, full of experts in VR and sculpting (I was neither). Nonetheless, I pulled up my sleeves and got to work, because building this gem wasn't going to happen overnight, and it wasn't going to be easy.

None of the information in this post is confidential, however it also reflects my views, not necessarily Adobe's.

Modeler's engines

In my previous post I wrote about the implications of deferring topology. Working on such system has a tactile immediacy that is quite delightful. The speed you can create and carve complex primary and secondary shapes feels quite amazing. But creation speed only happens if we have a geometry engine which can support the level of detail an ambitious artist demands.

Modeler ships with a sculpt engine and a rendering engine. The two work together to define our geometry real time pipeline. While Mediums feels quite different than Modeler, these two components were both initially inherited from it, and improved with the brilliant and diligent work of our engineers.

Sculpting engine

Joan of Arc, artwork by Danny McGrath

Let's dig in a bit and see how the digital sausage is made, so to speak. Modeler's sculpting engine is a library used to author Clay layers, the basic building block of an art piece. A Clay layer is primarily a hierarchy of blocks containing voxel data - a small block is 8x8x8 voxels. The ground truth is captured in the voxels. Voxels represent a narrow band signed distance field (SDF), meaning the value of a voxel is the signed distance from the closest surface, where 0 is the place where the surface lies. The level set data is clamped between -2.0 and +2.0 so many of the constant field blocks far from the zero isosurface can be compressed to a single value. Clay objects also have an associated GPU polygon representation, generated with a "marching cubes" technique (for CPU exported meshes, we switched to "dual marching cubes").

Next to this level set sits a color voxel grid, which is only defined on boundary areas (no volumetric color inside the objects). At a quick glance, this data structure is similar to OpenVDB, though the hierarchy details and the disk storage differ.

In terms of space, the data of each Clay layer feels practically unbounded on all three axes (the range was +/- 1M), and that is a great improvement from Medium (+/- 1024), where artists had to carefully manage their layer bounding boxes. In Modeler, they can just explore and create shapes of any size without worrying about their distance from the origin, regardless of the voxel density.

When an artist operates on clay using one of Modeler's tools, the engine can quickly detect which blocks were affected and only recompute the necessary pieces of the associated polygon mesh. The LODs are computed immediately, too, by downsampling the SDF and using the same marching cubes algorithm.

Here's the kicker. Initially we thought that volume authoring was the most expressive tool for an artist, but it turned out that high precision tasks like micro-displacement and crease control, operating on the surface rather than the volume gives better control. So over time we added or rewrote some volume tools into surface tools.

Here's how surface tools work in a world, where the ground truth is still voxels: we work on the associated GPU polygon mesh directly. Small vector offsets are applied on each vertex in the area of effect of the tool, then the voxel grid is quickly recomputed in the areas affected. The respective portion of mesh is immediately recomputed so we always have a fresh mesh with uniform tessellation. During many sculpting operations, this happens many times per second, reaching buttery real time update rates. When the operations are very frequent and subtle, there is a minor numerical drift. Initially the drift caused some small blocky artifacts, but we were able to reduce it down to a slight smoothing effect.

If a sculpt operation works on a very large, very dense mesh however, the operation can take over a second. That is entirely acceptable and common on desktop applications which come equipped with progress bars and dialogs, but we have a VR device, where any time you drop under 90fps, bad things happen to the user, including nausea and dizziness. We don't let that happen, because of how our rendering engine works.

Rendering engine

Rendering and sculpting are thread-decoupled in Modeler. Rendering is entirely focused on hitting target frame rates on VR devices. During heavy sculpting operations, happening on a separate set of threads, the engine continues rendering the meshes representing the previous committed voxel state unobstructed. The mesh being displayed is not the in-progress mesh. The new mesh replaces the old one on screen only once the volume is updated and the submeshes recomputed.

In Modeler, rendering large scenes is made possible due to multiple techniques. A Modeler scene is made of layers assembled in a scene graph, much like in many DCC tools. The scene graph supports instancing, so much data can be explicitly deduplicated. However, that is a creative choice, not an optimization, because we achieve the same optimization implicitly.

Modeler makes broad use of COW. A prime example is how our heavy voxel data is split in small hierarchical blocks (as described for the sculpt engine) but blocks with identical data are not duplicated, in memory or on disk. So if, say, a sculpt of a man is duplicated (not instanced), there is still only one in memory. If one of the copies gets a remodel of the right hand, the rest of the body data will remain shared between the two sculpts (even if they are each made of a single voxel object). The savings are implicit, the user does not need to create explicit instances of a model for this to work.

The submeshes and the LODs associated with each block are also deduplicated this way, so that even if we have to display incredibly detailed scenes, the rendering engine only has to compute a very small fraction of them at any given frame. Our LODs are selected and smoothly interpolated based on the submesh size on screen, using a variant of Transvoxel.

We then apply backface, frustum and occlusion culling. This all contributes to handle billions of polygon (in terms of capacity) in a massive sculpted scene. We had solid engines which would carry Modeler far, but Modeler was not a product that was ready to ship yet.

Adobe-fication of Medium

Taking a whimsical, fun tool such as Oculus Medium and turning it into a professional sculpting tool for the Substance Ecosystem at Adobe included quite a few steps. We expected and looked forward to the product work - change the UI, improve workflows, more advanced options. When it came to organizational work though, the workload was surprisingly substantial, especially for a team largely composed of hardcore graphics people.

We cheekily nicknamed these changes the "Adobe Tax". It included creating and aligning all the infrastructure across our Substance products related to build and CI, security, analytics, crash reports, as well as licensing and legal. It involved rewriting or refactoring big swaths of code and negotiating with many teams in our organization as well as the others in Adobe.

Keep in mind, the Substance group as a whole was also new to Adobe, as we came from a relatively recent acquisition ourselves. The Adobe Tax felt like a burden at times, but being part of Adobe had incredible perks. We met an army of excited and renowned researchers. We suddenly had access to resources and allies in business, engineering, law and marketing. We were also able to use some Adobe funds to support open source projects we relied on, which was a big priority for us.

I ended up volunteering for several of these because I was the tech lead and I wanted to make sure the graphics talent on the team would stay focused and motivated. It wasn't always fun, but I learned a lot and got to meet incredible professionals in other teams who would still be strangers if I hadn't stuck my neck out.

VR and 6 Degrees Of Freedom

So after a year or so, we just got out of alpha and we had a voxel-based sculpting tool. There were a few such tools around those days. Because we had a very compressed Alpha stage, Modeler was still quite rough around the edges and not quite an obvious sell. Based on what I learned presenting at conferences like GDC and meeting artists, it was clear that Modeler's (and Medium's) main differentiator was that it was first and foremost a Virtual Reality (VR) tool. VR morphs the act of sculpting into a much more visceral experience, where you can see and feel your way around your creation in a way that a desktop tool just doesn't.

Even more than the visual immersion, what makes sculpting and painting in VR compelling is its 6 degrees of freedom (DOF) controllers, which approximate much more closely the way artists interact with physical sculpts. The result is that while desktop sculpting tools attract digital artists that are already familiar with 3D, VR sculpting tools are more compelling to traditional artists, to those who would rather not touch a mouse and keyboard, and would prefer sketching on paper or working with clay and marble.

Tribal Mech Warfare, Artwork by Tomi Väisänen

What really defines Modeler's success for our VR users is: it makes modeling fun again. That was a recurring quote from our users. It makes it easy and fast to transfer what is in your mind into the digital world. It removes the barriers that separate concept work from modeling work.

The catch is that VR, at the time of writing, is still a rather small niche in the creative tooling landscape. A VR-first tool such as Modeler was in its early days really limited its user reach. It took a lot of strong will and optimism from our VP, to invest in a specialized team such as ours. It's worth remembering that Adobe normally deals with millions of users in its more successful applications. So in our search for a larger market fit, all the while improving our VR tooling, we had to pivot our focus away from VR and to a desktop-first sculpting tool.

Bringing Modeler to desktop

Making a hybrid VR/desktop tool is no joke. While other applications in the Substance ecosystem are typically based on Qt and QML, over time we replaced the legacy Medium UI system with two custom UI systems. Both were "immediate", in the way ImGui is, that is, little to no state is preserved across frames. These systems were aimed at allowing us to code our widgets, controls and dialogs simultaneously for VR and desktop.

By design we tried to maximize consistency between VR, which was well understood for us, and desktop, for familiarity's sake. But in time it became apparent that putting too much emphasis on consistency between the two worlds ended up doing a disservice to both. A successful UX for 6 DOF and VR does not map very well to a screen, mouse and keyboard. The reverse is even worse.

To start from a few trivial examples, in VR, too many panels crowd the working space, while on desktop they can be organized in dockable panels across multiple screens (which, to be clear, we didn't get to implement). On desktop, precisely clicking is easy but relying too much on dragging can induce repetitive strain injuries, something I'm well acquainted with. On the other hand, in VR aiming precisely on sliders can be eye straining and frustrating while dragging objects around the scene feels very natural.

Our early VR versions had no hierarchy view or panel. The scene hierarchy felt intuitive enough through "direct scoping" in VR. Entering and exiting "scopes" was as easy as a "hover and tap" action, accompanied by appropriate sound effects. However, desktop users felt the need for more familiar views such as a scene outliner. Adding an outliner for desktop effectively created needs we didn't have, such as selecting objects that are not siblings. That created serious complications to how the engine was structured, from the UX down to the core.

The VR render of the detailed geometry was, relatively speaking, fairly crude. That was acceptable in VR, but the need for a quality render became apparent on the desktop version. To tackle that we added support for custom cameras and lights. We also added a whole new GPU accelerated ray traced renderer, which would not have worked on a VR app (progressive rendering does not do well with moving cameras) or with our authoring tools that relied on vertex shaders. That caused some trouble while we were restructuring our rendering engine to fit in the new features without affecting the frame rate in VR.

Non-destructive Primitives

One other thing really helped make a desktop-first take of Modeler work well. It was a new type of data: the non-destructive primitives (in the app called simply "primitives"). These metaball reminiscent, Neo-like SDF functions can be combined in arbitrary ways. They bring back a workflow that is procedural rather than manual, non-destructive rather than viscerally physical. We had many desktop artists switch to use these almost exclusively.

Unlike Neo, they are still rendered as if they were made of Clay - maintaining the notions of volatile polygons and deferred topology. If we compare to ray marched SDF systems, this had advantages and disadvantages:

Among the advantages, we had a way to choose the render and compute quality of each component separately, in terms of voxel density. It helped keep the rendering pipeline clear, streamlined and extremely optimized. We could always up-res a set of primitives and the clay version would be analytically recomputed, with no loss of quality. It also gave a very good preview of what the mesh would look if exported "raw".

Among the disadvantages, well, we had to choose the render and compute quality of each component separately. That meant the user has to make some choices in terms of quality/performance that they wouldn't need to do in Shader Toy or Neo. It also meant the quality is determined by the voxel and mesh density rather than just the pixel resolution of the render, so when the camera gets really close, polygons become discernible.

Once the primitives shipped, the desktop version of Modeler finally started to feel like a real product. But switching to a desktop-first application was just one of the big adjustments we had to make. We started developing our alpha version during the year of the peak Metaverse hype. Then came other hypes. NFTs, crypto, Gen-AI, the new Mac processors, the iPad Pro, to name a few. Each time it felt like we needed to adjust the roadmap and learn new skills. However, despite the excitement in some of the leadership, our product was built within our core values and in the end we only worked on a couple initiatives. Here they are.

Mac port

When we started working on Modeler, it was a Windows-only tool. In fact, since much of the team and Medium itself came from the Oculus division at Facebook (now known as Meta Reality Labs), Modeler was working only on Oculus hardware, which is Windows-only. Given these restrictions, it made sense at the time to have the project be built in Visual Studio.

Sentaro Emotional Support Device

Artwork by Vladimir Petrovic

When the port to other platforms became more likely in our quest to broaden the user base, we had to consider making the project more portable so I started converting the project to CMake. As I was making progress, a saw a demo of a more recent cross platform meta-build system, internal to Adobe but rather similar in principle to Bazel. I tried converting a few dependencies to it. The difference was stark: the declarative nature, dependency handling and binary caching got me making progress a lot faster, so I ended up abandoning the CMake port. The debate between CMake and that Adobe meta-build system for other products is still ongoing and while there is no clear winner due to technical tradeoffs, Modeler stayed with the Adobe one.

The decision came to make a Mac port. We expected it because "When is it coming for Mac?" was the most common question we got on the Discord servers. At least for me, it was not clear however how many more users we would get, being part of the Substance ecosystem at Adobe. Adobe's creative user base is largely composed of Mac users. On the other hand, in the Substance ecosystem, Mac adoption is rather small, due to the fact that our biggest demographics were game developers (mostly on Windows) and VFX/animation (mostly on Linux). Also, a Mac port was going to be a desktop only version, because the newly released Vision Pro had (at that time) some imposed rendering API constraints that made it impossible for us to achieve the experience we wanted, so betting on a Mac port was not a guaranteed win.

We hired a brilliant new engineer in our team who teamed up with an equally brilliant peer at Apple. The two prepared a version of Modeler that ran on Mac hardware and macOS. It took about a year, and I didn't follow all the details, but it required getting the code to compile with Clang (which has stricter warnings and differences in the STL implementations, among other things), losing or replacing some Windows-only dependencies and a lot of Metal shader work. I got a chance to try it, and it worked really well.

The Mac build was in QA when it was decided not to ship it yet (and to date, it has not launched). It was a bit of a shock: we had been announcing the Mac version was "coming soon" for a while, and we sure felt a mix of disappointment and relief across the team at the beginning. But it was also the right call: we really needed to focus on the desktop and primitives work, which was critical to expand our user base, and not spend our team's limited bandwidth in the inevitable initial issues that come up when a port on a new platform gets released.

Modeler and AI

"But what about AI, Davide?". Don't worry, I didn't forget you. Yes, all tools in my org and beyond were asked to consider using Generative AI and evaluate its impact on the users. And evaluate, we did.

Turns out, to the surprise of nobody, our user base of professional and enthusiast sculptors weren't excited about Gen-AI if it was used to generate shapes. And that makes sense, since creating shapes is the fun and creative part of the process. We had lots of people ask about Gen-AI UV generation though, which is always a bit of a chore. And since sculptors aren't always lighting and photography experts, they didn't mind the idea of having Gen-AI make a better render of their sculpture, as long as it was accurate. So we did a bit of experimenting around using 3D shapes to guide GeniAI imagery, and we did get to something working and cool, but we also decided that Modeler was a 3D tool for 3D outputs (meshes) not 2D outputs (images). Adding features around rendering and GeniAI, that wasn't helpful for the sculpting process itself, was more of a distraction than an improvement and would have detracted from the creative "flow" that Modeler strived to achieve.

Some AI did ship in Modeler though. It was my pet project, and the evolution of an old internal hackathon. It used slightly older types of AI, that used deep learning to create embedding of 3D shapes into multi dimensional vectors. This technology, written in collaboration with Adobe Research and which I nicknamed Block to Stock, is what powers the kit bashing features of Modeler. The basic idea is: we indexed all of the Substance 3D Assets and stored their information in a small database, shipped with Modeler. The user sketches some shapes, maybe types a word, and voila, the UI proposes a number of models (human-generated by our crew of artists) from the Substance Source repository. On a click, you can swap your model (in place) with the one proposed. No generated geometry, the AI was used to find the closest matches to your sketch in the repository.

Block to Stock in action. Artwork by Gio Nakpil

To tell the truth, only a couple of years later, this tech feels obsolete already. Gen-AI models are looking more and more like the ones people make: perfect quads, believable UVs and good shapes. Searching a database of human-made models may start to feel limiting. In the end, we stuck to our choices, to show respect to the craft of our incredible users, whose talent and passion outshines anything an AI will generate for us any time soon.

Modeler taught me more than I expected: about VR, about graphics, about AI, about teams — and about how hard it is to build something that feels simple. More importantly, it reminded me that tools come and go, technologies rise and fall, but the spark that drives someone to create — that part stays. Modeler was built for that spark, and I’m proud of the part I played.

Deferring topology in digital sculpting

2025-11-14T16:45:00.000-08:00

During this long hiatus, with my brain in the trenches of working at Adobe and my free time well spent on parenting our lovely little one, I have been missing writing. Now that my 2 years at Allegorithmic and 5 years at Adobe awarded me some time for a short sabbatical, I have been thinking about how modeling was the first thing I took up when approaching Computer Graphics, and how it still influences what I do so many years later. So here are some of my reflections on the discipline, as well as my time spent as a tech lead in the Substance teams as especially on Substance 3D Modeler.

Screenshot from a working session on Substance 3D Modeler

Modeling is a craft more than an art form

Fan art project I did in college, 15 years ago

Ok, don't shoot yet. Hear me out. When I started getting into CG, I used to think I would be a professional modeler. Modeling as a hobbyist felt to me like an exquisitely creative process. That impression was shaped by the incredible work I saw posted on CG Talk and similar sites by extremely talented generalists, who could do everything from concept to rendering. I never minded learning the tools, so picking up new software never was more of a fun challenge than a chore, because at my core I a have always been fascinated by CG tools and their making, sometimes even more than the final product.

While my early career steered me away from modeling and into shading pretty quickly, it also became apparent that in larger production realities a large portion of the creative work is done outside the modeling realm, and often by someone else - it was done in the concept stage, usually in 2d renditions, or clay sculpts, where characters and environments are designed without the limitations and technical knowledge required by existing 3D software. In other words, the creative step felt like it was controlled by talented traditional, 2D and clay concept artists for whom modeling tools were holding them back rather than empowering them.

On the other side of the hand-off, we have highly skilled modelers. While it does require good artistry, a great eye for detail and a keen eye for aesthetics, traditional polygon modeling in the VFX and animation productions is in many ways a technical craft. Positioning edge loops, getting the poly count right, preparing for rigging and negotiating with animators can take more time than getting the shapes right. And that's before we even mention UVs, whose ownership sometimes feels like a hot potato, to be assigned to someone else if possible.

Sculpting: deferring the topology problem

Dragon sculpt by
Damien Guimoneau

(zbrush/substance 3d Painter)

This equation changes when we look at "sculpting" tools. In a major shift, the concern of polygon topology, central to modeling, is shifted "right" in the pipeline, that is, delayed until the primary and secondary shapes, and sometimes even the details, are fully formed in a digital 3D space, not in wet clay. In our industry and at the time of writing, the main player is ZBrush. Without going into the technical details, sculpting tools have a huge role in empowering artists to create shapes directly on densely tessellated polygon meshes. As the shapes keep morphing in the creative process, some of these tools can handily re-topologize portions of the artpiece, as the underlying polygons get stretched too far into very long, thin strips.

The final result of this process is a very dense mesh. However, if we temporarily disregard downstream technology like Epic's Nanite or NVidia's Meta Geometry, what is usually needed in production is more akin to a mesh with much less density, nice topology properties, well distributed UVs and a baked-out normal and/or displacement map, which makes up for the loss in detail. Also the mesh should preferably be quad-dominant and free of spirals of edges that make rigging so unwieldy.

Creating shapes while abandoning polygons

There is another class of sculpting tools that treats topology as even more volatile, to borrow a C++ term, and uses a different type of data as its "ground truth": Signed Distance Fields, or SDF.

Project Neo uses analytical SDF for modeling
Art by Luc Chamerlat

Some, like Adobe's Project NEO, but also many of the creations on the popular site ShaderToy use analytical SDF to build nondestructive workflows. Models are created from implicit mathematical functions that get combined using some customized boolean logic. The SDF is rendered directly as is, meaning there are no polygons involved at all. This technique is very powerful and can scale very well for very smooth, complex shapes that would need a lot of polygons to reach a similar quality. While there are recent techniques to greatly optimize rendering, the performance of this technique tends to degrade with the number of pixels and with the number of primitives (spheres, cubes) and the radius of their blending (smooth, bevel, etc). This is because the number of computations per pixel is proportional to the number of primitives that need to be considered in a given are of the render.

Deferring topology in Substance 3D Modeler

There are other tools, like Oculus Medium and its successor Substance 3D Modeler, that live in between these worlds. In additional to using analytical SDF, they store a discretized version of these SDFs in voxel grids and treat them as ground truth. This technique is also used (at much lower resolutions) by videogames, for terrain sculpting, especially when the terrain can be sensitive to destruction and needs to be recomputed. Polygons are computed from the 3d grids.

Teahouse, created by Joshua Eiten in Substance 3D Modeler

Clay

As a tribute to traditional sculpting - which is a hobby of some in the team, in Modeler the main data type is internally called Clay. Clay layers are voxel grids where each voxel registers its distance to the closest surface as well as, optionally, a color. The distances are clamped between -2 and +2 (just enough for the clay tools to work well), meaning we only store a thin band of data around the surface, and the rest is compressed away. Clay layers have a defined density, and unlike in Medium, they have a virtually infinite extent and are not limited by a fixed size bounding box.

Tools that work on clay volumes have an immediacy that is delightful and very tactile. The speed they can create and carve complex primary and secondary shapes feels quite amazing, especially in VR. Initially, most sculpting tools worked directly on clay volumes, but with time we found that introducing tools (such as build-up and smooth) that work directly on the mesh representation of such volumes gave better result on secondary shapes and surface detailing. That does not change the fact that all the polygons are volatile and ephemeral: after each operation on the mesh, we recompute the affected voxel grid and then recalculate a new polygon mesh off the new voxel grid, maintaining the voxel grid as ground truth. This effectively produces a freshly retopologized mesh fragment many, many times per second.

In fact we not only compute the mesh that represents the grid. In addition to the full resolution mesh, we also continuously generate its first three LODs, sampling the grid at sparser intervals. This makes it possible to use algorithms akin to transvoxel to smoothly transition between them and render large scenes in real time. How we got billions of volatile polygons to render smoothly in VR is worthy of a separate post.

The price of deferral

If your goal is a 3D print or a pretty picture (and your renderer can handle high res models), or guiding AI generative models, relinquishing the control over the precise polygons is quite liberating. If on the other hand your end product is a production quality mesh, to be used in some CG pipeline, that's not quite a free lunch.

Seven Dwarfs, created by Daniele Trovato in Substance 3D Modeler

The problem of topology and maybe UVs is deferred to after the creative process is done. We move from a world where topology is the very medium the craft is applied to, to a post process. The tension in this process is: how do you tell the computer what one artist's idea of a perfect output is, but also spend as little attention to it as possible since it should just "do it right". The process should ideally be entirely automated, and ideally would be very fast. At the time of writing, that ideal is still not quite reached, by any tool in the market that I can see, but great strides are still happening and papers are still being published. That too is worthy of a separate post.

That said, there are signs in the way AI is entering the industry that make me think maybe it won't always matter as much.

We need to talk about digital backlots

2020-03-06T17:30:00.001-08:00

Having a great big library of assets, of any kind, is a lot of work. I have run into this problem and its increasingly sophisticated solutions all over my CG career.

After I joined a panel about it at MIFA (Annecy) a couple of years ago, I was amazed at how little knowledge is generally available on the topic. So I spent some time talking about this with my friends in the industry, from ILM, to Pixar, to EA in gaming as well as in commercial libraries in Adobe, Turbosquid, Google.

The result is this piece I had initially written for this blog, but that I think is worth sharing on a larger platform: the Substance 3D Magazine. Enjoy!

MaterialX as a Portable Shader Graph

2020-02-17T09:06:00.002-08:00

The MaterialX initiative, started by Lucasfilm, has been around for a several years and I have been looking at it and in some form contributing since before I left Pixar. Perhaps the main reason I have stayed interested in MaterialX over the years is its relationship with USD.

When we designed the UsdShade schemas, we did it carefully so it would evolve in lockstep with MaterialX and have entirely compatible concepts for Looks, Collections, Shaders, Materials. However, as an engineering lead, I took the key (looking back, maybe controversial) choice of leaving specific shading math out of them. That was deliberate, because I was designing the schemas for Pixar use. Holding 100% of the production for our own movies, we were less concerned about exchange (which is still true today). More importantly, at the time, our custom shader graphs were rather huge and they were assembled with nodes that were not easily representable in MaterialX constructs, both in terms of UX and in terms of RenderMan-specific side effects. In fact, when we cared about portability, we ended up baking down all the shaders to textures, effectively losing all their proceduralism. So my realistic goal for the collaboration with Lucasfilm was that anyone using MaterialX could work well with USD, while knowing that Pixar had little appetite for MaterialX.

Still, joining Allegorithmic (and later Adobe) has made more clear to me how USD can improve its portability by leveraging MaterialX, and how MaterialX holds value even outside of USD. Things of course did evolve once I left. UsdMtlx is a significant step forward, though I don't think we are done.

Building patterns and illumination through shaders is an art. And doing it through graphs is, in my opinion, still the best way to do it. Making those shaders useful and portable no matter what render target you are working on is critical for many reasons. For Adobe, it helps making our Substance Materials more general and portable. For film and game studios, it helps to exchange shaders without losing their feature set by baking everything down to textures.

But to get started, we needed a good node graph toolset. Substance Designer was the obvious choice, as it is already a node-based tool where we build materials. We had to get Substance Designer extended so we could add more, arbitrarily complex graphs with new semantics. So in 2019 we pushed this Labs project for SIGGRAPH.

We presented a collaboration with LucasFilm and Autodesk, in a push to help standardize shading pattern generation. In this article on Substance Magazine I describe more in depth what we did and how.

©&TM Lucasfilm Ltd. Used with Permission

A lot of credit for this work goes also to David Larsson, a great friend and colleague who also shares my passion on this topic. He did a ton of the work and helped with the collaboration with our counterparts in the Autodesk team. I should also thank our friends at Lucasfilm: Jonathan Stone and Doug Smythe - who have always been great at accepting our inputs and helping us through tough design decisions, while I was at Pixar and even now at Adobe. Finally, special thanks to the tireless Substance Designer team, as well as Jerome and Andrea in our Special Projects group, for their deep MDL knowledge and for keeping us real.

Enjoy!

Au revoir, France

2020-01-05T22:05:00.002-08:00

When I left Pixar, I told myself and everyone that Mayah and I were going to try to live in France for at least a year, maybe two. Then you never know in life, we might stay for good. Life was getting comfortable in San Francisco, years were flying by and one thing I learnt that makes it slow down for me is getting out of my comfort zone and try something different. It was my eighth international move, so I knew something about that.

As exciting as it was for me, leaving a dream job to move to France was even more so for some of my friends. Some told me I was living their secret dream and some even booked the flight to come visit us in France before we got our own.

As it turns out (duh) moving in two is more complicated than moving alone. How you feel about a place gets more nuanced. The reasons to be in a place or another stack up, making it difficult to make up your mind. In the end, after almost exactly two years, we decided to move back and as of three days ago, we are living in San Francisco again.

Coming back to the Bay area is a joyful thing. Seeing our friends and family really is making it easy. Leaving Lyon, on the other hand, wasn't easy at all. We fell in love with that city, the friends we made there, and the life we had. Making Lyon feel like home was actually easy. Perhaps it's because looking at the state of things in each place, I would say that Lyon felt more civilized. Culture is everywhere and approachable, with intimate concerts, plays, workshops. Streets are clean, walking around at night feels (and is) safe, even pleasant with breathtaking views of lit up historical buildings, hills, river banks at every turn. Streets are sprawling with affordable cafes, bakeries and bistros, and people are out enjoying them. Public transport is clean fast and safe, people on it often read books or connect with each other. And the markets! oh the French markets in Lyon are such a treat for someone who loves food and cooking.

Speaking of cooking, as promised when we decided to move to the French gourmet capital Lyon, I did a number of cooking courses. Initially it was hard to follow, as they were entirely in French, in the evening, after a full day of work. But I stuck it out and gained a great deal of theoretical and practical depth.

We got a nice apartment with a lovely kitchen, which we made good use of, making some new French food and refining some of our classic recipes using French techniques, which as extremely applicable in many other cuisines.

Time did slow down for us, as I had hoped. The last two years may have flown for some, but it felt like five for us. Everything was new. Our home, the language we spoke everyday, the local culture, the landscapes, the opportunities for day trips. We could not see much of Mayah's family while we were there, though they did visit, along with a large amount of friends from Pixar and California. On the other hand, we were only four hours drive away from mine, so we often went to spend the weekends in Italy.

We organized a wedding party, almost two years after we got married, with my Italian family. We picked the remains of a little medieval fortress, now used as a brewery, in the valley of Susa, which connects Italy to France through the Alps.

These two years we also took advantage of our location to explore France a bit more, as I got to check a bunch of items off my bucket list. We went to see Paris and Versailles (where we ran into more Pixar friends of course). We explored the frosty Bretagne in the winter, along with Mont Saint Michel, and explored the idyllic cattle farms of the region.

We ate our way through the red hills of Dordogne in the Fall, road tripping the countryside with some of our favorite people. We dined on chestnuts and duck confits, and slept in farmhouses and castles, and we marveled as we saw ancient caves covered with prehistoric murals.

In the Summer we cruised across the dreamy landscapes of Provence, which was just like in those movies that I always felt were simply idealizing the romantic vibe of a blooming countryside and the exaggerated charms of the French joie-de-vivre. Turns out, that vibe is real. It's a unique place and I felt like moving there and opening a bed and breakfast. If I did't love my job I would probably have done it.

As the heat picked up - something that is easy to underestimate when you are used to the Bay Area climate - we decided hike the rocky cliffs of the French Riviera, and to bathe in the blue waters of the Mediterranean sea, to escape the burning heat of the city.

Speaking of bucket lists and French Rivieras, we attended the Cannes Film Festival. We walked the red carpets and enjoyed the local delicacies. We saw a screening of the Black Panther on the beach screen, presented by the Oakland-born director himself. I am positive we ran into a number of French and international celebrities, although we could not always recognize them.

Staying local was exciting too. Beyond Lyon, less than two hour bus ride away, lies Annecy. Besides having a beautiful lake and a lot of delicious Savoyard delicacies, it is the seat of the Annecy Animation Film Festival, which is a pilgrimage destination for many in our Animation world. We ran into a ton of friends from across the globe, which makes the CG world feel small and cozy, yet huge and marvelous.

So, was it worth leaving Pixar after ten years?
It's a bit tough, but I think so, as the life experiences we got made us grow, personally and as a couple, more than two extra years in SF.
Do I miss Pixar?
You bet! it was an amazing place with incredible people, and I am reminded of it every time I visit. There are things and people that I love and that I will always miss - although now I'm not so far, so I'll definitely see them more.

So... What am I up to in SF now then? I am not going back to Pixar, as many asked. Adobe, as you may know, has acquired Substance last year, and it is giving my teams a shot at doing something very impactful - and fun too. I am so very excited about the new challenges and opportunity ahead. Tomorrow morning I'll start my first day at the SF office!

Au revoir, France! I'll come visit soon again, because I miss you already.

Texture Streaming from Substance Painter

2018-12-12T01:11:00.001-08:00

Hi all!

I posted another tech piece on the Allegorithmic Blog.

This one is about the work we did last Summer to try and speed up the interaction between painting and rendering. This was a collaboration with RenderMan as well as with Appleseed.

While still a prototype, the results we got were great! We went from a workflow that required seconds to minutes, to a near-seamless stream of textures.

Here is the full video we showed.

Find out why and how we did it on the Allegorithmic Blog.

3D Photos on Facebook and Substance Painter

2018-11-05T06:16:00.000-08:00

Today I went tinkering with Substance Painter and the new 3D photos on Facebook. Turns out the feature is viewable, but not uploadable yet in France.

However, if you "like" the official Facebook 360 page, and restart your phone Facebook app, that feature will be enabled, as a "beta" before it becomes official.

I got it working, and it was quite fun! Here is my original Facebook post, using a sample scene provided with Substance Painter.

1. I had to write an openGL shader extension to output depth in a usable range. The code is pretty trivial, I am rerouting a function of gl_FragColor.z to _outputs.emissiveColor.xyz.

Here is my code snippet:

// change distance curve to something visually readable

float dist = pow(gl_FragCoord.z, 256.0);

// add contrast

dist = dist * 2.0 - 0.5;

// feed to the emission component of the pbr shader

outputs_.emissiveColor = vec3(dist);

2. Save screenshots with the color and the depth. There is no tool for that AFAICT, so I simply used my OS screenshot tool. The depth is quantized to 8 bits, but it's good enough for this.

3. Then, off to Gimp (photoshop, what not) to play with the depth contrast if needed and to crop it to 1200x800, or 800x1200 for Facebook to accept it. Layer the depth and color so they match. Export them separately. I suspect having different resolutions might in fact work, but I didn't test it.

4. Save to ICloud (There is definitely an android way too, but I used an IphoneX).

5. Import to your Iphone photos by saving it from ICloud.

6. In "Depth Cam", which I found available on the app store, load color and depth, then save photo.

7. Reload the combined photo in Depth Cam, and turn on "Portrait" mode. Save again to your Camera Roll, as "Facebook 3d Photo".

8. Upload from the Iphone facebook app. You need to specify you want to upload a 3d Photo, which will restrict the choice of valid photos to select from, to just "Portraits". Your depth images should be in there.

Again, if you are not in a region of the world with access to the early release of the 3d photos, you need to enter into the beta program. Or wait a couple of weeks, hopefully!

Finally, yes, there are artifacts. The extreme angles that expose what's behind the statue show some white areas.

This is due to the mesh, procedurally generated by the Facebook viewer. I think the issue is the discontinuity in depth at the edge of the statue, which is perhaps higher than most depths generated by photos. Likely, this will be fixed soon by smart people in their group, like Ocean Quigley.

Finally... thanks Ocean and Daxx for the initial pointers!

It’s a small (CG) world after all

2018-10-19T00:35:00.000-07:00

In my first Substance blog post, I talk about how we built an experimental, or rather educational (for us), build of Painter for the Oculus Rift.

Our 2018 Painter proof of concept on a Rift

Why Oculus you may ask? You would not be the only one. Why Oculus and not, for instance, StarVR, Microsoft, Playstation, Vive, or something else entirely. I am a Steam fan, for example, and have good friends in each of those groups.

As I say in the article, and to be fair, the Oculus team has been a fantastic partner and after our feedback, they reached out to us with just what we needed: a way to get something out there without having to redesign the whole UI, so we can get early feedback and insight, including from artists that may not be our traditional users. That's what we are interested in, when working in Labs, to shape future steps that the product groups may decide to take.

Nothing was announced of course, but it is easy to imagine this working with more platforms, probably in a more mature state.

A story that I did not put in the article but is fun to remember is a bit of an origin story.

At the same time as the first Oculus test was being presented at GDC 2015 by Jean-François, only a few blocks away, I was becoming good friend with my neighbors, Eric and Jodi. Eric is a veteran in both AAA and Indie games, and he moved to San Francisco to join Kixeye. After a while, we started meeting weekly for drinks and dinner, and we started playing Pandemic Legacy season 1, still one of my all time favorite board games, almost every week. We kept going even after they moved down the peninsula, when Eric joined Facebook, or rather Oculus.

Pandemic Legacy (season 1). Lots of fond memories. We just started Season 2.

A couple of years later I left Pixar, and Eric was also the first friend to visit us when we moved this winter to France. He was at that time starting to brainstorm about how to better increase the reach of Oculus in other fields. So it was not surprising that he was the one, this Summer, who reached out to my team to see if they could make life easier for DCCs.

I love how small the CG world can feel, even with so many people in it. I’m reminded of it every Siggraph. This year I went with my new badge, and met all my friends from MPC, Animal Logic, Pyro Studios, Pixar, Blisscomedia, and it felt like I never left.

Even moving continent, I still feel like I’m part of the same community.

Baking and procedural patterns

2018-08-09T06:01:00.000-07:00

I am actually interested in baking pastries too, but I think I shall tackle that in the fall maybe, when it gets cooler. Let's talk about baking procedural patterns instead!

A few months back I wrote about the challenges and solutions of Material Layering. The post was quite useful for a lot of old and new friends. More questions came up, and since I joined Allegorithmic I learned much more about extra-pixarian approaches. In this post I'll focus on patterns and baking.

The composition of a Material

Taking a step back, Materials like a "painted dusty metal" are effectively plug-ins to a renderer (like RenderMan). In a physically based renderer, the Integrator is responsible to answer to the question "what color is this pixel"? To do that, the Integrator asks questions to the Materials and to the Lights.

A material can be conceptually split into Patterns and Bxdf, or shader. The shader is responsible for the illumination response component, and it has a number of parameters. Some are constants (presented as color swatches, sliders, etc...). The ones that vary across the surface can be connected to Patterns, and are hence called "varying". Patterns define how those properties change over the surface and the volume of the objects being rendered.

Patterns are implicitly organized into logical node graphs. You don't always see that graph though. Sometimes the graph is exposed in tools such as Substance Designer, Slim (yeah! remember that one?), Maya's Hypershade, ShaderFX. Often times though, the graph is displayed in a simpler manner, as layers (Mari, Substance Painter, Photoshop), or hidden altogether behind a bunch of presets and sliders (similar to Lightroom and the upcoming Substance Alchemist).

Patterns: Procedural and Data

Patterns are not all the same in nature. I like to split them into Data and Procedurals.
Procedurals are a type of pattern that computes values on the fly, with math and/or any applicable algorithm. If you are familiar with 2d graphics, this split is rather analogous to Raster versus Vector Graphics. Procedural networks have many advantages:

Flexible at all resolutions.
You can "zoom" into a procedural as much as you want. It will in general always look good, within the limits of your parametrization floating point precision.
Dynamically adjustable.
Procedurals have parameters and changing them will simply make the procedural evaluate to a different result. This can happen during the render, at runtime.
Little memory Footprint
Procedurals are in general extremely cheap in terms of memory.
Procedurals can use any input: they don't always need UV coordinates, as they can key off any geometric, be it 1D (e.g. height, facing ratio, time), 2D (UVs), 3D (point, normals, uvw, reference positions) and 4D (arbitrary, time) coordinates.

There are some cons too:

Not always practical at render time
While the memory usage is low, the cpu/gpu load is another story. Computing a ton of procedurals in real time at each sample of your renderer can really kill the performance.

Example of aliasing (top portion)

Aliasing.
While zooming in gives good results, zooming out may result in artifacts that look like there are patterns where there shouldn't be. This is due to under-sampling, and called aliasing. This can be fixed if the procedural can be correctly filtered (or blurred) analytically, that is, using math rather than over-sampling it. Pixar's wavelet noise is an elegant example of a modified procedural that behaves well at variable distances, without exhibiting aliasing.

On the other hand, flat data is a type of pattern that outputs values that are read in form some form of file, be it a texture or a volume, or a point cloud, etc...

Flat data has advantages too:

It is fast to compute. In fact it hardly has any computation at all.
Filters well at a distance.
Quite the opposite of procedurals, textures are usually well equipped to provide pre-computed filtered results to be used when "zooming out" on the pattern.

And some cons:

The quality and detail of data based patterns is limited to the resolution of the data itself. If you zoom in too much, you may end up with grainy results
In a data based pattern, you trade off memory for speed. Loading, or streaming in, data will take some of your memory and bandwidth, and if the data is not available locally in a fast location (e.g. a local drive, or the RAM, or the GPU memory), it will affect the speed too.
To read data, you need some parametrization. This is typically, but not limited to, UVs. Having a good parametrization is key to generating and using data successfully.
Not all files behave the same way. If a renderer doesn't know how to handle large amounts of a certain type of file efficiently, you may lose more than you gain in speed. For example, we had to be judicious and figure out when it was actually worth baking to Ptex in our Renderman projects at Pixar, because Renderman does not have an architecture that is as optimized for Ptex files as much as Disney's Hyperion. Most of the times, UV-textures performed better (for those assets where we had good UVs)

Baking

Fortunately, any amount and combination of patterns can be converted into fast, performing data, as a pre-process. This step is what happens when you are loading a level of a game, or sometimes when a render is starting. This process is named Baking.

As cool as baking sounds, it is a lossy process, in two important ways:
Static data: once a procedural is baked to flat data (i.e. an image), it is no longer, well, procedural. That means you can no longer change its parameters and change it dynamically. Speaking of baking, this is just like you cannot change the ratio of butter after a croissant is baked.

Resolution: the baked result is fixed in resolution, and inherits all the properties of static data, including its limits in quality when zooming in. So one needs to be careful about the size of the bakes. This consideration becomes even more important when dealing with less trivial file formats, such as volumetric data, or Per Polygon Textures (Ptex).

In general, baking trades speed for memory: save on computation cycles and use pre-computed data instead, which will require space. This trade-off varies a lot depending on the incoming network. If it was cheap, it may not be worth it at all. But if the original network included a lot of data too, this process ends up saving both memory and time.

You also don't have to bake the whole pattern network. Placing strategically baking points to get the most savings (bake most procedural costly nodes) without losing all the quality and tweak-ability will get you the best of both worlds.
For example, keeping a high frequency noise to add variation where the texture resolution is not sufficient will save you from having to bake humongous textures. Adding a live HSV adjustment after the texture read will save you from having to bake multiple textures for each variation of a model, saving both time and memory.

Pattern graphs and Compositing graphs

I am going to talk a bit about Substance, not only because that's where I work, but because it's super relevant to illustrate the topic of patterns and baking. The nodes in Substance Designer are for the most part procedurals. Many of them - especially FxMaps and Pixel Samplers - are computationally intensive and are processed by the Substance Engine which is, at its heart, a compositor, and its outputs are images. That is why it can really be seen as a baking engine on steroids. Of course, leave it to the French to show us how baking is done!

Samples vs Buffers

To fully grasp the difference one needs to visualize what a compositing node is working on, as opposed to a GL or OSL pattern.

Live pattern graphs work on individual samples: a small set of values that sometimes includes their derivatives, such as in OSL.
Sure, that operation is then run on batches, often by the thousands, but each variable can only access its corresponding input, on the same pixel/texel/sample.
Concatenating operation of this type can be reduced to a code generation problem, as the output of one pattern is used as the input of the next one.
This type of patterns does not have implicit spatial limits, so it can create infinite variations (at least theoretically, you may still run into floating point precision issues, but that's not a very common problem).

A compositing graph, on the other hand, works on buffers. Each texel at coordinates [u, v] in the output buffer may optionally depend on any or all of the texels in the input buffer, rather than just on its corresponding input at coordinates [u, v]. This allows for much more powerful effects. You can have random access nodes such as blurs and warps and sub-procedural bombing which are often prohibitive features in a live render. To be fair, bombing is a technique that can be kept live (RenderMan ships with a texture bombing OSL), but you need to bomb pre-existing textures, while Substance can bomb live procedurals.
Compositing graphs are spatially limited to their output buffers and resolution, so their variation is limited to that space. In the texturing case, that can be attenuated by making them seamless when tiling.

In Random Access patterns, the result of a given output texel can be affected by any (or all) input texels

Combining the two

So we can further decompose a material into a pre-baking (data based) and a post-baking (procedural) network:

The two types of graphs, are really not in competition, and it is important to understand how to make the best of both in advanced material design that can scale in complex productions. Scanning, processing, and rendering is nice to make tile-able materials, but that's only good for some categories of assets.

One of my favorite pastries here in France is called Demi-Cuit, or half-baked. It's sort of a molten lava cake that was not frozen before baking. It also conceptualizes nicely the essence of getting the best of baked and procedural, still fluid patterns.

Disregarding the difference

The semantics of the two types of networks are quite different, and there are operations (e.g. blur) that are only practical in one of them.

But that may not be true in 20 years on in a completely new type of renderer. There are a lot of "nodes" or conceptual operations that are pretty much identical across the two types of graphs (add, multiply, texture read, perlin noise), which makes it tempting to go ahead and describe both networks in one, and let the renderer figure out where to insert the baking points as appropriate.

This is not a solved problem today, but the MaterialX project put forwards by Industrial Light and Magic is certainly trying to do just that.

For more info about this topic, check out my older post about Portable Material Descriptions.

Lyon heat, Labs, Siggraph and California

2018-08-08T10:13:00.001-07:00

Since we moved six months ago, we got to see how cold (and now how hot) Lyon can really get. Many people who moved to San Francisco say that they miss seasons, but I have to confess I appreciated the mild Bay Area weather. I did not miss is having to fight off mosquitoes all night long, and spending sleepless nights trying to stay hydrated in this heat wave. On the other hand, I discovered I really did miss some of these balmy nights that we am getting here in Lyon, walking around in shorts and eating ice cream.

Being in Europe, we got to travel a bunch, by train and by car. Not just for leisure though.. I had the opportunity to be the smiling face of Allegorithmic a few times, adding some Italian accent to our French tech company. I presented my view on Materials and layering at FMX, in Stuttgart, and rubbed shoulders with Google and SketchFab at a panel about Digital Asset Libraries at the MIFA, in Annecy. I'll write about those a bit more once the summer craze dies down.

So, six months! I started initially as an Animation and VFX advocate here, trying to get features we care about prioritized and clarified. That was one of my initial propositions anyway, as I get much better visibility on the challenges and solution from here.

Encouraged by Sébastien, more recently I focused my efforts on regrouping some talented engineers, scientists and creative technologists, into a team which we call Labs. The group's goal is take on riskier projects, test new partnerships, and create prototypes that show the potential of new techniques. Hopefully you will hear about us and our cool projects soon.

This is an exciting time, because next week, at Siggraph in Vancouver, I'll be representing Labs for the first time. First, at the MaterialX Birds of a Feather on Monday. My friend David Larsson did most of the work I'll be showing there.

And then you will find us at the Pixar RenderMan's Science Fair on Wednesday evening. That one will be awesome, as we are presenting something very cool that François Beaune and I have been banging on for a while. That, plus the fuzzy feeling of presenting alongside my friends at Pixar. I'm looking forward to it.

The Vancouver convention center is a might view on the cityscape

That's not it though. The following week I'll still be in the West Coast, roaming California, visiting film studios and seeing friends and family. It will be a hectic time, but I am ready. See you there, and stay tuned!

FMX 2018

2018-04-21T10:37:00.002-07:00

For several years I have heard of how cool FMX is, from my friends at Pixar, and how pretty Stuttgart is. That made me want to go check it out and meet the artists, engineers, technologists and scientists that build its community.

This year, finally, I was going to present Flow, our Pixar shading tool, at the Lighting and Rendering session, dedicated to Materials, and chaired by my friend Christophe Hery.

Well, that's not going to happen now that I live in France. So I will go to FMX, and present the Substance suite, instead. "Materials and Substance" will take place on Wednesday at 2pm, at the Lighting and Rendering session, dedicated to Materials, and chaired by my friend Christophe Hery.

I guess it was meant to be.

My talk will be part of a longer course, hosting some great talks from NVidia, ILM, Isotropix, Pixar as well. I'll focus on the composition of materials in a physically based renderer, as well as the challenges and goals of layering, baking and proceduralism. I'll try to capture, from a film-maker perspective, how the Substance suite fits in the middle of all this.

Come see us, or at least come say hi at the Allegorithmic booth. I'll be around all week!

Living Lyon

2018-04-16T14:59:00.001-07:00

A few weeks ago I heard the sentence “I was almost run over by a bicyclist. I got distracted, looking at someone crossing the road with a cake”.

French cuisine may be good, but French patisserie is incredible

Some days ago, I was waiting in line in front of the prefecture, waiting patiently for my turn in a long line of other immigrants. A croak made us look up, and we met the gaze of dozens of hungry looking crows looking down upon the fools, waiting for bureaucracy to happen.

And after a tiresome week marked by long distance bus transport (because, you know, train strikes), yesterday we went to a small indoor concert. We danced to the catchy tunes of a bare-footed band of white dudes singing in a Caribbean accent and playing some weird mix of jazz and afro-cuban music, using fiddles, guitars, accordions and clarinets. Next to us, a guy in a fedora was bouncing too, holding his basket of bread on top of his slate with cheese and charcuterie.

Maybe I'll get used to these little things that make me smile now, but I’m loving the new world we are living in. Lyon feels clean, safe, and it’s so darn pretty at any time of the day, but especially at night. Especially tonight, that it's so warm. We've been walking by the river, eating ice cream, listening to all the foreign tourists' languages, and watching a few cute couples awkwardly dancing swing to the tunes of 70's hits. It felt nostalgic, like I was on holiday, walking along the beach on an early Summer night in my early teens.

The Basilica of Fourviere is impossible to miss from anywhere in the city.

Ok, I'm not going to lie, adjusting here can be trying at the beginning.

Apartment renting agents require a bank account. Banks want a permanent residence and a phone number. Phone companies won’t give you a contract without a bank account and a permanent address. It is incredibly hard to update your address and phone number after that. The amount and type of documents needed for any given appointment is quite unpredictable. But after all those little hurdles, it's worth it.

The whole city is walkable.

French is harder than I remember it. I studied 8 years in school, but never using it let those French neurons be overwritten with Spanish during my stay in Madrid. So I’m slowing making my way up that learning cliff again. It doesn't help that my coworkers are both nice and want to practice English.

Speaking of, I am learning so much about so many things. In my newly expanded industry, I'm no longer the expert, and climbing back out of the pits of my newly found ignorance demands a lot of my spare time.

Pumpkin cream with bacon emulsion.

Maybe a bit prematurely, given my French language proficiency, but I have been taking a long course in French cooking techniques. Those same ones that defined the gold standard for the western culinary world. And yes, French cuisine is very technical, but what I'm learning can be applied to a lot of other cuisines I have been acquainted with in the world. I look forward to using my friends as Guinea pigs. You know who you are.

This was my favorite one so far: scallops with onion compote in filo dough

Joining Allegorithmic

2018-02-16T01:32:00.002-08:00

Hank, meet Mat

Lots of people have been asking me what I’m doing now that I’m in Lyon.

Well, there are wonderful museums, famous cooking schools and language academies, monumental theatres, and a whole local firmament of Michelin starred restaurants. Rest assured, I’m making good use of them. The city is beautiful to walk and that’s a good thing, because I sure have a lot of food to burn off.

Perhaps less well known, is the fact that Allegorithmic, the makers of Substance, have an office here too. It's still a small but growing team, working in a few offices in a cute co-working space and incubator called MamaWorks, dangerously close to Les Halles de Lyon, by famed chef Paul Bocuse.

I have decided to join this team. I have been playing with Substance for years now, and for a long time I have been wanting to help it becoming a successful tool suite for us in the film industry. Finally, now that I am in the region, this is my chance, and I'm taking it.

If you're going, I hope to see you soon at FMX!

Nouvelles Aventures

2018-01-20T00:22:00.000-08:00

2018 is a brave new year.

Last August Mayah and I took a vacation to Europe. We stayed in France for a while, then continued to Italy, to meet with my family and join up with hers, and a few good friends as well, so we could happily drive a van around and explore my "Bel paese" (beautiful country).

My sweet lady and I must have liked it there, because we got engaged in Paris, and made beautiful memories around the rest of the France and Italy.

Once we got back, we gave it a lot of thought, and decided to look for a new adventure and try something a bit different, before we settle down a bit more.

I've been wanting to get closer to my family, but going to my hometown did not feel to us like enough of an adventure. We like learning languages, and we have a passion for cooking. That pulled us to this city called Lyon, the capital of traditional French cuisine, an international destination for its beauty and its wine country. Also, only four hours train from home - and not even on a fast rail - the closest I have ever been since I was in college.

What are we going to do in Lyon?

Well, first off, refresh our French! I studied it for eight years but to be fair I haven't practiced much for, well, all my life. And this romantic language is pretty new to Mayah too.

What else? Eat our way through Bouchons (small traditional French restaurants) of course! And bakeries! I want to learn to make croissants, bread, and boeuf en croute and all those delicious things. We'll take cooking classes.

We'll go to film festivals. Cannes and Annecy are around the corner. And fear not, we'll stay connected with our industries. I'll be at Siggraph and FMX although with a different badge (stay tuned about that).

After almost ten years, I have put some roots in San Francisco and Pixar. It is hard to say goodbye to my amazing friends - you know who you are - and my amazing job. We can't predict exactly what the future holds, and when exactly we'll come back, but we are going to live this adventure to the fullest. That's why 2018 is a brave new year.

Coffee and pastries in the Croix Rousse neighborhood, Lyon

In the meantime, come visit! We'll have pastries and coffee together.

Portable Material Descriptions

2017-12-04T09:16:00.000-08:00

Let me share my thoughts about a problem I've had to deal with for a few years, since before Finding Dory.

The creative vision behind Coco, paired with the time constraints we had, forced us to get creative on the technology. In a post I talked about how we wrote a new surfacing tool, called Flow. One of the immediate problems we had to address was the exchange of materials: how do we make sure the appearance in our interactive renderer would be compatible with our rendering on Katana and Renderman? Even more now after Coco, how do we round trip these material across these different suites?

Screenshot from the Coco VR experience.

In our industry, describing portable material descriptions is not a new problem. In fact it is many problems in one:

Vfx studios have to work on the same show, share props, characters or whole environments, but for technical, competitive or legal reasons, they cannot share the technology used to render them.
A high quality material needs to look fairly accurate in a fast preview, so animators can use it to make creative decisions in real time.
A studio may need to export assets so they can be adapted for video games, toy makers, or VR.
An asset that renders today, may not work in a future pipeline or renderer 10 years from now. There is a need for making the materials work not only across studios and tools, but also across time. Storing whole archives of assets this way is often referred to as "digital backlots" (at least ours is), and I'll talk more about them in a future post.

There have been several remarkable attempts at describing looks for exchange. Let's examine a few:

Alembic

Alembic is the most widespread pose caching (post-animation) format in the film industry. Sony Imageworks and ILM worked together on its development and its Open Source release, and as of version 1.1 they introduced lights and materials. The internal implementation includes a resolver of materials, so hierarchies can be described with inheritance and sparse overrides, which is a pretty powerful feature, similar to how Katana resolves Materials.

However, while the geometry is used all over the place, this rather stable format has seen few updates recently, and the industry adoption of its material description is extremely sparse. That is true even within some of the owners of Alembic, who moved on to newer technologies. In fact, major third party Alembic plugins don't even support the exchange of materials.

Pro:

Tiny dependencies
Widespread abc format
Stable library
Well compressed files with de-duplication
Open Source
Includes a resolver for bindings, hierarchies, overrides

Con:

Material description is hardly used
No enforced standardization

MaterialX

While Alembic theoretically can support material descriptions, studios have recognized a need for content standardization and compatibility. ILM took this to heart, with Doug Smythe and Jonathan Stone spearheading the development. The first conversations go way back to Siggraph 2012, and early embracers counted our tools-shading and USD teams at Pixar, who got to contribute to its shaping, and The Foundry. This resulted in the 2017 Open Source release of MaterialX.

At this point in time, Autodesk is the primary facility other than ILM that has put some real engineering muscle behind MaterialX, writing a pattern generator called ShaderX.

MaterialX exchange files are simply implemented as an XML, but its real strength is in its schemas. It is very rigorous in its pattern description, somewhat inspired to compositing systems and to the Mari internal node graph, with canonical nodes shipped with openGL and OSL implementation to clarify the behavior and maximize compatibility. It has well defined rules for targeting different platforms, and for describing complex color spaces. It supports node grouping and encapsulation, as well as some basic referencing. Because it only supports materials and no geometry, MaterialX features specific rules but no coded resolvers for procedural material bindings or references.

At ILM, MaterialX has been used on a few projects, starting with "Star Wars: The Force Awakens", and "Rogue One", where some of the shots were rendered fully in Unreal Engine, using MaterialX as a common description. The VR project "Trials on Tatooine" is also a great example.

As of today, a handful of other studios are testing MaterialX, but DCC's are working on robust implementations that will enable its spreading.

Pro:

Simple API
Small dependencies
Rigorous pattern standardization
Includes implementations
Self contained
Growing interest in the industry
Open Source

Con:

Separate from geometry
Some rules expressed but resolution is left to each client to implement
Bxdf standardization not fully flushed (yet)

Universal Scene Description

After releasing to Open Source in 2016, Pixar has been hard at work on USD's shading description. It comes off the bat with all the features of USD's core, such as references, variant sets, layering and composition, and a very optimized file format. While the Shading description is now getting more mature, lots of work is still going into Hydra (USD's render service protocol), so its output to each internally supported renderer (GL, Optix, or cpu/offline) will be accurate.

To some extent, USD looks like an Alembic on steroid, in the sense that it describes geometry, shading, dressing, lighting and animation in one composed layer stack. As such, it resolves references and layering, provides facilities to bind materials to geometry and schemas to resolve bindings and collections.

Working closely with ILM, my team greatly aligned itself with the concepts and language of MaterialX, ensuring compatibility and, in a near future, full support, meaning that you will be able to exchange MaterialX data directly in USD.

At Pixar and on Flow, we used USD of course. It has to fit in our pipeline which is almost all USD. Besides, we actually use a lot of the advanced feature sets of USD for Shading.

Flow native source data is USD itself, as opposed to having its own file format and then "export" to USD. It references pre-authored library materials, applies overrides and exports only the sparse overrides, as well whole new materials, shader bindings and public interface.

We can create derived Materials (similar to Katana's child materials) to any level of hierarchy depth. We can import and visualize alternate modeling variants, and export shading variants (currently in Katana, coming soon on Flow too), each with arbitrary complexity.

Finally, we export all this back to Katana, which in turn applies more sparse overrides, including collections and new materials, and exports back to another USD that will be composed on top. Assets shaded this way are then USD-referenced into shots during Layout or Set Dressing, ready for more shot-specific overrides. At some point soon, we'll be even able to render directly from USD to Renderman (r22 and later).

Did we use MaterialX too? I claim we did, at least in spirit. In the sense that MaterialX design has influenced USD shading a lot. What we don't use is the exhaustive list of nodes provided with MaterialX. They don't match closely with how we set up our networks, because we use less, larger nodes with multiple outputs, and MaterialX defaults to using many smaller nodes with a single outputs. You can see my older posts if you are curious about how we keep our renderers matching.

Other than Pixar, Animal Logic, Luma and SideFx have been actively using and developing USD and its plugins.

Pro:

Very powerful feature set
Composition resolved, clients don't need to implement it
Can read Alembic
MaterialX compatibility
Wide DCC support
Growing industry adoption
Great compression and fast file I/O
Open Source
Easy setup for Public Interface of networks

Con:

Moderate build dependencies, more if Hydra and python bindings are needed
No rigorous standardization of node types (unless using MaterialX schemas)

Multiverse

One of the advantages of the systems above is that they are open source. Because of that, some DCCs have been additionally relying on Multiverse for exchanging material data. Multiverse, distributed by Foundry, in itself is not a file format, so it relies on these existing ones. It can currently support, to varying degrees, USD, Abc and MatX, and provide plugins for loading and storing such data. We haven't tried it here, but I love the spirit of it.

Material Description Language

Independently from these efforts, NVidia came up with a different solution that spans their own and some external renderers: MDL, or Material Description Language.

This product is significantly different than the ones above in that it's not open source, and it's less of a node graph description, but rather a shading language (although one can often be mapped into the other).
MDL tries to be a standard representation for multiple targets, from real time engines, to interactive one (IRay), all the way to photoreal offline renderers, like Mental Ray or V-Ray. It is the responsibility of each engine to provide, full or partial implementations for each call of the material.

As the science of physically based illumination is settling over the years, MDL has put a lot of efforts in defining standards in its description, so Bxdfs and lights are enforced to be quite portable. Pattern description is simpler, and while more portable, it looks a bit like a more declarative version of the industry's favorite Open Shading Language (OSL) from Sony Imageworks.

With the exception of V-Ray and Substance from Allegorithmic, the penetration of MDL in the film industry is not clear to me at this point. I'd love to hear from you readers.

Pro:

Standard Bxdf's and lights
Large user base in other industries
Large corporate backing in developing and maintaining it

Con:

Not Open Source
Low film industry usage

What to do, what to use?

All the notes about pros and cons are, of course, quite personal, so this is my view. If you have read so far, chances are you may be looking for a solution for exchanging Materials Exchange. Unfortunately, there isn't one solution that fits all, and the specific needs of a studio should shape their choice of technology. I am partial to USD because of all the things we can do with it, but there are valid arguments for each one.

One consideration is these products have been developed for years and have had to solve problems that most people don't even know they will run into. Before moving ahead and writing a new one, make sure you tried these. And if you are missing features, well, many of them are open source, why not contribute?

Material Layering

2017-10-16T00:13:00.003-07:00

How do you shade a shiny metal object with a layer of dirt or rust on top (or underneath)? Every application seems to have an opinion on this. The problem they solve is called Material Layering, and while it seems simple as a concept, it has quite a few catches, and quite a few solutions.

Coco packs an incredible level of detail in each sequence. Each material is made of many layers.

As physically based approaches are now pervasive, material layering has become again a topic of development and efforts are going into trying to standardize its description and its desired look.

I have been meaning to write about this for a while, as a lot of questions come up, from how to flatten a material to make it more efficient to render, to why blending between shiny and rough items doesn't always look right. I'll try to cover some of the answers that have I have seen in offline renders as well as some authoring tools.

Color Layering

If we go back far enough, Renderman had Reyes, a renderer that lets you literally ask for lighting at any point in your RSL (Renderman Shading Language) shader.

A popular way to calculate Material Layering was to get the result of each material, complete with its illumination, turn it into an RGBA layer, or ColorOpacity, and do an alpha composition. We also passed along just enough information to do displacement layering. The structure was called ColorDisp (1 color, 1 float, 1 int, 2 vectors). Simple enough, this approach was quite good at getting realistic results. As we started being able to package and publish material layers for later use, the average number of layers grew over time, and we ran into this technique's main drawback: the cost in computing complex, complete lighting at each material layer. We didn't have physically based rendering or importance sampling yet, but in fact, even after we introduced physically based rendering in Monsters University, we still relied on this quite a bit. So 5 or 6 layers meant calculating all the lights 5 or 6 times, which quickly became a bottleneck.

Pattern Layering

Starting in Cars in 2003, we experimented with blending not the results of each layer but rather their inputs. Albedo got layered with albedo, specular color with specular color. The final composed parameters were used to calculate illumination lobes, and a final color.

Not all materials use the same illumination system or the same parameters, so this was not a complete solution. If you grouped your layers carefully, with this technique you could get good results and reduce, say, 8 layers into only 2 or 3 pre-composed materials with unique illumination properties.

Pixar Cars, 2006, was rendered with a mix of ColorDisp and pattern layering techniques.

If you use this blending for a some or all of your layers, with a bit of hand holding, results still look good with a much reduced overhead. So after we switched to Physically based Rendering, with improvements on Monsters University and Cars 2, this became the main layering technique used at Pixar.
Used carelessly however, the technique is far from perfect: not all parameters blend equally well. If you blend directly related parameters separately, across two materials, you will get badly blended pixels around the layering zone (where presence is not solid 0 or 1). Why is that? Say you have a grey, RGB = (.4, .4, .4) material with .5 diffuseGain. Its overall diffuse response is (.2, .2, .2). Now let's say you mix it with a black material: (0,0,0) and 0 diffuseGain. If the presence of the black overlay material was exactly half, or 0.5, you will end up with a color of (.2, .2, .2) and a diffuseGain of .25. The resulting color is... RGB (.05, .05, .05)! While the accurate average of the two should have looked more like a (.1, .1, .1), double that.

That is because we are blending separately two aspects that are always considered, in the renderer, as one. We use the result of diffuseColor * diffuseGain, so that is what we should be blending: premultiplied results. That goes for any color that also has a multiplier applied to it.

This same premultiplication gets trickier when you are calculating factors that affect non-linearly the resulting pixels, like a dmfp (diffuse mean free path) in subsurface scattering.

Surface normals present their own challenges. We have historically preferred to layer bump vectors (the equivalent of displacement vectors, they create a virtual point which we later used to compute a final normal). Bump Vectors have the advantage of blending more correctly than surface normals, and allow for more complex layering behaviors, such as displacement/bump accumulation, erosion and softening. Unfortunately, Bump Vectors rendered much faster in good old Reyes, where the derivatives needed for normal calculations were basically free. The extra cost in a path tracer is measurable.
Recently, the development focus is going more towards pre-computed maps we call BumpRoughness, based on LEADR mapping. While they share the same issues as layering normal maps, they are more accurate, faster to render, and they behave better when the shader details get filtered away.

An interesting parameter to interpolate is specular roughness. For at least two reasons. One: in most classical descriptions of physically based microfacet models such as Beckmann or Ggx, roughness is not a linear quantity. That means that double the roughness will not give you a highlight that is double as large. Most valid values for roughness tend to be crammed between 0 and 0.2. Some shading models, such as Disney's Brdf (PxrDisney in Renderman) address this by blending a normalized and linearized parameter instead, that behaves well between 0 and 1.

But that's not all. Quite simply put, the average of the render of two speculars is not, in fact, the render of the average of the same two speculars. This becomes quite obvious when you have a lot of variance over the roughness (e.g. a shiny metal and a dull layer of dust), with a not-quite-solid coverage (e.g. presence of the dust is 50%). The artifacts you get are rougher surfaces with little detail, and even some visible errors in the blending areas.

On top of this RenderMan preview shaderBall, mixing roughness will make the underlying shiny layer look rougher, rather than dimmer with a rough layer on top. Note the roughness artifacts on the highlights of the base too.

Mixing resulting values from the speculars directly, will yield a result that is closer to reality.

We work around the roughness blending errors by having more than one specular lobe. We have three, sorted by their recommended roughness range. ClearCoat (0-0.1), Specular (0.1-0.3), RoughSpecular (0.3-1). These three allow us to reach interesting, rich sheen looks. But especially when layering, they ensure that the blend between speculars does not create interpolated roughness that are too far from either material being layered.

Rust shader by Harsh Agrawal, rendered in Renderman

Without getting into more exotic surface effects such as metal flakes, scratches, fuzz or iridescence, layering these three lobes separately in practice has solved 95% of our visible layering issues, giving us accurate results.

Cars 3's Storm, closeup of metal flakes

At Pixar we still layer patterns, compose the results in a virtual structure of premultiplied values (V-Struct) and feed the resulting values directly to PxrSurface. We are not alone in this: many studios do their look development in procedural and/or painting applications where a lot the visual complexity is baked and at the end, pre-layered values are fed to an individual uber-shader with a largely predetermined number of illumination lobes (or Bxdf's)

Bxdf Layering

In a perfect path tracing world, with accurate prediction of where most of the light will be coming from, layering BXDF results directly is not as much more expensive than pattern layering any more. The layering logic needs to know what is covering what and how, affecting the Probability Density Functions (PDF) and their resulting samples.

We did try this approach during Finding Dory, but did not publish it with Renderman (do not confuse with PxrLayerSurface, which relies on pattern layering). While more work was needed to do it really efficiently, there are advantages to it: blending is more accurate, and complex pipeline solutions like VStructs become unnecessary. Also, more lean and diverse Bxdf become feasible with less reliance on one UberShader to rule them all. It will be exciting to see what happens in this field in the coming years.

Illumination lobe based layering

Bxdf, in an UberShader world, are composed of multiple illumination lobes. Seemingly simple ones, like diffuse or specular, as well as complex ones, like Path Traced Subsurface Scattering. Often times the resulting light coming out of these lobes is simply added together into a final color.

In the real world, however, these illumination properties often come from distinct layers of the physical material. For example, in a car paint, there will be a metal layer, a paint layer with metal flakes embedded, and a clear coat on top of it all. The more light the top lobe reflects, the less light the underlying layers will receive, and therefore emit. Also, the top layer might modify the light properties and diffuse it in a way that will affect what happens to the lower layers. This can get complicated quickly, beyond what pattern layering can do.

An easy way to begin emulating this, is using the Fresnel coefficient that affects your specular layers, inverting it, and applying it to all the other lobes.

E.g. If the light is largely reflected by the specular clear coat at grazing angles, we should dim appropriately the response of the other layers. That is so the overall response doesn't blow out beyond physically valid coefficients, which is bad if you care about your render times and quality. If you have more than one specular lobe, you may have to do this multiple times.

There are a few good papers on this topic, like this one in 2014 by Jakob, D'Eon, Jakob and of course, Steve Marschner. While the science is good, I haven't seen a full featured implementation in commercial renderers.

This is something that will be fun to see evolve in these years, as well as how standard descriptions of layered materials will take shape, in Usd, MaterialX, and MDL, to name a few.

Kicking it underwater

2017-09-30T17:09:00.001-07:00

In almost a decade living on the Pacific coast, Hawaii is a place I have only dreamed of. Especially from Italy, it is seen as the quintessential exotic place to visit. So beautiful and so remote. And yet so easy to get to now.

Finally we resolved to go visit the Big Island in February. As it turns out, it a great month to visit Hawaii. The weather is mild, the surf is great.

But before going, we prepared by taking scuba lessons. So much of the beauty of the islands lays hidden under a feet of salty water, and I was not going to let that escape. Not after all the work I put in my ShaderToy project from a couple of years back.

The Big Island has a lot to offer. Several volcanoes, that you couldn't tell from the shore, but they are almost 14.000 feet in height. Almost as high as the tallest mountain in the Alps, the Mont Blanc.

We ended up settling in Kona for a few days, and drive around from there. The place is notorious for the coffee plantations on the volcano side, also the only place in the US that has the right climate to grow it.

Hawaii has their own endemic nuts: the Macadamias - also one of the tastiest and fattiest nuts in the world. The ideal treat to bring back to the continent and nibble on for weeks.

The sharp, porous volcanic rock turned, after millennia, into beautiful black sand beaches. Those become the theater for the birth of sea turtles, who travel to the Big Island to lay their eggs.

Shading at Siggraph 2017

2017-08-01T16:33:00.003-07:00

This summer downtown Los Angeles got hotter, with awesome progress being presented at Siggraph left and right. Here are some oh my highlights in the world of shading.

MaterialX goes Open-Source

Years ago, I remember sitting down in the lounge of a hotel in Anaheim. Sitting with me were friends from Industrial Light and Magic, The Foundry, and several film and software studios. We discussed how hard it is to share material descriptions across software and studios, and have them look right, without having to hand off all the secret renderer sauce.

ILM really took the helm in this effort, and came up, year by year, with a better and more refined spec, called MaterialX, for a standard in pattern descriptions. Used alongside Alembic, MaterialX helped them share assets with vendors, and across OpenGL based software such as Mari, and final render quality. And now, the spec is mature enough to go open source. Congratulations!

Back at Pixar, I have looked at this spec quite a bit, and worked with our friends at ILM to try to improve it and make it converge with our Universal Scene Description (USD) shading libraries, which in turn were heavily influenced by MaterialX. As a result, it's easy to see a near future where MaterialX will be natively encoded in USD.

I will get deeper into this in a separate post, but a somewhat similar goal was set by NVidia's Material Description Language (MDL). NVidia has put some real muscle behind this project, which is already established in other industries, such as product design, automotive, architecture and even fashion. MaterialX differs from MDL in its focus on film-centric problems, such as flexible deferred binding and color spaces (which can become a problem when handling different hardware and cameras). It is also very focused on pattern network definition, while MDL has successfully captured the physically accurate illumination side of the equation.

Flow Graduates

While Coco is still in its final stages, our new single-asset shading tool Flow (see a couple of posts ago) is getting more mature, and is getting used in both characters and sets in newer shows.

Come see my friends Florian Hecht and Peter Roe present the fruits of our labor at the Nvidia booth on Wednesday at 10am. Or at the Renderman Science Fair, if have an invitation.
[update] Here is the link to the talk at the NVidia booth.

A lot of Flow's success in production can be credited to the speed and accuracy of our Optix-based preview renderer, called RTP. This prototype has shown a lot of potential and we hope it will prove useful to our team in Seattle, to continue to push Renderman speed.

We also focused on streamlining the most common operations, leaving the more complex things to Katana. As a result, accurately shading assets can take hours, where it used to take days.

But really, Flow and its level of integration across our complex pipeline wouldn't have been possible if we didn't have a such a powerful scene description as USD.

Renderman Advancements

Renderman's core tech advances are also nothing to laugh at either. Much faster curve rendering, which can make a difference where everything is furry, hairy or even just a bit fuzzy. Newer, better subsurface and denoising techniques are available, in use already at Pixar. FxGuide just published a great article with a deep dive in path tracing and subsurface, featuring our work, especially that or Christophe and Ryusuke.

Cars 3 Production Sessions

Top research and academic papers are what have historically distinguished Siggraph from other events, but Production Sessions and panels are still some of my favorite things to watch. This year we are presenting Cars 3 in all its beauty.

My friends from technology, art, characters, sets and lighting are showing some of the remarkable work that went into it - including the deferred texturing techniques mentioned earlier. Scratches and bump/roughness interactions are not new concepts in rendering, but their quality leaped ahead with new authoring techniques.

Smash and Grab

Pixar also has an experimental department that tries new things, not for theatrical consumption, called Spark. This year the group, only a handful of part time artists and TDs with no Pixar pipeline set up, put together a beautiful little short, called Smash and Grab. There is a talk about it on Wednesday at 9am. Hope you can make it!

http://s2017.siggraph.org/talks/sessions/alt-workflows

and much more...

Trying to cover everything noteworthy that happened this year is well beyond the scope of this post. But I did see some exciting news about Epic's new DataSmith Alembic-based toolkit, as well as Allegorithmic's impressive new Substance/Unreal integration.

Faster workflows, standard exchange and simpler controls seem to be the focus of this year. The near future of shading is getting brighter, and that makes me smile.

Cars 3 and deferred texturing

2017-05-29T12:59:00.000-07:00

I haven's spoken much about Cars 3 in the last few years. This time I will, because I think the film deserves attention, as it looks incredible, and it forced us to solve a lot of interesting problems. One of them is deferred texturing.

I am going to go back to the initial issue in Finding Dory, the show when we switched renderer to RIS. That gave us fantastic new potential for render quality, but brought with it the end of our classic Renderman Shading Language, along with all our traditional shading tools, including one, we called Picasso.

Using math figured out and later patented by my friend David Ryu in 2008, Picasso allowed a user in Maya to build a set of poses, e.g. T-pose, exploded prims, or open mouth, so hard-to-reach parts could be painted with ease. Each pose could be rigid, as in transform matrices would be the only data stored per pose, or deformable, so an additional 3d primvar was stored with each point.
Each pose had an associated rig of cameras, and each camera (usually orthographic) was able to project paint on a subset of objects, on one or several "channels", labeled arbitrarily or using specific standards, such as Albedo, ShinyDull, SmallDisp, etc... Our shading networks would have strategically placed special "receiver" nodes with matching channel names.
The order of cameras would determine the composition order of the paint, that would alpha compose like in photoshop layers (in "normal" mode). The math is pretty specific about how much of each layer to see based on camera depth, clipping planes, incident normals and a number of parameters. At render time, when the objects deformed, the paint and its layering was calculated using reference points and normals, so the paint wouldn't "swim".
The matching of receiver nodes with projection paint was resolved dynamically at render time.

Now, was this the most efficient possible way to texture an asset? Probably not. Certainly not at render time. But in the Reyes algorithm, the relative weight of this texturing was pretty minor, and it allowed to do complex texturing for assets that are traditionally hard to UV. Perhaps also because of Picasso, Pixar has always been resistant to adopt UVs.

The origin of the name is no longer clear, but I think that Picasso should draw its name from the Spanish painter who, for some time in his life, adopted cubist techniques, where paint on the surface appeared to be coming from projections other than the point of view itself.

After being built on hierarchical Renderman attributes, the later incarnations of Picasso I wrote on Monsters University, progressively relied more on dynamically generated, or code-gen'ed, RSL co-shaders that looked at the camera setups and layered the paint appropriately.

Why did we have to lose all this? It's not just about the disappearance of RSL. In Pixar's RIS, we are running shaders two orders of magnitude more times for a final render than we did for Reyes. The whole concept of co-shaders (hierarchically bound objects with an API that could be fetched dyamically by name) went away to help improve the new Renderman's performance. While those changes were needed, they had serious consequences on our traditional workflows: if we want to apply a single layer of paint on top of 100 object with 30 different bound shaders, we would have to go into each one of those objects and splice some texture and projection nodes into their shading networks, and hope we don't have to change our minds or make mistakes, or else do it all again.

Fortunately, we didn't only switch to RIS, we also switched to Foundry's Katana, away from our traditional, declarative lighting suite called Lumos. In Katana, deferred but inspectable proceduralism is a solved problem, and at the core of the tool. In our shading networks we started adding nodes, named "Receivers", which only have a channel name (to tune in with a paint broadcaster) and a default RGBA, for when no paint is found at all, or alpha holes are found in the paint.

I wrote a Katana "SuperTool" that looks somewhat analogous to the built-in "Gaffer". That is no accident, as we had modified Katana's Gaffer quite heavily for our Blue Umbrella short (by Saschka Unseld), and had gained a lot of insight on its workings. Instead of creating lights, this supertool creates cameras and settings, and based on that set up, at render (or rib generation) time, generates and wires in the appropriate textures, projections, and blending nodes, purely based on the channel name and their composition order and style.

The nice thing about this is that the authored shaders didn't have to be setup knowing what paint would be available in the future, that part would be figured out later, even on a shot per shot basis. Instead of a texture node, we just put in a Receiver with channel "GraphicMask", or "MetalColor", or "Displacement". This can be extra important for Cars movies, because we use a ton of graphics, which are often finalized and approved long after the assets are shaded and painted. So there, deferred projections solved.

In fact, why even stop at projection painting? Why not make it be able to read paint no matter where it came from? That's why we picked a multi-talented scientist and artist to name this new tool: Leonardo.

Since the introduction of Foundry's Mari in our pipeline on Monsters University, UDIM-based painting has taken a growing role in our workflow, especially in characters. This is notably later than a large part of the VFX community, and that is in part a consequence of how much Pixar cherishes the ability to make creative choices live, late in the pipeline, while uv painting sort of bakes it in. Leonardo allows us to switch techniques at any point, to cater for that need. So we added support for UVs, UDIMs textures. We extended it with the support to Disney's uvless texturing, called Ptex, which came in handy with complex meshes that are typically difficult to correctly UV.

We didn't end up stopping there. By the time the show was wrapping, we were able to read point clouds, primitive variables (or privars), openVDB and even lights and interactive space-shapers in Katana, which we call Rods. That came in very useful when we had to drive complex, shared paint such as animated streaks of dirt and damage, shared between characters and environments. Eventually, we created a "Broadcaster" setup that allowed deferred splicing of arbitrary networks with matching channel names, for those special cases that were not covered by our built-in Leonardo layers.

I hope you get to see Cars 3. A lot of hard work and love went into it, and what came out is a surprisingly involving emotional piece, as well as a real eye candy.

Coco and the Flow

2017-03-15T22:45:00.000-07:00

So what exactly have I been doing Coco? Not much I can specifically point at. I mean, our work is everywhere. That's the thing about working in Tools: we enable others to do amazing work, and that is incredibly rewarding.

Among the other things, my fantastic team and I have embarked on a mission to make shading fun again, working really hard on a shading tool, which we now imaginatively call Flow.

Last year at Siggraph, Pixar was hosted at the NVidia booth, and Danny Nahmias and I gave a preview of what was coming. Check it out! Other videos at the booth are hosted here. We also presented it again at Pixar's Renderman Technology Fair.

When we made Finding Dory and Cars 3, we pushed Katana for Shading as much as we could, given time and resources. The challenge was great, because we really started from scratch: we switched to the new Renderman RIS renderer, losing with it our well seasoned Renderman Shading Language and our existing tools. The results were great too, because we were able to create new workflows that were impossible before.

Not all was perfect though. With the risk of oversimplifying things, the iteration of look development is often as simple as bind a shader, move a slider, see the result, and keep tweaking until it looks good. With our software, hard things became possible, but these simple things took a long time. In an effort to address this, the Coco environment artists and Tools teamed up to build something different.

Flow puts together some of our best technologies. It uses a real time GPU viewport, that is a full path tracer developed at Pixar, built on NVidia's Optix framework. The application framework is using parts of our very own Presto animation system. Our shaders are kept in sync with the Renderman ones via a OSL to RTP (cuda) translator. Kees - you might know him for being the author of ShaderFX - built a shading specific node graph that enables some of the coolest workflows. We have a cross platform media browser for finding and publishing textures and materials. And last but not least, Flow is the first tool to load and save USD (including USD Shading) as a native format.

There are a lot of people to credit for this work. But the biggest shout outs go to Peter, Brandon and Florian, who fearlessly carried this project so far.

Would you like to see it? We'll be presenting it again this year at Siggraph, probably both with NVidia and RenderMan. Once again, stay tuned!

Coco is coming

2017-03-15T22:04:00.002-07:00

Wonder what I have been busy on all this time? Given my absence, you could say I'm back from the dead, after being head down on a movie about, well, the Day of the Dead.

Remember Disney tried to trademark "The Day of the Dead" as a movie title? That didn't go too well. Anyways, the film was eventually renamed to Coco, and the trailers are out! Here is one, still gives me shivers.

Eighteen years later

2015-09-23T16:40:00.000-07:00

My very first trip overseas by myself was a student exchange program, in a cute town on the English Channel, called Bournemouth.

I have fond memories of that place. Many first times. Other than the obvious first exposure to real spoken English language, I got to learn a lot about the world just by seeing something different than what I was used to. It was my first time clubbing, my first time experiencing a cold beach, a sunrise at 5am, and my first time trying several interesting (for an Italian kid) foods and beverages.

Novel delicacies that were frankly hits and misses with me included fruit-flavored chocolate, pasta with ketchup, lemon curd, vinegar fries, honey glazed salted popcorns on the pier, but also hard cider, and British ales. It was most of all my first time making international, longtime friends. I would bet the experience had a lot of repercussions in my life although, not having a time machine handy, I will never know.

So here I am, eighteen years later, on my way back to that cute English town. And for work, none the less. My old friend and ex colleague Sofronis invited me to join, so I'll be talking about what's new at Pixar. I'll be part of a panel and introduce our Universal Scene Description at BFXPro, and talk about our experience changing from a Reyes Renderman to a Path Tracer Renderman, at BFXCore next week.

For those who may not be familiar with it, BFX is a fairly new conference and festival. It takes place in Bournemouth, UK and, while hosted by the Arts University of Bournemouth, it has many industry partners such as Dneg, Framestore, MPC, The Mill, Cinesite, Creative Skillset and BAFTA.

I will also be hanging out in London a day or two. If you're around, come say hi!

A Scene Description for All

2015-08-31T20:09:00.000-07:00

I can't say I went to Siggraph this year, although I was physically there for one evening.

I went to the Renderman event, which this year was extra exciting, not just for the crowd-drawing yearly teapot giveaway, but because we turned it into a full on science fair. Besides the presentations, which this year were including production updates on Finding Dory, we had booths showing some of the cool work we have done in the last year.

I got to help man one of the booths, mainly to show off some of the integration work we did on Katana and Rfk (Renderman for Katana), so shading would be possible on the - mostly - lighting platform. Other booths included our Real Time Preview, an Optix based renderer that can be used for high quality previews of your shading and lighting. We had Presto, our in house Rigging-Animation-Layout-Sim-You Name It tool. We also showed Hydra, our GL renderer used across multiple platforms.

Among the other things however, we announced the open sourcing of our new scene description format: USD - or Universal Scene Description. I am very proud of it and I hope it becomes a new standard in the data exchange for Visual Effects and Animation.

So besides it rather grandiose name, what does it offer that is not already out there?

First off, Usd is not just an optimized pose cache format: if that's all you need, Alembic is plenty good enough - by the way, Usd can load Alembic files. Usd attempts rather to solve more complex pipeline issues. At Pixar at one point we had a dozen files and respective converters we had to keep up to date. That made us reconsider whether we should simplify the pipeline and adopt one exchange format instead.

Our pipeline is rather complex, many departments work on the same data at the same time, so we had a system of hooks and cues, and in general files that expressed opinions on the same objects, even the same properties, and those opinions had to be layered correctly so that the strongest one would be available to use for rendering.

Hence the first main feature of USD: Composition. Take a character. Then add some rigging information, then load it into a group, where you can add garments and hair. Add some shading. Then load that group into another context, like a shot. On the shot you have layout, then animation, then lighting and FX. But the shot also loads another version of the same character as well as hundreds of thousands of props and architecture parts. Each one of those elements are Usd files, that reference each other and override or enrich the previous layer's results. Better yet, you can load the Usd of the whole shot, and express overrides that will be saved directly onto a low level, like the model itself.

But what if the character is a template for, say 10 different characters, each of them with 15 different sets of characteristics? Usd can address that with a built in mechanism, called Variant Sets. We use them for many things, starting from modeling variants (which can be switched as late as shot lighting), shading variants, dressing variants, etc... Any set of overrides can be encapsulated into a Variant Set, and given a name. Those can be switched at any time, which made our shading workflows a lot more flexible than, say straight Katana Look Files.

There is a lot more to it, such as schema API's and viewers and editing tools, but hopefully, you get the idea. This has been a game changer for us, and we hope it will be for you too. Stay tuned for updates!

The last Reyes project

2015-07-21T21:30:00.000-07:00

In my engineering position in Tools at Pixar, I am fortunate to get to work on pretty much all of Pixar theatrical projects. My department tends to start as early as four years from delivery, and we help along the way until one year or so to the release date. I was actually done with my work on The Good Dinosaur a long time ago and seeing the trailer almost caught me by surprise, a great one, as I hadn't seen recent imagery in months.

This show marks a special milestone at Pixar, though, for us who work on the technical aspects of our films. It is the last show we will release with our Reyes implementation of Renderman. It still uses a lot of the work we did on Monsters University, where we implemented a physically based engine using a hybrid raytracer on top of Renderman's Reyes. The quality of the environments in the huge world portrayed in The Good Dinosaur, however, raised the bar again and is a true achievement in itself.

Here is the latest trailer. I think it is a true beauty, and shows how far were able to push photorealism before taking a leap into the new renderer: RIS, Renderman's path tracer.

In the last couple of years, I had a chance to give some talks around the world about the differences between Reyes, Raytracing and Path Tracing, and how we use them, and have learned to appreciate their strengths and weaknesses. Now we are still hard at work with making the new renderer work well for our artists on a platform like Katana.

So why did we change? I mean, other than the fact that most of the VFX world is working with path tracers by now - be it Manuka, Glimpse, Arnold or you name it.

While the selling point for us was the quality of water renders - they tend to matter when making a sequel to Finding Nemo - the main appeal to many of us was the speed we can get feedback of complex illumination phenomenons. It was a bit later though that I started seeing the real difference in the quality of the renders: in the same render times we got so much more detail that my eyes hurt. While in Reyes we were shading 4 to 10 shading samples per pixels, we are now doing thousands. All the pain we went through in the transition was quite frankly worth it.

The challenges I have seen in the new architecture is that we have lost our classic bag of tricks, and had to dive in with a fresh mind, but not so much time to do so. Things were weird and unintuitive. Bump (at least the way we formulate it) is now much more expensive than displacement. The cost of primvar is, comparatively, much higher. We lost a whole shading language: RSL, as well as some good old tools we were used to. More importantly, we lost the flexibility that dynamic networks of coshaders and lights provided us, and have to make up for it with a whole new toolset to connect bxdf's and patterns in complex ways.

We are now ramping up the following two shows to use the new toolset, and it is just now starting to look more appealing than daunting. We're over the hump.

Two in a year

2015-06-02T09:33:00.002-07:00

For the first time in Pixar history, 2015 is the year we have two movies released. The next one is The Good Dinosaur. The exciting news is, we have the first official trailer out there. It's pretty darn exciting, so enjoy!

The film has been anticipated by many posters, such as this one. If you pay attention though, the main characters and the environments actually appear quite a bit different than the look Pixar decided to land with. On the technological side, I tested a lot of my work on on the older assets, and I feel a bit nostalgic, but the new characters are much more appealing.

The Good Dinosaur is slated for later this year, and we are in the final laps of this race. Cross fingers for us!