Direct3D9: (ERROR) :All user created D3DPOOL_DEFAULT surfaces must be freed before ResetEx can succeed. ResetEx Fails.
Direct3D9: (ERROR) :ResetEx failed and ResetEx/TestCooperativeLevel/Release are the only legal APIs to be called subsequently

One additional thing that was throwing me was this error message:

Direct3D9: :Window has been subclassed; cannot restore!

This can probably safely be ignored, and may be a symptom of the back buffer leaking specifically, but generally this whole error set is a sign of leaking D3D objects.

If this is something innocuous, like a single texture, you can sometimes figure it out by simply enabling the DirectX debug libraries, running and exiting your application, which will give you the AllocId of the leaking asset, which you can then break on. However, a large number of leaks indicate that something lower level may be leaking, and potentially a leaking reference to something, like the back buffer (which was our case). How to find it?

Here's what I ended up having to do.

First we did what we need to do for every bug, reduce it to the fewest steps possible.

Next, run PIX and take a full capture, until you've gotten to the point of the leak, then close. If you have a lot of steps to reproduce, you'll want to make sure you've got a lot of hard drive space, and some time on your hands, because it can take a while.

Once you have your full PIX run you can look at this column:

And set it to Never. This will show you every D3D object that's leaking, regardless of how many AllocIds it has (interesting tidbit, a single D3D object can allocate multiple AllocIds). If you move to the last frame (and wait for PIX to catch up, which, again, can take a while) you can even see how many references exist during shutdown. You could even walk each frame and see when the reference counts of your leaking assets go up.

In our case, like I said, we were leaking a reference to the back buffer. Most obvious was to look for calls to IDirect3DDevice9::GetBackBuffer, but none of those were leaking. Less obvious was to look for cases of IDirect3DDevice9::GetDepthStencilSurface and IDirect3DDevice9::GetRenterTarget(0). In our case, a call to GetRenderTarget was leaking as a result of a short circuit in a logic check, resulting in the back buffer reference to dangle in a very specific logic case.

Granted, this leaking does not pose a lost device recovery problem with DirectX 10 or DirectX 11, but since many engines are still DX9 based, I thought I would share.

There are two very weird issues when debugging embedded Mono. First, it currently works exactly opposite of how most debuggers work. In most soft debuggers, the application starts the server and the debugger connects to it. In Mono soft debugging, the debugger creates the server and the client connects to it. Second, the Mono soft debugger uses signals / exceptions to work halt threads. This means that you can't debug both with Visual Studio and Mono Develop, because Visual Studio will pull the signals before they get to the application. This also prevents

However, there are changes in the works from the Unity team which fix both of these issues. However, they are not currently part of a stable mono release (or in the Elijah's version for Native Client). Once they are, I may revisit this whole article, but that's where we are right now.

So, with all of that in mind, how do you get this working right now?

Set Up MonoDevelop

First up, grab MonoDevelop.

Second up, you will want to add an environment variable to your system called MONODEVELOP_SDB_TEST and set it to 1. This gives you access to the following menu item when you have a solution loaded:

Which brings up this dialog:

Now, for some reason, I've never been able to get MonoDevelop to launch the application directly. It always fails attempting to redirect standard out, and then immediately crashes (I could look at the code to figure out why this is, but I haven't had a chance to). We'll come back to this dialog in a minute.

Prepare Your Code

There are a few things you need to add to your code to get debugging to work. The first few lines can occur at initialization, regardless of whether you're debugging or not:

mono_debug_init(MONO_DEBUG_FORMAT_MONO);
mono_debug_domain_create(mMonoDomain);

These should be done before you load any assemblies.

In addition, if you're running Mono from a background thread (like we are), and not running it on your foreground thread, after initialization you need to detach the foreground thread. This is because when Mono attempts to halt all threads when it hits a breakpoint, it can only do so when you're executing Mono code. If your main thread never executes any Mono code, it never halts, which means debugging doesn't work. This is good practice regardless, because if the GC needs to halt all of your threads, it will also fail if a non-Mono executing thread is still attached.

Last thing you'll need to do in code is set up the debugger. Here's the code:

     bool isDebugging = AresApplication::Instance()->IsDebuggingMono();
     const char* options[] = {
          "--debugger-agent=transport=dt_socket,address=127.0.0.1:10000"
     };
     mono_jit_parse_options(1, (char**)options);

This needs to be combined with any other options you may have (I only have one other, which is –soft-breakpoints), and should only be done if you want to start debugging. If the Mono client can't connect to the debugging server when it starts up, it will call exit, which is no good.

Lastly, if you're using the Microsoft C# compiler, instead of the Mono one, you'll need to generate mdbs from all of your pdbs. There's thankfully a utility for this. We actually now perform this step as a post build event, which keeps the mdbs up to date with latest code.

Put it all together

Alright, so let's get this party started!

Open MonoDevelop. Set some breakpoints, then open this dialog again.

Don't change anything, and click "Listen".

Now start your application outside of Visual Studio with whatever command line arguments you chose to signal to your app you want to debug in Mono.

Everything should now connect and you should be debugging Mono in your app!

Let me know if you have any issues!

I'm back from GDC 2012, which is my 11^th GDC straight, and I'm almost fully recovered. This year was certainly better than the last two for sure, but it doesn't take much to top getting minor food poisoning and getting (almost) deathly ill (I still can't top Darius who, three years ago, entirely lost his voice). I still have yet to find that balance of parties and not-parties, and when to just go hang out at the various hotel bars. I think I did better this year, but I still don't have it quite down.

This year, the high points were definitely the two events I ran. The first was the awesome Pre-GDC board game bash, which hosted probably about 130 people this year. This is always a lot of fun, and way more low-key than most other GDC parties which is why I like it. We're growing again next year, probably to about 150+ people, but I'm also looking for a bigger room with more tables (table space was certainly a problem this year).

In addition to that awesome event, this year I organized a private scotch tasting. This was really really awesome. So awesome, I don't have pictures of it. It was another great, low key, and educational event. Everyone there was really awesome, and I hope everyone had a great time. We'll see if that will happen next year, and if it will grow or not.

So what's up for next year? First, the board game night will certainly be running again. I've already started putting out requests for space the Sunday before GDC (and already been turned down once o_O!). Other events? Maybe the scotch tasting, and potentially a Black Tie Only reception. These last two require a good amount of planning and funding, so they'll probably be maybes until closer to GDC.

What else? I'll be posting more! Fire Hose announced that its next game (and the game I'm working on) Go Home Dinosaurs, will be shipping on Chrome Web Store using Native Client. I haven't been able to talk about this too much until now, but now that I can I'll be posting a little bit more on my experiences with Native Client and with Mono (which is what we're using for scripting). So keep an eye out for that!

This jam was way less stressful than any other jam I've participated in. We essentially had something fun to play by middle of day 2, and just polished things throughout the end of day 2 and day 3. As a result, this is probably the most complete jam game I've ever worked on. Very little went wrong. It was great.

So what when right?

1. AngelXNA – Power of Angel combined with the power of XNA! This was the third time I've used Angel in a jam and every time it's improved. We were able to get levels in games quickly, spawn actors, and do all sorts of things quickly and easily. It actually worked really well. Again.
2. Abandoning Design – The original point of the game was to have a cyclic back and forth of helping and hurting the other player. On your turn, you would have your goal, and a directive to help or hurt your opponent. On the next turn, you would be given a bonus based on how well or poorly your opponent did. At some point, we realized there was never any reason to help your opponent. It just wasn't worth it. So we abandoned the helping, turn based component, and went with real time versus. This makes it questionable whether we obeyed the theme, but whatever. The game was much better as a result.

What went wrong?

1. AngelXNA? – So, the one thing I don't like about Angel and XNA is that it does have a huge barrier to entry for people playing the game after the Jam. I know I don't bother playing a GGJ game that requires a download unless it comes VERY highly recommended. I will, however, look at pretty much every HTML5 or Flash game. That said, Angel is awesome, so maybe I have to figure out a way to fix this. Native Client maybe?

That's it. Not too many lessons this time out because everything went so well. It's a weird feeling. Really looking forward to next year!

To that end, I've added two demo screens showcasing two features that are in AngelXNA that aren't in Angel. The first is the Editor. While I had a problem with the Editor in the last GGJ, Fire Hose used it a lot for prototyping and I think we've worked out most of the kinks. It's not perfect, but it does make creating levels in Angel super easy. There are shortcuts for cloning, scaling, and rotating actors, which makes the level editing process super quick.

The second feature has several aspects, but comes down to the ability to use sprite sheets and the ability to create named, data driven animations. There are two parts to this. The first part is the Tiling system. By adding a .tileinfo file parallel to any given image file, it will automatically start rendering Actors that use that image as tiled. Here's an example .tileinfo:

IsAnim = true
FrameSize = Vector2(160, 200)
AnimDirection = TiledAnimDirection.Horizontal
StartFrame = Vector2(0, 0)
EndFrame = Vector2(0, 2)

It specifies that this file contains an animation, that each frame is 160x200, that the animation frames run left to right, starts at 0,0, and ends at 0,2. You can also use a non-anim and specify a specific tile to render.

Once you have these .tileinfos specified, you can specify animation files. Here's the example anim file:

BeginAnimation("Idle", TiledAnimRenderPath.Create())
    AnimDelay = 0.03
    File = "images/devo/devo_idle_strip"
    AnimType = AnimationType.Loop
EndAnimation()

BeginAnimation("Run", TiledAnimRenderPath.Create())
    AnimDelay = 0.03
    File = "images/devo/devo_run_strip"
    AnimType = AnimationType.Loop
EndAnimation()

BeginAnimation("Fall", TiledAnimRenderPath.Create())
    AnimDelay = 0.03
    File = "images/devo/devo_fall_strip"
    AnimType = AnimationType.OneShot
EndAnimation()

This allows you to call just Actor.PlayAnimation("Idle") or Actor.PlayAnimation("Run").

This is still a bit more complicated than I'd like, but does work really well. The only thing I'd like to improve is the specification of the render path. AngelXNA actually generally selects the correct render path automatically, but the data driving system, specifically the animation system, isn't detecting its render path. I can probably fix this, which I'll hopefully do after GGJ.

Happy New Year!

For those that don't know, in May (or so) I moved from Boston to Charlottesville, VA. Thankfully, the awesome people at Fire Hose have allowed me to stay with them and work remotely, so I haven't moved companies. I have, however, moved communities.

Charlottesville isn't a huge center for game development, but I think if could be a great place for game companies in the future. There's a lot of talent to be had from the neighboring universities (UVA, JMU, GMU, Virginia Tech, and William and Mary are all fairly close) and it's just a great place to live. It's not a big city, but that's actually what I like about it.

To try to help this along, I've created the Charlottesville Game Developers meet-up group. We've had a few small meetings so far, with mostly me talking, but coming up over the next few months, we're going to have some great talks about starting companies, working with the city of Charlottesville, and maybe more! (By the way, if you'd like to come visit Charlottesville and give a talk, let me know and we'll work something out).

Tonight, I'm giving a talk on tips for game jams. Since UVA has been nice enough (at my urging) to host a Global Game Jam site this year, I'm making sure everyone comes in prepared. Most of the tips come from my previous post mortems of game jams, but there will be some new stuff in there. I'll try to be sure to post the slides tomorrow as well, so everyone can get the benefit.

Anyway, if you're in the Charlottesville area, come out to our meetings! I'm really looking forward to seeing what the Charlottesville community can produce over the next year.

So what gives? Why am I using them here?

Generally, there's nothing intrinsically wrong with a Singletons or globals. It's their overuse that becomes a problem. As mentioned above, each singleton needs to be managed carefully because it can become a potential bottleneck across your entire code. However, when you're working with something that does need to be global, a Singleton is sometimes a better option, since you can insist certain conditions be met before an instance can be returned.

A good example is a Fixed String Table. If you're unfamiliar with this, here's the idea: every string in your code is translated to a hashed number. In some cases, this translation is done manually, but for the sake of code readability and simplicity, I do the translation in code any time a FixedString object is created. In order to make sure that same string always returns the same hash, and to make sure we can translate the hash to a string, we put the string into a table. This table could be a static member of FixedString, but if you want to, say, tell the table where to allocate its memory from, or how much memory it's allowed to use, you need to initialize said table. Again, you could use a static method for this, but this is hiding what's actually happening. Furthermore, if your table class is any amount of complex, or has functions for accessing usage data, you're essentially going to be pushing that all into static methods of FixedString. It's a pain, when you are, in essence, creating a global interface anyway, since FixedString itself will probably be in global scope. At least by moving things into a singleton you potentially have the option of creating multiple tables in the future, should you so choose.

But what about all those Factory singletons from my initial post?

First, a point of clarification: in the previous post, I made it look like every factory was a singleton. I did this for simplicity, and I apologize. It actually isn't true. My factories aren't singletons. In reality, the code reads like this:

Application::Instance()->GetFactory()->RegisterComponent(…);

Application is one of the other few singletons I have in my engine. Now, I'll admit that this isn't much better than having the factories themselves be singletons, since they're retrievable from a singleton anyway. If you look at the code, it looks like my component code is now coupled to both the factory AND the application.

But in reality that's not true. My factories aren't coded or tested as singletons. If I wanted to create another factory I could, and all my unit tests will still work. And the components themselves aren't coupled to the factories or the application at all. The only thing that couples factories, the components, and the application is the auto-registration system, which serves as a bridge between all three systems. Each system is still usable stand alone.

This just leaves whether or not the Application as a singleton / manager is a good idea. The answer is, no. No it isn't. But since it is one of few, it sticks out like a sore thumb when you type it, and in code review. Every time you see or type Application::Instance() should be like warning lights going off in your brain that you're potentially doing something dangerous. BUT, since I'm mostly doing this during initialization, and at no other point, I'm okay with it.

An example is a self-registration system. This is a fairly common occurrence: you have a factory class that takes care of instantiating other classes based on something data-driven. In my scenario, this is a component system, but it really could be anything. Each component needs to register itself with the factory class so the factory can proceed to instantiate them. You have two options here: one is to have a giant init method that looks something like this:

void InitFactory()
{
     Factory.RegisterComponent(ComponentA);
     Factory.RegisterComponent(ComponentB);
     Factory.RegisterComponent(ComponentC);
     Factory.RegisterComponent(ComponentD);
     Factory.RegisterComponent(ComponentE);
     // and so on...
}

The problem with this is that it requires you #include every component header, and remember to register every component manually. Not my idea of fun.

The second is to use an auto registration system. Basically, you create an auto-registration template class:

template <class Component>
class ComponentAutoRegister
{
public:
	ComponentAutoRegister();
};

template <class Component>
ComponentAutoRegister<Component>::ComponentAutoRegister()
{
     Factory::Instance()->RegisterComponent(...);
}

And then add that to any component class:

class MyComponent
{
    static ComponentAutoRegister<MyComponent> __AutoRegistration;
};

When the static auto-registration template class is constructed, it will automatically register your class with the factory.

The problem is this: If your factory class has any other dependencies, it will need to initialize those first. If this usually happens post-main, or as part of any other variable being statically initialized, you can't use this auto-registration system. Damn.

Here's my scenario: I have a table of fixed strings that needs to be initialized with a table page size and an allocator before it can be used. My component factory uses fixed strings as keys for each component. Instead of attempting to make a long chain of singletons which would only end up coupling all the systems together (something I wanted to avoid) I slightly edited my auto-registration system to auto-register a callback instead, which is then called post initialization:

template <class Component>
class ComponentAutoRegister
{
public:
	ComponentAutoRegister();

	static void PerformAutoRegister();
};

template <class Component>
void ComponentAutoRegister<Component>::PerformAutoRegister()
{
	Factory::Instance()->RegisterComponent(...);
}

template <class Component>
ComponentAutoRegister<Component>::ComponentAutoRegister()
{
	Application::AddPostInitHook(&PerformAutoRegister);
}

The callbacks are held in an opaque linked list, managed by a zero-overhead pool pulled out of system memory. It could actually be held in an array as well, but being able to free the memory once post-initialization is complete makes me happy.

And this could easily be extended to support multiple layers of auto-registered post-initialization, so long as the layers are kept free of dependencies. For now, I only need the one layer, so I'll keep it simple.

This may seem hacky, but it does work pretty well. What are your thoughts?

Types of Crunch

The thing is, there are lots of types of crunch. And they're all different, and affect the team in different ways, and on different levels. In my mind, there are three major types:

1. Self-imposed crunch: This is the crunch that some people in the game industry do on their own accord. They stay late every night because they want to. They want to get that one last thing in, they want to polish the hell out of whatever, or get that one last boss battle in. This is a personal choice, and there's really not much you can do about it. However, it should be clear to other team members that this type of behavior is not condoned or required. Don't reward them for this behavior. Otherwise you get into the second type of crunch.
2. Team de-facto crunch: This is where the team "decides on its own" that it's going to crunch, when the culture necessitates it, or when management does nothing to stop it. When too many people start self-imposing crunch, other people at the team will start to look down on those that "aren't working as hard," and peer pressure will cause them to stay later, or work longer. This is particularly true in companies with lots of younger workers who are not only particularly vulnerable to this type or peer pressure (they really want to impress their superiors) and also believe that this is just the way the game industry is. If this bubbles up to management, there becomes this situation where long hours are expected, and people that don't do them are considered less valuable employees. Don't let this happen. As a manager, it is your job to make sure self-imposed crunch is seen as an anomaly, and to do your best to keep people from doing it.
3. Mandatory Crunch: Just what it sounds like. Management, for some reason, requires longer hours. This is the most common form of crunch we hear about, and the worst. To producers / managers out there, just because you do the same hours does not make mandatory crunch okay. Actually, nothing really makes mandatory crunch okay. There are no excuses here. Mandatory crunch is always a failure somewhere down the pipe.

The first two types of crunch are cultural and occur at all studios, but can be mitigated once you recognize the warning signs.

In addition there are three levels of crunch:

1. Mitigated crunch: The increase in hours is not huge, maybe 10 extra hours of work during the week, the length is known and can be planned for.
2. Long crunch: The increase in hours is high. Working 60 to 80 hours during a week with no break. The length can still be known, but may be a long way off.
   
3. Death march: The increase in hours is high, and there is no end in sight, or the end is repeatedly pushed back.
   

A Mandatory Death March is the worst type of crunch, and when we talk about "crunch" we're usually talking about mandatory long crunch or mandatory death march. There are no excuses for them, but we should take a look at the causes to figure out if they're avoidable.

Causes of Mandatory Crunch

Everyone looks at crunch as a failure of management. I agree, but there are a lot more factors to it than that. Here, in my mind are the general causes of crunch.

1. Company Policy: The Epic model. There are no important deadlines coming up, there's nothing that's actually requiring crunch, but the company has decided to require everyone always work 60 to 80 hour weeks. Not okay. There is no excuse for this other than management being completely inept. It affects work quality and general quality of life. Don't do it.
   
2. Bad Information: This is what happened to Team Bondai. They were constantly told (and believed) that the game was almost finished. In such cases, many team members, believing the end is in sight, will do self-imposed crunch, which eventually will lead to de-facto crunch. In addition, this is where a death march starts, and tends to never end.
   
3. Bad Planning: This is the most common cause of crunch, and management gets rightly blamed for it all the time. But there are two types of bad planning:
   
   1. Bad Low Level Planning: Even the best management team can make a mistake, because they can only make decisions based on what they're told by the people below them. If what they're told is wrong, their planning will be wrong. So, if you're at the low level, don't fudge your numbers. Tell management the truth so they can make the right decisions.
      
   2. Bad High Level Planning: Even with all the right information, managers, especially bad managers, will plan incorrectly.
      
4. Shifting Requirements: Even with good planning and good information, games are a highly creative and highly iterative product, and when something's not working or isn't fun, it has to be changed. Unlike other industries where you can work "to spec", the "spec" of a game is frequently fluid, which makes planning for the long term almost impossible. This, unlike the other causes, isn't anyone's fault, per-se, since a team can come to the mutual decision that something needs to be reworked, but it's something that frequently gets ignored while doing planning
   
5. Deadline: When all is said and done, this is the real cause of crunch. When a deadline is involved any amount of error in planning or requirements hits a head when the deadline looms. Some studios are in a position where the deadline can be pushed, but not all, especially when dealing with a multi-game publisher that has to allocate resources at various times (QA, marketing, operations, etc) to make sure the game launches without a hitch, so they can move on to the next one. In these situations, moving the deadline is impossible.
   

The Myths: Crunch is a Given and Crunch is Always Avoidable

Both of the above statements are wrong. If the causes of crunch are listed above, avoiding crunch is as simple as avoiding those mistakes. Don't make crunch company policy, always give your employees all the information, plan well at all levels, have a solid design document, and don't have a hard deadline. Simple. But, if any of those slip, you have to make a hard decision: cut features, cut polish, or crunch.

If your company has a lot of experience or a lot of leeway, you can always avoid crunch. However, if you're a start-up working for a publisher, on a hard launch deadline, under contract, and you want to make the best game possible, chances are someone, and it's not necessarily management, is going to make a mistake, and that mistake is going to cause some amount of crunch.

Was it avoidable? Yes, probably at some point it was, and yes someone made a mistake somewhere. Crunch is always a failure somewhere down the line. But at some point in the project, it will become obvious that some amount of crunch will be needed. The next step is to mitigate it.

Mitigating Crunch

Crunch does not always mean Quality of Life or Quality of Work has to suffer, it's just that in most circumstances it does, because of the way it's implemented. I have gone through two sets of crunch, both mandatory, one long, one mitigated, and both sucked. But there are a few things that I found helped me, and I think would help others, get through any period of crunch

1. Provide All the Information: Inform everyone how much more you expect them to work, what needs to be done, and where the deadline is. If ANY of this information changes, make sure the team knows. I have been in and heard about crunch situations where information about deadlines or cut features was withheld, and this only makes people angry
2. Allow people to choose their extra hours: People are different and have different home lives. Some people work better in the morning, others at night, others over the weekend. Do not blanket ask everyone to stay late. Instead, let them choose when to work and they will choose times that will not only make them most productive, but will interfere with their lives the least.
3. Allow people to work from home: Set up your infrastructure to let people work from home. Some people can be just as productive at home as they can be in the office, and this removes the stress of a long commute and improves QoL since they can be around for their loved ones.
4. Allow people during longer periods of crunch to take some time off: This is huge for me, and I've never been anywhere that does this. If your crunch is going to last a long time, or looks like it will need to be lengthened, give people a day off. Say "Okay, everyone can take a day off free sometime in the next two weeks." Giving them a time frame allows them to plan for things like short trips, or time with their family. They'll come back rested, happier and more productive.
5. Make sure they have and know the reward: In a start-up, the reward is in the stock, but in larger companies, make sure people know they will be compensated of extra hours, and then compensate them.

Hopefully this makes sense. I don't like crunch either, and it is avoidable for companies that have really experienced teams and a certain amount of give, but some companies do not and will not have the luxury to avoid crunch entirely. Obviously, great companies will always avoid crunch, usually because they've learned from their mistakes. But realize that a good company may crunch, but will always find a way to mitigate it when and if it happens, and will learn from their mistakes on their way to becoming a great company.

First, there's no tech choices or learning curve. At game digital game Jams, the first thing you have to do (once you have a game you want to create) is decide what technology you want to work in. This can be tough when you have multiple people who all come with different tech backgrounds and make them try to work together. Either you end up choosing teams based on tech that people know, or a few people end up working in tech they are unfamiliar with. This can make many digital game jams more about overcoming technical challenges, rather than overcoming design challenges in the game.

Second, other than the theme of the materials, there was no theme for this Jam. Though originally I thought this was a detriment, I am beginning to think that the lack of theme contributed to making this Jam awesome. In every game jam I've participated in, the themes have been aesthetic. They're themes like "immigration," "extinction," or "deception," and although they offer a good constraint on the Jam (and place everyone on equal footing for implementing an idea), they force most Jammers to think in terms of aesthetic first, mechanic last. This is why most Jammers can't "finish," because they're actually pushing for an aesthetic, rather than mechanical goal.

Last, jamming in board games meant that iteration cycles were really, really fast, which meant that you could focus on and tune the game and the mechanics quickly, which resulted in better games across the board. The game I worked on took about 10 minutes to play, and after initial discussion, we did nothing but play it, with short discussions in between. We probably play tested the game 50 times, each time with slightly tweaked rules to attempt to address problems we were seeing in the previous play through. Iteration times like this not only let you learn more about the game you're creating, but more about game design in general, since you see the results of you actions quickly. This speed of iteration and learning would have been impossible in a digital game jam, since implementing rule changes requires too many cycles. In digital game jams, if you get two to three full playtests in before time is up you're lucky, and that's including for teams that use prebuilt engines like Unity.

So in closing, if you get a chance to participate in a board game jam, you absolutely should. There's less stress, more learning, and way more collaboration. That, and you end up with an actually complete board game at the end, which is super awesome.