edit: Slides Posted!

Powerpoint here. Please install the font included in the zip first or the presentation will look super funky!

PDF here – some of the slides are mangled, and there are no notes, but it works without powerpoint.

This post is meant as an accompionment to my GDC Lecture: From Struggle To Redemption: Building the Chat Service for League of Legends. I reference a ton of different things in my lecture that I didn’t have time to dwell on, on, so they are listed in more detail here. Also I will edit this post and add the slides as soon as GDC allows it!

Chat Service Backend

XMPP – Open Standard Chat Protocol

EjabberD — Open Source Chat Implementation

Erlang – Language EjabberD is implemented in

MySQL — Open Source Database

Chat Frontend

Adobe Air –  Standalone flash runtime

Adobe Flex – Open source flash application framework

Adobe LCDS - LiveCycle Data Services – for communicating with the backend.

XIFF - open-source xmpp actionscript implementation

MVC - Model View Controller design pattern used in UI

Load Test Harness

Smack – XMPP client library in Java

Ec2 - Amazon’s Elastic Compute Cloud

Data Gathering

SNMP - Open Source Data Collection and Visualization

Cacti - Data Visualization tool

References

Technical Debt – Martin Fowler discusses the concept of technical debt.

Facebook And Erlang – Lecture about Facebook’s chat service.

From Struggle To Redemption is a post from: Christopher McArthur

Headed to GDC Online in Austin, TX next week to give a talk about the chat service for League of Legends! I will be speaking there Thursday afternoon at 2:30 come check it out!

My talk is a history of the problems we had launching and scaling our chat service and about how we overcame them. Many of the lessons we learned here ended being things that we could apply throughout the development of League of Legends. I also talk about the surprising ways in which UI design made a significant impact on the stability of our service.

A brief description is found on the GDC Online website:

DESCRIPTION:

Communication between players is one of the most important pieces of a social game. Yet, the nuanced relationship between a fully-featured chat service and the rest of a game are often overlooked.

In this lecture you’ll hear the story of how the chat in League of Legends’ PvP.Net was launched and how it evolved over time. We had three expectations for our chat service: that it would be scaleable, not impact stability, and the UI would meet the player’s needs.

We initially failed on all three fronts. But that’s not the end of the story.

We were able to turn around to achieve and surpass all three of our initial expectations. This talk will cover the surprising emergent issues and mistakes that caused these problems, and will detail the specific solutions that were implemented in order to solve them during a live product.
TAKEAWAY: Attendees will come away with an understanding of the pitfalls and challenges of building a scalable, user-friendly chat service. Attendees will gain an understanding of how significantly UX design and user behavior can impact scale issue. They will also gain insight into how to structure their UI code to leverage rapid-iteration on a live product. In addition, attendees will learn from case studies on specific chat UI features that worked well, and those that did not.
INTENDED AUDIENCE: No prerequisite knowledge is needed and there will be takeaways for all disciplines during the lecture. However, the lecture will be especially useful for programmers interested in scalability, UI programmers, and UI/UX Designers.

Also, it seems like Riot is totally taking over this conference! We have a total of 6 different talks we are doing there! Check ‘em out:

“Cloud Gaming: Present and Future” with Brandon “Ryze” Beck

Letting Go: Creating Self-Managed, Self-Directed Teams by Travis “Volibar” George

Designers are Human Too – Causes of Poor Design Decisions by Tom “Zileas” Cadwell

League of Legends: What it Means to Be Player-Focused by Steve “Pendragon” Mescon

and Putting the Plane Together Midair by Andrew Woo

This talk is special for me because, as I said in a previous post, I absolutely love GDC, and it has always been a career goal of mine to speak there! I was ecstatic when I found out my talk had been accepted. But, I did not know what I was in for!! Preparing for an hour long technical GDC talk is a ton of work. I really was not prepared for how much time I needed to put into this in order to make sure it was a truly quality presentation. During this time I found my friend Glenn Fielder’s blog post How to Prepare a talk for GDC an awesome read, and extremely helpful. Thank you Glen!

The subject matter is not really exactly what I’ve always envisioned doing at GDC. I’m not giving a highly detailed technical talk about a cool system that I built. Instead, I’m telling a more broad story about a rather large system, outside my specialty, that a whole team built. So I had to do a lot of work researching, and interviewing members of my team in order to get a deep enough understanding to be able to talk at least somewhat intelligently about it.

Furthermore, this was a talk about the problems we had and the mistakes that we made. It’s one thing to be humble and talk about your own mistakes, its a much different thing to have to talk publicly about the mistakes your teammates, whom you have a lot of respect for, might have made. I needed to be truthful, and share enough of our problems so that others could learn from them, but also respectful and fair to those who came before and did the really tough work of getting this system up and running under not the best of circumstances.

Overall, I think that the presentation has come together quite well and I look forward to giving it next week! An internal rehearsal at Riot was thankfully well received, and I was much more nervous about the internal presentation than the actual GDC one. But, I’m not sure I’m gong to jump at the opportunity to do another lecture like this anytime soon! (But, I hear they do get easier to prepare for!)

I will post my slides and notes up here shortly after GDC (as soon as they let me!)

Speaking at GDC Online! is a post from: Christopher McArthur

There are many issues in the programming world that seem to divide people down religious lines. These issues seem to cause endless internet flame wars and programmer rants without ever coming to a consensus.

photo credit: Philippe Put

One of these issues I’ve been seeing a lot lately is about whether to use the old school C style casting, or use the specific C++ style casts.

Here at Riot C++ style casts are part of our coding standard and the use of C style casts is generally discouraged in our codebase. This is a pretty common standard, as seen even in the Google Style Guide.  But, conversely you have several major game studios with the inverse standard, who ban any use of C++ style casts and allow only C style casts. I’ve spoken with excellent programmers at places like Blizzard and Insomniac who say they will personally never use C++ style casting in their game code.

Isn’t it quite strange that multiple AAA studios can have completely opposite coding standards on this topic? At first I thought, like with a lot of C++ features, this was a console studio vs PC studio type of issue, but from my conversations with colleagues, I dont even think this is the case anymore. It seems to simply be a matter of style and personal taste.

C++ Style Casting

syntax: int myInt = static_cast<int>(myFloat);

For a detailed description of C++ style casting check out this paper on the subject. C++ broke up casting into several different keywords with different use cases, here I will take you through a super-quick overview of them:

static_cast: The most common C++ cast. Casts between like types, with some compile-time checking.

dynamic_cast: Casting with runtime type checking. Checks to make sure a cast between a parent and a child class is valid, and can return NULL or throw an exception if it is not.

const_cast: Used for adding or removing the const keyword from a type.

reinterpret_cast: Used to cast between any two types, even between incompatible or opaque types.

There are many advantages to using C++ style casts over C style, including:

1) The intent of the programmer is more explicit. Since there are different keywords for different types of casts, it is easies to understand what the programmer wants to accomplish. For example, when you see a reinterpret_cast, you know the programmer is intentionally casting between incompatible types for a specific reason.

2) C++ style casts are safer. The most common cast that replaces C style casts is the static_cast. If you try to static_cast between incompatible types, the compiler will warn you, whereas it will not with a C style cast. For example, if you have the forward declaration of a class, but forgot to include the complete definition, and perform a cast to that type, then a C++ static_cast will complain at compile time, however a C style cast will compile fine and  most likely crash at runtime.

3) C++ style casts are greppable. It is easy to do a find in files to find all instances of a C++ style cast such as const_cast in your codebase. However, this is much more difficult with C style casts.

4) dynamic_cast provides runtime type checking, while a C style cast can not. Most games have RTTI disabled for performance reasons, so this one isnt as interesting, but this feature is still occasionally useful in tools code.

C Style Casting

syntax: int myInt = (float)myFloat;

C Style casting is the one that most people grew up with and are most familiar with.  Put some parenthesis around the type you want to cast to and away you go. So why do so many people stick with this old school way when C++ has eliminated the need for it?

1) The C style syntax is much easier to read. The C++ style casts takes the short form of the C style, then adds a verbose keyword, and angled brackets, making it much uglier and harder to quickly parse. It is very important for programmer’s to be able to read code quickly, and easily get an understanding for what is happening at a glance. Many people feel that the janky syntax of C++ casts obfuscate the code when compared to the more elegant C style casting syntax.

2) Everybody is already familiar with C style casting. It is widely used and understood, and when writing code using them you can almost guarantee that any other C/C++ programmer recognizes it and understands that. C++ style casting is just not as widely used, making it so that your code is not as universally understood by other programmers.

3) Since the C++ casts are more complex than normal ones there is more potential for programmer’s to not fully understand them and potentially misuse them. Simply using static_cast for example can sometimes change the value of a pointer, and as one programmer I know expressed: “I  never use any language construct that requires moving the ‘this’ pointer behind my back.”

4) dynamic_cast will slow down your game because it uses RTTI. In addition to being an expensive operation, relying on dynamic_cast means that your code is probably badly designed and very brittle.

My Stance

At Riot we are particularly partial to the use of C++ style casts, and I fall squarely on that side of the fence as well, however I agree completely that dynamic_casts are bad news in runtime code. Like many studios we have RTTI disabled, so the option to use them is not even there.

This is, however, an unlikely place for me to fall on this issue. I typically am a “readability above all” type of programmer, and the arguments that C++ style casts make code uglier and harder to read at a glance ring true with me. However, I feel that the added safety of C++ style casting and the explicit intent they communicate much outweigh this downside and make them a welcome addition to any C++ codebase I am working in.

How do you handle casting in your codebase?

Programmer Religious War – C vs C++ Style Casts. is a post from: Christopher McArthur

I received a ton of responses from all of you job hunters out there to my last post on resume tips. Most of the questions I received was from people wanting to know how to prepare for the next phase of the process and be able to do well on their interview.

There is already a ton of general interview prep help out there, so I wanted to give you guys something a little more specific to game development. I also wanted to give you guys a little insight on how I prep for interview. I am kind of a technical book maniac; even when on vacation, lounging on the beach you will find me with a stack of programming books. And yeah I am talking about actual BOOKS, not just websites and Google. Google is a great resource, but for real learning there is nothing like the physicality and singular purpose of a book when trying to force your focus onto new concepts.

So following are my top 5 book recommendations for preparing for that game programming interview. Not only have I used these books myself when preparing for an interview, but many of the questions I ask when interviewing come from these. They cover the basic fundamentals that  a game company will expect you to know.

With these recommendations, you really don’t need to read each one cover to cover. Look at each one as a menu you can sample from. Flip around, read what piques your interest, and mark chapters to come back to for further study.

I want to mention a few common book recommendations that did not make my list. These are Design Patterns and Code Complete. These are great books, and working in the industry you will hear them mentioned alot, but I dont really feel they really offer much practical material that will be specifically helpful on an interview, especially when studying under a short time frame. In addition, while these books are great classics, which also means that some of their material is a bit dated now.

1) C++ Coding Standards  by Herb Sutter

2) Effective C++ by Scott Meyer

C++ is probably the single most essential and commonly covered skill in game programming interviews. Riot is a little more C++ focused than many game companies since we are so PC focused, but you will find C++ language questions a staple across the industry and you better make sure you are ready to tackle them confidently.

“C++ Coding Standards” and “Effective C++” are hands down the best C++ books available. Their style of short, enumerated topics, each only a few pages long make them extremely easy to read and very friendly to skimming. Be aware that both of these books rely heavily on parts of the standard library that companies like Riot use, but many game companies out there do not use. Even so, however, the concepts still apply. If you don’t have time for anything else, at least read these books!

3) The Pragmatic Programmer by David Thomas

This is perhaps my favorite programming book of all time.  Its funny, its practical, and it sometimes goes against the grain of “standard” programming advice you may have had drilled into you at school. This book is a great look at many of the common every day issues that come up on a programming team, it will give you the vocabulary and talking points you need when discussing general engineering discipline during your interview.

4) 3d Math Primer for Graphics and Game Development by Fletcher Dunn

When walking into a game programming interview you’d better know your dot products from your cross products, because basic 3d geometry and linear algebra is a staple of almost every interview I’ve seen. Mr. Dunn’s book is a perfect refresher for the exact things you will be expected to know. Certainly there are much better math textbooks out there, that cover this stuff in much greater detail. But, the great thing about “3d Math Primer” is that its not a textbook. The information it covers is extremely practical, its very readable, and its actually pretty funny (not something you can say for many math books!). While the book has a focus on graphics, don’t overlook if just because this happens to not be your specialty, most of the material in the book applies very directly to general game development that employers expect you to know even if you are not a graphics programmer.

5) Data Structures and Algorithms for Game Developers by Allen Sherrod

I feel weird recommending this book, as its a little beginner focused and honestly, it has a lot of very questionable C++ code in it. There are a lot of awesome textbooks on the subject which are just a lot higher quality and go into great detail on the subject. But that exact weakness is also the strength of the book  in a cramming situation, it focuses on the high level parts that you need to know. A book like this is perfect for quickly getting back up to speed on your basics of which sorting algorithms are O(N^2) and which are O(n log n).

As far as data structures are concerned, at Riot we use the Standard Template Library (STL), and you should familiarize yourself with it’s basic data structures. Even though many game companies never touch STL (especially console-oriented studios), often their internal implementations are heavily influenced by the STL interface, so its very useful to be familiar with it.

5 Book Recommendations for Preparing for Your Game Programming Interview is a post from: Christopher McArthur

Post GDC we received a lot of resumes. Which means that my team has to spend a lot of time going through resumes of game engineers and deciding which to follow up with, and which to pass on. And because of the high student population at GDC the majority of these resumes were from students with no professional experience of any kind. So how do you decide the fate of someone based just on a quick glance at a summary of their life?

You really can’t know for sure, resume judging is a black art more than a science. At some level it is just a roll of the dice and a hope that you dont critically miss. The only way for me to truly judge these canidates would be to talk to them, and to spend time with them. Unfortunately, that is just not possible with the amount of interest we get, and the field of applicants needs to be culled.

The good news is, that even as a green student there are things you can do to make your resume stand out, and with the right preperation, and a little luck you can survive the bloody battlefield of resume sorting and find your way to the phone interview.

1. Projects. Projects. Projects.

We all know the classic catch-22: no job without experience, but no experience without a job. Fortunately, in the game industry there is an awesome loophole: you don’t need a job to make awesome games!

Make sure to provide a link to a website where your projects can be viewed. Videos, screenshots, and game downloads that showcase the games you have made are what convince me that you are a hotshot worth wooing.

Don’t have any projects? Well get on it! Its never been easier to work with the same tools the big boys are using, UnityEngine has a cheap student license, Half-Life and Unreal have a very active Mod communities, and the XNA framework is just awesome for creating XBOX and PC games. This is how I got my start in the game industry, and this is still the easiest way to impress me with your resume.

2. The importance of your school.

So this might be a little hypocritical coming from me, as I never finished my degree! But, a hard science or engineering degree can go along way, especially if its from a top school such as CalTech or MIT.

The game development specific schools out there are a mixed bag. Their hard math and science regiment isnt as in depth as a tradition school, but they have the huge upside of getting to make actual games. With people who went to these types of school I am looking to see if they took a leadership role in any of their projects. Lately, Ive been seeing a lot of resumes from students of the game development schools who just list the projects on their resume with a short description of them and no links for more details. Don’t make this massive mistake! Without being able to see the actual projects you are working on its almost like you didn’t even do them, make it easy to check out the cool work you did!

3. Focus on your specialty.

You really want to work in the game industry, so much so that you would be willing to take any job: Coder, designer, producer, heck even getting coffee! Your passion is awesome, but this lack of focus can be  poison for your resume. Pick what you are really good at and go after it, showing that you know what you want and have been focusing on learning all you can about it.

Even as a student I like to see programmers who have deep-dived into some area of expertise. Highlight the graphics programming you did on one of your projects, or the awesome tool you built to design content faster, or the gameplay systems.

This doesn’t mean you are doomed to be pigeon-holed, it means that your potential employer can envision exactly what you will be contributing from day one. If I find someone who is really smart in a few key areas, I know that they can learn other areas and branch out. But, when someone is just mediocre in a lot of things, I am much more likely to pass them over.

4. Highlight your achievements.

When looking at your resume I am looking for signs of an overachiever. If your game projects have won any awards or received any special notice make sure to call this out. But, even non game specific stuff that you really kick ass at is important to show off. Maybe you have a black belt in Karate, you got to world finals in Magic: the Gathering, or you were Editor in Chief of your school paper. Let me know about these things and I am more likely to give you a second look.

5. Keep it short and sweet.

Keep your resume short: one page or less. A short resume forces me to focus on exactly what you want me to see. Leave off the stuff that is not relevant and just not flattering. I really dont care that you flipped burgers at Jack in the Box, or were a waiter at Olive Garden, this stuff is not showing that you have a work ethic, it is just sabotaging you’re chances of getting an interview by distracting from the important stuff on your resume.

You can also lose the long list of skills that lists every single programming language you’ve ever heard of and every software package that comes installed with Windows. Listing “Microsoft Word” on your Resume is not impressive, and unless you can answer in depth interview questions about all of those programming languages that you listed you are just going to make yourself look bad during the phone screen. Remember tip #3, focus, and pair that list down to just what you are really bad-ass at.

5 Resume Tips for Total N00bs to Land a Job as a Game Programmer is a post from: Christopher McArthur

Just got back from GDC 2011 and had an amazing time.

As always I have come back from GDC feeling inspired. Talking to people at the top of their game, pushing the envelope on game development always starts a fire in me to do the same. At no time do am I more excited about coming into work every day than the weeks after the GDC.

My favorite lectures this year come from the excellent engineering team at Bungie. Their lectures were both interesting and very practical, their engineering team is extremely open with the game development community, which leads to better engineers and better games for everyone, which is only a good thing for our industry.

On to my top 5.

1. Networking the gameplay of Halo: Reach

by David Aldridge of Bungie

http://schedule.gdconf.com/session/12126

David took us through his journey of updating and optimizing the network code for Halo: Reach. We got some cool videos of their profiling tools, which allows you to visually inspect any object in the gameworld to see exactly when and how the engine  sends packets over the network. This allowed Bungie to do a ton of fine tuning on a per-object basis. For example, this visualization caused them to notice an anomaly in dropped weapons and they reduced their entire network bandwidth by 10% simply by adjusting the rate at which weapons slid down hills!!

One of the most interesting innovations was the addition of a “lag button” in playtests. They instructed their testers to press this button whenever the game felt laggy. They could then take the replay and fast forward to these points in order to debug the networking. What they found was that players were pressing this anytime something confusing happened to them in the game. If they got killed from behind quickly and the deathcam didnt show the killer, they would press the “I Lagged” button. From this they were able to fix game design issues they had never noticed before. I love this idea and I am definitely stealing it!

Unfortunately, he only spoke briefly about how to predict bullet-based insta-hit weapons which is a really interesting talk I hope he covers in detail at a future GDC. However, there is a great article on how Valve tackled this problem from 2001 available here: http://tinyurl.com/coljwf

2. Halo:Reach Special Effects

by Chris Tchou of Bungie

http://schedule.gdconf.com/session/12127

Only 25 minutes long! Yet packed to the brim with awesome stuff. A highly technical lecture that Chris made easily understandable. A particle system that runs completely on the GPU packing basic particle state into a texture. Using the depthbuffer to handle particle collision allowing for completely GPU-based bouncing rains and sparks! 

3. Iterating on a Dynamic Camera System

by Phil Wilkins of Sony Santa Monica

http://schedule.gdconf.com/session/12364

Another lecture that just had a ton of practical, useable techniques. Phil took us through the history of the God of War camera from its first iteration in God of War 1 to the final version that shipped in God of War 3. The camera system starts by taking designer placed nodes around the map that are near the action, and blends them together to get a final camera. The combat system then puts in its two cents and modifies the camera to perfectly frame the action taking place in the scene.

Usually defining the rules for a camera system is the easy part, its all the little edge cases and gotchas that really make it difficult, and Phil walks you through several of these edge cases that they ran into, making this talk especially valuable.

Phil’s original lecture is available here: http://web.cs.wpi.edu/~rich/courses/imgd4000-d10/lectures/camera.pdf His updated lecture for God of War3 is not available yet.

4. Forensic Debugging

by Elan Ruskin of Valve

http://schedule.gdconf.com/session/12129

This one really opened my eyes to a lot of interesting debugging techniques. Taking us deep into what the compiler actually does with your code, Elan stressed the importance of understanding some basic assembly in order to track what was actually happening in your release executable. At one point the audience gasped when Chris revealed a stack trace in which the callstack was completely corrupted, that the pattern of the corruption formed the letter “A” if you looked at the memory just right, revealing a problem in the font code! Unfortunately, Chris moved a bit too fast for me and left me in the dust a bit. But thankfully, he posted his slides up (http://tinyurl.com/624efkf). This is one of those lectures that I’m going to be spending a lot of time going back through and reading on to try and fully grasp.

And the one that surpised me the most:

5. Industry Lessons Learned

by Cliff Bleszinski of Epic

http://schedule.gdconf.com/session/12352

I didnt know what to expect for this lecture, but it ended up being very entertaining and inspirational. Cliff talks of the idea of a “Power Creative”,  a developer who is highly visible, highly active, and plays a leadership role on their team. He encouraged more companies to nuture and grow their own Power Creatives, and expressed concern that he had never met a developer from Rockstar, despite all their successful games. He also stressed the importance of understanding how to sell, pointing out that every great idea has to be sold first to your teammates, then to your company at large, and then to the public, so it is a very valuable tool to possess even for those of us in the trenches.

–

* The cheese sucked

Best of GDC 2011 — Talks and Lectures and Cheese* is a post from: Christopher McArthur