bitquabit

Cargo-cult debugging

2012-05-11T10:23:45Z

I’ve been coding full-time for only a few weeks, and already I’m going somewhat insane by people engaging in what I’d call cargo-cult debugging. Cargo cults were religions that developed when primitive societies, who’d had little exposure to any technology, were suddenly confronted with top-of-the-line modernism in the form of World War II military machines. When the armies disappeared at the conclusion of festivities, they took all of their modern marvels with them. The locals, believing that they’d observed gods bringing them promised goods, attempted to make the gods provide more cargo by building crude imitations of what they had seen—bamboo airplanes, fake landing strips, wooden radar dishes, and the like—without really having any proper idea what any of the things they were building actually were.

Cargo-cult debugging is when, having seen effective debugging, you imitate the motions, but without having any actual clue what you’re doing or why.

Let’s take a look at some cargo-cult debugging in action.

Problem Description

Two problems, likely unrelated, but including both for completeness since they started at about the same time: car tilts to the left and slightly backwards, and there is a grinding noise coming from the rear-left wheel-well.

Current Hypothesis

Engine belt is likely applying too much torque, resulting in car listing to one side.

Solutions Attempted

Tried turning on left blinker, since car tends to lean right during left-hand turns.

Tried having extremely fat man sit in passenger’s seat and drive car via strings attached to steering wheel to balance the frame.

Attempted to drive car backwards. (Note: visibility extremely hampered in this mode; file bug upstream with manufacturer.)

Tried covering wheel-well with thick cloth to muffle sound, thereby evening out the frame.

Tried driving with windows down to change airflow. (Note: test cut short due to sparks flying from rear-left wheel-well and a small resulting fire, which also prevented further testing.)

The structure of a good debugging session is there—the facts, the possible solutions, and a hypothesis—but even being very generous to the tester, their ideas and hypothesis make no sense. You’re going to solve the problem with exactly as much speed and precision as if you played Tetris for an hour and then changed five lines of code at random.

I’ve seen every developer engage in this behavior sometimes, and that includes me. Not only that; every developer I’ve seen who does this does it for one of two very closely related reasons:

They do not understand the problem.
They do understand the problem, but the most likely solution involves rewriting some very hard-to-understand code, or some code that they’re personally very attached to.

I’ve got a bug right now that’s right in between these two, and guess what? I’m guilty.

Our internal build of Kiln doesn’t run on 32-bit Windows 2003 systems, even though the 64-bit build works fine, and the 32-bit build works fine on all other Windows platforms we support; just not Windows 2003. The error message Windows generates is beyond unhelpful, and googling it indicates that the problem likely has to do with a bad manifest—something I barely know anything about. They go in resource forks, I modified one in Copilot once to get theming in Windows XP, and I seem to recall XML being involved. That’s it.

Here is the right way to address this bug:

Learn more about manifests, so I know what a good one looks like.
Take a look at the one we’re generating for Kiln; see if anything obvious screams out.
If so, dive into the build system [blech] and have it fix up the manifest, or generate a better one, or whatever’s involved here. This part’s a second black box to me, since the Kiln Storage Service is just a py2exe executable, meaning that we might be hitting a bug in py2exe, not our build system.
If not, burn a Microsoft support ticket so I can learn how to get some more debugging info out of the error message.

Here’s the first thing I actually did:

Look at the executable using a dependency checker to see what DLLs it was using, then make sure they were present on Windows 2003.

This is not the behavior of a rational man. But it’s the behavior of a man who’s flirting with the edge of his knowledge and has no desire to screw with the build system: fixing DLL dependencies involves tweaking a line or two in setup.py, which is way easier than learning a bunch of new stuff about manifests, diving into how py2exe makes its particular brand of sausage, and then patching it upstream or mucking about on the build server to add post- or pre-build steps as appropriate.

So here’s my call to action: do not engage in cargo-cult debugging. Whenever you’re about to try a debugging session, force yourself to answer the question, Why do I believe this is the most likely solution to the problem? If the answer is, “I have no idea,” or even worse, “It’s not, but it’s easier to check than this other more likely solution,” then keep looking. Is your app slow even though the CPU is basically idle? Increasing the size of the thread pool probably won’t help. Web service occasionally timing out? Your Redis server is unlikely to hold the answer. Occasionally getting your app’s network messages out-of-order? Attacking layer 4 of your networking stack with Wireshark is probably premature.

We developers pride ourselves on being “lazy” in the sense of “developing solutions that minimize the work we have to do.” If you engage in cargo-cult debugging, you’re being intellectually lazy, which is entirely different. Intellectual laziness turns you from a great developer into a mediocre one.

Be a great developer. Don’t engage in cargo-cult debugging.

Coding is priority number five

2012-04-13T10:54:19Z

Let’s set the scene. It’s the summer of 2010. Kiln had been launched into the wild for all of six months, after a grueling year-long, no-revenue sprint to turn my dinky prototype that ran only on my personal laptop into a shipping application that worked both in Fog Creek’s hosted environment and in a gazillion ever-so-slightly-different on-site installations. We’d had all of a few months actually charging people, and were only just barely making a month-to-month profit, let alone having a positive ROI. We were thinking about What Would Come Next, and how to deliver That, and What Would It Look Like, because everyone knows that standing still is death. And me? I was enjoying coding every day to turn the vision in my head into something that our customers could actually use.

But then it happened. I came into work one day, and Ben Kamens, the head of the Kiln and FogBugz teams, called me into his office to tell me that he’d decided to leave Fog Creek and join the Khan Academy.

I felt sick. Tyler, who’d help found Kiln, felt sick. We left the office in a daze and had a lovely lunch that consisted mostly of beer, then came back and did a half-assed job pretending we were doing work for the rest of the day before we went home.

Part of our despair was personal, but a lot of it was professional. Here’s the thing: you may not know what your team lead does, but if they’re awesome the way Ben was, a huge number of problems get solved without you ever hearing about them, which frees you to work on actually shipping amazing products. So his leaving meant that:

No one would be shielding us from all of the problems peripheral to actually shipping new versions of Kiln; and
We were going to be making fewer, crappier releases as a result.

Of course, it wouldn’t actually come to that; one of us was going to learn how to become a team lead so that (provided we were good at it) we’d be able to continue to shield the rest of the team from all that stuff. At the end of the day, I stepped up and took the job.

Six months later I’d be at the end of my rope. I hated my job. I felt like I wasn’t accomplishing anything, I felt like I wasn’t competent at what I was doing, and my attitude deteriorated until my fiancée came home one night to find me physically ill from stress.

The thing is that I didn’t stink at my job. In fact, both my team and the rest of the company have told me that I’ve been a great team lead. The problem I was suffering from is that the only job I’d known at Fog Creek up to that point had nothing to do with the job I suddenly found myself thrown into.

We have a general cultural issue when it comes to coders. It has to do with career paths. Here’s how your typical coder career path works:

Hey, it looks like you can code! Why not try writing this feature over here? There’s the spec for it; just follow it and you’re golden.
Whoa, neat! You do a great job writing exactly what I tell you! Here’s a problem customers want solved; can you do that?
Awesome! That’s totally amazing. I heard you have an idea for something that’d be the bee’s knees, or at least one of its major joints; can you build it? Here’s a team who’ll work with you.
Insane! Congratulations! You are an amazing coder. You have founded a product that shipped and is in the black! You know what we should totally do with you?
Based on your long history of dealing with insanely complex technical issues, we’ve decided to make you a manager in charge of five to twenty people, because this is totally both the same skill-set and the same general area of interest as what you had before! Please as to enjoy with maximum intensity!

You spend years of your life focused on building things. Sometimes you have the idea, sometimes other people do. You like it better when you have the idea, because you “know” it’s the right thing, but the point at the end of the day is you build stuff. You build hard stuff sometimes that requires you to talk to and convince five, ten, fifteen, twenty developers that yours is the right way to go about things, which is definitely leadership and kind of looks managery, but you’re still down in the trenches writing code the whole time. Chances are pretty good hg churn and git log are gonna have your name there at least as much as anyone else’s.

Team leads are different. Your job, should you accept it, is to become what I’ve lovingly dubbed Shit Umbrella. Your goal is to find all of the peripheral stuff involved in getting the product out the door—important stuff, such as making sure the delivery schedule for the new servers makes sense for when you want to ship the product that needs them, or taking customer calls at 11 PM on a Sunday because their account quit working and they want to know why they should keep paying you, or figuring out when doing features the sales and support teams want makes financial sense—and then coming back and presenting a focused direction to all the developers so that they can get the features written without worrying about how they actually ship. You switch from doing the building yourself to enabling others to build stuff on your behalf.

I keenly remember sitting in our General Manager’s office, explaining how I just couldn’t do all of my job responsibilities, and him responding that I probably just wasn’t thinking about my job properly. We sat down and enumerated all of my responsibilities. We came up with five tasks, and then we ordered them by priority.

“Writing code” was priority number five of five. If I had anything I needed to do in one of the other four categories, that had to come first.

Once I accepted what I needed to do, I was both better at it and vastly more relaxed. As a developer, a good day is one where I land commits all day long, or hammer out a blog post that gets people talking, or solve a complex support case. As a team lead, a productive day might consist entirely of phone calls, one-on-ones, and emails that let the rest of your team get their stuff done. If you’re serving as a multiplier for your team’s productivity, then you’re being a great team lead. You need to redefine your success as helping others achieve success.

I learned a lot and became a pretty good Kiln Team Lead over the last couple of years, but several weeks ago, I came to the realization that it wasn’t what I wanted to be doing right now. I missed building things with my own hands, and I wasn’t really learning anything new about leadership, management, or the process of writing software.

At a lot companies, this is where you’d see me writing something like, “so I’ve decided to leave $COMPANY, take a sabbatical, and then join $OTHER_COMPANY to find new challenges.” At Fog Creek, that’s not how it works. Joel and Michael have a strong attitude that good developers should be rewarded as developers. When I went to them and told them that I wanted to get new experiences and get back into the writing part of writing software, they were really happy to make that happen.

So today? Coding is priority number one. Or occasionally two or three, but never five of five. I’m the senior-most developer on the Kiln team, and I’m back to writing code in that capacity. At the same time, to keep learning new things, I’m working both with a high school in Queens on their CS curriculum, and with an awesome TechStars company as a mentor providing leadership advice and technical guidance. I’ll hopefully have a few other announcements over the next several months as well.

If you like the sound of an environment that understands developers that way, you should come join me. Me? I’m right where I need to be right now.

Learning coding from boredom

2012-02-27T12:00:00Z

I think the point of math class is probably to teach people math, but what many of the best developers I know actually learned in math class was how to program.

Nearly every high school math class I took was really, really boring. Not through the fault of the teachers; they were actually awesome. But I consistently knew just enough to be bored, yet not enough to actually skip the class. At first, I tried to act like I was paying attention, which meant that my face had to be vaguely directed at the teacher, even if I was actually studying the posters on the wall. It was in that mode of thought that I finished off Algebra II able to regurgitate π out to maybe 50 decimal places.¹

But that only lasted so long before it occurred to me that, “hey, I know how to program!” combined with “hey, I have a programmable calculator!”² meant the inevitable “hey, I can program this thing to do my math homework!” So I began writing programs to do factoring, to solve equations, to help rotate ellipses and parabolas by arbitrary angles, and pretty soon reached a point where doing a lot of homework mostly involved firing up the right program, plugging in some numbers, and writing down the result.³

While I’d done a fair amount of programming by this point, my calculator programs were the most exciting to me because I was really using them. Yeah, okay, the what-kind-of-rock-is-this database I made for science fair taught me more about data structures, but these were the first programs where I cared about the user interface, where I spent time refactoring so that extending the thing would be easier, or where I’d spend 20 minutes optimizing just to shave off a second or two.

That’s my story. But I know a lot of developers who have almost the exact same story. Some couldn’t use their programs on assignments; some wrote video games instead of math helpers. But in all of these cases, people started getting passionate about what they were writing. And when I’ve talked to them about why this was the first time they really cared, I always get back the same answer: because for the first time, they had end users they cared about. Themselves.

My high school also had a CS course. I didn’t take it. No one I knew who really liked programming took it, in fact. And those who did take the class universally went on to do something, anything else, when they got to college. What was painful for me was that even as a clueless high school student, I knew why: because the whole damn course was totally irrelevant to them! No bearing on what they were doing at all. Here the programmers-to-be were writing toy programs that they were using daily, even competing to see who could make the fastest, most useful ones. The AP class? Write another three implementations that sort random numbers. Yip-de-frickin’-do.

It’s not that you shouldn’t teach that—and, in fact, the number of applicants here at Fog Creek who don’t grok basic algorithms drives me crazy. But it’s completely the wrong way to introduce people to programming.

Programmers like to program because they can do cool things, or because they can solve problems, or both. It’s both creative and it’s practical. If the goal of a high school course is to get people interested in programming, then the course must build around these two pillars.

You wanna appeal to the first group? Show them Mindstorms so they can show their friends a cool robot. Show them Twilio so they can see how to make a little voice-controlled system. Show them LÖVE so they can put together a simple video game.

You wanna appeal to the second group? I think you could do worse than to introduce them to something like VBScript or AppleScript so that they get a flavor of manipulating the applications they use every day. If the total lack of ideological purity of those two bugs you, then introduce them to some Greasemonkey, or some light shell scripting, or maybe even (gasp!) TI-Basic.⁴

When you attack programming this way, students get hooked. They become their own critics and their own users, so they genuinely care about improving their work. They’ll naturally hit the limit of what they can do, and the good ones—the ones who are going to go on to become programmers—will naturally want to start learning more of the theory, the underlying technologies and techniques, so they can continue to improve their craft. When those students get to an AP prep class later in the curriculum, they’ll be ready for it and enjoy it, because they’ll understand why it matters. It will have become relevant to them.

I’m lucky right now to be working with Bayside High School in Queens, who’s developing a program for CS students that looks a lot like the above, with introductory classes focusing on tangible results students can play with immediately (web applications, little GUIs, and dumpster-diving through massive datasets) rather than wading knee-deep into theory right from the get-go. But there are many, many high schools out there who have nothing remotely like this, who teach programming the same way they teach math. If the high school in your area’s like that, volunteer to teach something more inspiring. The students will love it, and we’ll get more impassioned developers as a result.

I can still recite about 25. Why I’ve forgotten about half the digits of π, but can still happily rattle off the advertising jingle to Kanine Krunchies, which isn’t even a real product, is anyone’s guess. ↩
Anyone else think Texas Instruments has the best racket ever? ↩
My teachers took what in retrospect was an unbelievably forward-looking attitude to this: as long as I was writing all the programs myself, and could explain how they worked, it was fair game. I have no idea what would have happened otherwise. ↩
I know some of you want me to mention UserRPL here, but I think it’s time to admit that the HP-48 is the Amiga of graphing calculators. ↩

Enslaving your interns for evil and profit

2011-10-07T15:34:16Z

I should be in the middle of an interview right now. About fifteen minutes into it, in fact. About the part of my interview where we stop talking about awesome stuff the candidate has worked on in the past and start diving into writing some actual code. A stack with O(1) data access that also always knows its maximum, for example. Or perhaps a rudimentary mark-and-sweep garbage collector. It’s usually my favorite part of the interview: I get to see how the candidate thinks, how they process information, how they problem solve, and how they code. The best candidates even teach me something in the process.

But I’m not in the middle of an interview right now. I’m in Emacs. And instead of being excited about watching someone solve an interesting problem, I’m upset and full of righteous indignation on behalf of the now-former job candidate.

You see, the candidate works for a company that is very scared.¹ I know this because they made the candidate—let’s call him Bob—they made Bob sign a non-compete contract.

Non-competes are annoying, but not the end of the world. I can understand why, say, Apple, might be legitimately angry to have a senior iPhone manufacturing executive jump ship to HTC: a large part of the candidate’s value to HTC would be his knowledge of the internals of Apple’s manufacturing process. But very few jobs work like that. And even there, most companies don’t forbid you from working at all for a competitor; they forbid you from working in the same area of expertise. So, for example, maybe the iPhone executive couldn’t work on HTC’s manufacturing operations, but he could still head a software development team.

But Bob’s company decided that, nope, Bob couldn’t come work for us, because the existence of Joel on Software’s careers board made us a direct competitor with them.

Let me tell you something about the Joel on Software careers board: Fog Creek doesn’t even make it. We outsource the whole thing to a little-known company called StackOverflow.

It’s true. The lie exposed. If you’re in doubt, take a look and notice the admittedly subtle similarities:

Shocking, I’m sure.

Yet based on this, Bob’s company told him that Fog Creek and his firm were direct competitors, and therefore he couldn’t even come to work with us on, say, Kiln.

Here’s the thing: Bob’s a junior in college, and the company doing this to him is one he merely interned at. There is no way that Bob has inside knowledge of how a job board runs that could help us. And even if he somehow did have that knowledge, we don’t even run the job board! There’s no way he could actually give us anything that would help us! But Bob’s company made him turn us down, before we could even interview him, because they were absolutely, utterly terrified that their intern, who is still in college, was so amazing that his coming to work for us could crash their entire company.

I actually feel really bad for Bob’s company. They’re so unsure of their own ideas, so negative on their own potential, that they believe a former intern being physically near a company with an outsourced jobs board would be enough for that company to absolutely crush them. I don’t want to imagine what it feels like to get up every day and face that world. But that’s no reason to inflict such a morose world view on your interns.

So I’m upset on Bob’s behalf. Bob got shafted by a company he interned at. A company that has so little confidence that they’ve decided the best route to their success is to limit Bob’s choices. Limits that mean we miss out on an awesome candidate, and Bob misses out on an awesome job.

I’ve got two real points to make, at the end of the day:

If you’re an intern, don’t sign a non-compete contract. You have absolutely no idea where your life is going to take you, and you don’t want your direction being shaped by one crappy employer. And, trust me on this: no reputable software company I know of (Google, Microsoft, or Fog Creek) makes their interns sign non-compete contracts. You can find a job that won’t make you do that.

If you’re a company, don’t be a vampire. If you’re so scared of everyone else that you believe that you have to give your interns a non-compete contract in order to stay competitive, then guess what? You’re not competitive. Get a better idea.

And based on the quality of their homepage, probably with good reason, but that’s neither here nor there. ↩

Make Love, Not Flamewars

2011-06-28T08:23:08Z

I sincerely doubt that the statement “I like Mercurial” will catch anyone who reads this blog by surprise. I brought it to Fog Creek. I evangelized for it on the Fog Creek World Tour. I helped build a whole product around it. I’ve gone to a Mercurial coding sprint, I’ve sent a whizkid to a Mercurial coding sprint, and I’ve even written a few patches (mostly trivial) for Mercurial. So let’s agree that I like Mercurial an awful lot.

Reading crap like this pisses me off.

The question seems innocuous enough at first glance: Why is Mercurial considered easier-to-use than Git? A legitimate, simple question. And in a sane world, perhaps, a fair one.

But that’s not how things work amongst us developers, because we have these utterly inane religious flamewars. Emacs is for octopuses, Vim is for beep mode. Ruby is a language for potheads, Python is for BDSM fetishists. DOS is for PCP-using masochists…okay, that one may be legit, but it’s also kind of moot at this juncture. Point is, there are some topics where developers just cannot have a rational discussion anymore.

Mercurial versus Git is one of them. To save you time, here is every single Mercurial v. Git discussion I’ve read since 2005:

Harry: Mercurial is awesome, because it is easy.

Sally: It’s not easier than Git! You’re just too dumb to see the light!

Harry: Okay but like Git is written in Perl! And it doesn’t run on Windows! And I had my changesets fucking garbage collected once! And the man pages are like 900 pages!

Sally: It’s not, it’s written in C, and it’s ninety bajillion times faster than Mercurial! And I had to screw with its database once! Also Git does too run on Windows, and I lost nine weeks worth of stuff in Mercurial once due to enabling the MQ extension!

But don’t take my word for it; you can go watch the internecine fanboyism play out at Reddit, right now! In response to that crap question! Again! God! I live in Groundhog’s Day!

You know what?

Both sides are full of shit.

Yeah, I believe that Mercurial is easier to use than Git. But the best example anyone can provide on StackExchange is converting the author names in a pile of commit messages? Really? Because, okay, yes, I’ve done that. Maybe twice. And even if it completely stank, I’d just memorize how to do it and get on with my life.

What about the daily workflow? Is that easier? Why? Is it fewer commands? Is there more protection from you shooting yourself in the foot? Does the protection come with less power? Why do Git users swear by branching, if “it’s in Mercurial since <some low version>”? Why do Mercurial users swear by their branching system? Does Git have an emulation of it? Why or why not? What are the trade-offs here? Do they actually end up mattering?

These questions matter. And they’ve been answered, very eloquently, many times. But for whatever reason, that doesn’t resonate with people. They want to find the weak argument that they can refute, and then that gets traction specifically because it’s weak, and everyone can have the fight all over again.

This stops now.

I have a solution.

Instead of talking about why you are better than the other guy, let’s focus purely on why your system of choice rocks. That’s it. No whyfooisbetterthanx.com. Just whyfooisgreat.com.

Here, I’ll get us started. I whipped up I Love Mercurial, a site where you can talk until you pass out about why Mercurial is awesome. Just tweet with the #ilovemercurial hashtag, and we’ll pick it up and post it.

Note: why you love Mercurial. Not why Mercurial is better than X. Not why X causes brain tumors in lab rats and sterilizes your children. Just, things that Mercurial does that you love? Talk about ’em on Twitter with the #ilovemercurial hashtag. We’ll post ’em. Then, the next time, someone asks you why you like Mercurial, just point them to that site. Or the hashtag. I honestly don’t care. Point is, show them why Mercurial rocks, and not why the other guy sucks.

And then maybe, just maybe, just maybe, at least on this one little topic, the flamewar can die in a pile of love on why my tool is awesome instead of the 92835 deficiencies your tool has that mine doesn’t, and I don’t have to wake up to my alarm clock playing I Got You Babe ever, ever again.

Please remember to update manners to tip before tweeting.

Cache me if you can

2011-06-10T12:03:12Z

I remember, when we first launched Kiln, that I was desperately hopeful that it’d actually make a dent. I was honestly quite scared that we might end up in a situation where, after we’d put in all of this effort, spent all of this time designing and coding and testing and writing really arcane and annoying crap like billing code, that no one would want Kiln, and we’d be forced to close shop.

Good news! That’s no longer the problem.

The problem now is that we’re too successful.

What?! I hear some of you ask. Don’t you want to be too successful?!

I think I speak for my entire team when I say: hell yes! We want to keep on being too successful. If anything, we want to be even more too successful than we currently are.

But success does present problems. It’s awesome to have thousands of customers and terabytes of data, but then you start dealing with the boring details of questions like where do you put those terabytes of data, and how do you get that data to the customers in a timely manner. And that can be a really tough cookie to crack.

I’m going to be talking about several different aspects of how we’re handling scaling Kiln over the next few weeks, but today, I want to focus on one single narrow thing: caching.

The WISC Stack

The main part of Kiln that you all know and love—the website—is a fairly typical WISC application. We have a couple of web servers running IIS and the Kiln website, which talk to several SQL Server boxes. The nice thing about well-engineered WISC stacks is that, like LAMP, you can scale the databases and the web servers independently. This is the bread-and-butter of designing a scalable application. So far, so good.

The thing is, just adding more SQL boxes isn’t always the answer. If you have complex queries that take a long time to run, then adding another box won’t help anything. It just gives you another box to run your complex query on at the same slow speed. Even if you’re only doing simple queries, adding more database boxes isn’t necessarily the answer. Good SQL boxes are expensive—doubly so if you’re using a big commercial DB package such as SQL Server or Oracle. While you might be able to afford buy more, you don’t want to if you can avoid it.

Instead, you should focus on just not hitting the database in the first place.

The S in WISC

It turns out that there are already some mechanisms we had in place to help with this. We prefetched certain data that we knew we needed nearly every request (like the full list of repositories), and then used that cache for any other lookups during the request. And LINQ to SQL does a bit of its own per-request object caching in certain situations (such as querying an entity by primary key), so we already had some actual data caching going on.

While that kind of stuff can help, what we really wanted to do was to try to avoid talking to SQL at all for common operations. Those complex queries that Kiln does—things like showing the amalgamated DAG for all related repositories—take a long time to run, but the resulting data doesn’t actually change that often. This is a clear and wonderful win, if we can pull it off.

Making it Happen

There were two problems we had to solve: where do you cache the data? and how do you get it there?

The first one was pretty easy: use Memcache. Memcache is used everywhere, is extremely well-understood, and has great implementations for both Windows and Linux. We ended up settling on Membase as the particular server implementation we liked, due to its easy-to-use administration console and trivial clustering support. For the client, we ended up going with the BeIT Memcached library, due to its easy customizability and simple design.

The second part was much more difficult. Kiln uses LINQ to SQL for its database access layer. That meant we had a problem: LINQ to SQL is a very complex beast, where objects have a database context that in turn is aware of all the objects that it’s managing. If you just grab a random LINQ object and throw it into Memcache, then it is not going to deserialize cleanly. Throw in that we have piles of custom logic in our LINQ-to-SQL-backed models, and you’ve got a recipe for pain.

We ended up solving this in two different ways:

We modified our models to allow for detaching and reattaching to the database context. Before serialization, the object is detached, so it has no controlling database context. On deserialization, we attach it to the current context. This isn’t as fast as grabbing an attached object out of a cache (such as the old per-request prefetch cache mentioned earlier), but ends up incurring minimal overhead for the common case.
We also had to modify our models to know that they might not have come from the DB. We rely heavily on signal handlers to make changes in a given model class propagate to all the parts of Kiln that need to be notified. These were firing erroneously as the deserialization code set piles of properties. The fix we came up with was to suppress signals for deserializing objects—which, since most of our model modifications are done by T4 templates anyway, was very easy to do in a DRY manner.

With these two changes, we were able to reliably store LINQ database entities in Memcache, get them back out, and work with them.

It was easy enough to verify that the number of queries was down, but would the caching code make a real difference?

The Result

I think this graph of what load looks like on one of our DB boxes, before and after the caching deployment, says more than I could in several paragraphs of text:

We’ve cut the amount of data we’re getting from SQL by 75%.

The benefits we’re seeing are already impressive. We have faster load times and less DB traffic. But we can still do a lot more: now that we have the outlines of a caching framework, we can continue to profile and to move more of our expensive queries into the cache. Based on what we’ve seen so far, this should yield immediate and real benefits to our Kiln On Demand customers.

Making Your Interns Addicts: a How-To Guide

2011-05-27T08:22:15Z

I was thinking back last week on why I started working at Fog Creek. If you don’t know, I got started on this thing called Project Aardvark, which eventually ended up becoming Copilot, the project I worked on for my first couple of years at Fog Creek. I don’t generally reminisce much about that time, simply because that was a very different point in my life, back before I found fashion, yet after I figured out how to end up in front of cameras constantly.

So my friends are usually quite good at reminiscing about that time on my behalf, which is plenty. But with this summer’s batch of new Fog Creek interns arriving in just over a week, I’ve been thinking about it a lot. What was good, what was bad, what about the experience made me need to stay at Fog Creek.

I’ve realized there are quite a few parallels between what got me into Fog Creek and what gets people hooked on hard-core drugs.

Not in the bad way, mind you. I have yet to see an intern maniacally chase source code into a dirty toilet, for example.¹ What I mean is, I almost feel like our internship is custom-tailored to pound the crap out our interns’ sense of awesome so that they just have to come back for more. Just, unlike with drugs, the effect is that they have a great summer, and they want to work for us, and I honestly feel they end up being way more productive in the process. What I find remarkable is that while nearly every part of Fog Creek has changed since my initial internship, the actual way we run our internships continues almost unedited.

I really want all the interns in the world to come work for us, but I also want all the interns in the world to have amazing summers, and the best way to do that is to tell you what we’re doing to get our interns hooked. So without further ado, I’m happy to begin my first and last guide to deliberately getting someone addicted.

Step 1: Get Them Excited

The first thing that matters is getting them psyched just to be there. When I was an Aardvark, that was easy: we knew we were going to be in a movie. Being in a movie is exciting. It’s even more exciting when the film-maker comes to your house before the summer starts just to interview you and film you doing a really botched version of a Chopin nocturne that he manages to edit so you sound competent. I mean, who could say no to that?

Our interns this year may not be excited because they’ll be in a movie, but they can be excited for a different reason: they’re going to be working on a young project that still has thousands of customers and manages over half a terabyte of data across nearly 25,000 active repositories. They’ll get a chance to be making a difference.

Find something that you yourself are excited about, and get your interns excited in that. Or, if you don’t have anything at your job that excites you, find another job. We’re hiring, after all.

Step 2: Get Them Doing Real Work

Of course, if the thing you’re getting your interns excited about is playing with a widely used application, it does require you let them, you know, touch the code. My first coding internship, at a company that shall remain nameless,² involved me writing a tremendous amount of Ruby code. That could be awesome, and in a way, it was. Except that the Ruby code I was writing were scripts to do things like automatically upload new advertisement photos to the intranet, collated by the date the photograph was taken. Or, when I wasn’t so lucky, it was to make a blog engine that could hold exactly one blog post, and then display a little red icon if the blog post it was showing at the time was Bad For The Company™. Not exactly central to the product, you know?

So at Fog Creek, we always make sure our interns are doing real stuff for the project they’re working on. When I was an intern on Aardvark, I wrote most of the Windows helpers. Other intern classes have written a new wiki for FogBugz, and given Kiln its API and its large files support. Hell, an intern class was integral to getting the first version of Kiln done in the first place. It feels so much better to actually be contributing code to something you know is going to make a real difference to the product.

Or, you know, have them write a one-post blog engine or whatever. It’s cool. Just make sure they know at the end of their internship that we’re still hiring.

Step 3: Give Them Instant Feedback

You know what sucks about automating some random task that you’ve been assigned to do as an intern? Getting absolutely no feedback on what anyone thinks about it. Is it useful? Does it stink? Who knows. It’s a fricking compilation of advertisements on a corporate intranet. For all I know, no one even reads this at all.³

With one exception,⁴ Fog Creek interns always get to see Creekers using their code immediately, because we code right alongside them. We’re running their changes on our development systems, and, in many cases, running that code on our dogfood systems just a day or two after it’s written. Not only does this generally mean that our products are better, because we’re using new features heavily ourselves long before they ever reach our customer’s hands; it means that our interns are getting feedback almost immediately, both from within and outside their team, on what worked and what didn’t.

Or you can send feedback three weeks after the internship ends. By email. That works, too.

Step 4: Make Them Feel Awesome

People like feeling good. And the simple fact is, if you’re using your interns’ code almost immediately, and giving them rapid feedback, you’re going to end up having your interns produce really great stuff. So: tell them that.

Not in a fake way, mind you: some things just honestly take a lot of revisions before they’re good, no matter how awesome the people involved are, and your interns are going to know you’re offering them bull if you tell them something’s awesome that they know isn’t. But there’s all those intermediate steps, the little victories, the occasional big one, where you owe it to your team to acknowledge that, yup, they did well.

Saying “good job” is one way to show you care, but actually getting to know your interns is even better. At Fog Creek, we hang out with our interns outside of work by taking them to documentaries like Harry Potter 7, to Yankees “games”⁵ at the new stadium, and, my personal favorite, to top-shelf-martini-powered bowling. If you don’t have these available in your town, take them to the airport so they can bond over a TSA freaky frisk-fest. The point is, do something with your interns so you actually get to know and like them as something other than a coding automaton.

Step 5: Ship, dammit!

But all of that comes to naught if you don’t actually ship stuff. Sure, maybe you worked on Kiln, and sure, maybe Kiln makes a difference to the company, and sure, maybe your boss likes you. But what about your own stuff? Do customers care? Does it just go into a bit bucket at the end of the summer, never to be seen again?

Our interns won’t have to wonder too much about that: thanks to Kiln’s fast release schedule, they’ll be seeing their code go out to customers multiple times this summer, get feedback from customers about their changes, and probably even do a second round or two that integrates that feedback. So while they’re busy being excited, writing production code, getting feedback, and doing fun stuff, they’ll also be knowing, even during their internship, exactly what a difference they’re making.

You see what I mean? It’s exactly like an addiction. Just because our interns are awesome, work on great products, and make a real difference, they get this crazy positive feedback loop that means they not only have a great summer, but that they want to keep coming back, again and again. We’ve literally had freshman interns who come back every single summer because they love it here so much. We’re doing something right.

And I hope that, if your interns aren’t already feeling the same way, this guide can help you get there. I remember what it was like to feel like an undervalued intern, and I remember what it was like to feel like I was king of the world.

Help your interns have their own king-of-the-world moments, and everyone wins.

I hate Trainspotting so hard. ↩
The East India Company. ↩
Actually, I know people read it, because I got bored and figured out how to use “DHTML” to log who was viewing the page. But you get the point. ↩
It turns out that, try as we might, even Fog Creek can’t achieve perfection. Just reallygoodtion, with an eye towards learningfromyourmistakesery. ↩
My editor informed me that calling them “Red Sox beat-downs” was not appropriate if I didn’t want to offend my audience. ↩

Why how is boring and how why is awesome

2011-04-28T11:48:18Z

Last fall, Joel came to me and said, “Congratulations! We’re doing another World Tour. Also, we want to teach distributed version control. That’s your job. Make it happen.”

This sounded totally awesome. Not only would I get to one-up George Clooney in flight time; I was made for doing something like this. In high school, I was in the NFL, which, sadly, means the National Forensics League, which means the National People Who Talk Good and Wanna Learn To Do Other Stuff Good Too, and not the National Football League. My specialty was original oratory, where you give an original speech you’ve rehearsed ahead of time. Add in there that I love to teach, and that I’m the one who introduced Fog Creek to Mercurial in the first place, and I should be the perfect person to do this.

I of course said yes, went back to my office, and fired up Emacs, ready to go!

And then promptly suffered through two weeks of intermittent writer’s block.

Here’s the problem: yes, I love using distributed version control. Yes, I love teaching. And yes, I believe I have a mission to teach people about DVCS. But the talk I had to give, by definition, was an introduction to the basics of distributed version control.

I’ve been using distributed version control for over half a decade, first as arch and Monticello, then Darcs, then super-early versions of Mercurial. The basics of how do you do this? were super old-hat to me, and, well, boring. There was nothing exciting to me about how to commit a file, or do an elementary merge, but I had to cover that material.

Further, if you only cover the basics, you can’t possibly demonstrate why you want to use a DVCS. There are minor benefits you can note—hey, you can work offline!, or: hey, things are actually fast now!—but you don’t really get to any of the actual meat, the things that teach you why do you want to make this transition.

After two weeks of running in circles, I sat down with my friend Jason (who now works for the Khan Academy, named after Genghis Khan, who invaded the Enterprise in Star Wars II to teach Captain Kirk calculus), and asked him how to get out of this mess. Over half an hour of conversation, I realized that I already had the answer.

Yes, I needed to cover the basics. No, those would not be super-interesting. But I could keep that as short as humanly possible, and as soon as I’d done that, I could spend the rest of the talk discussing why you should care. We could cover workflows that it took lots of trial and error to discover. We could show merge scenarios that caused traditional tools to barf. We could show how Mercurial made it really easy for us to ship lots of versions of FogBugz without dropping bug fixes. In other words: spend half the talk with the basics of basics, just enough to understand how things worked, but then jump straight ahead to answering the question of what did distributed version control ever do for you.

I went back to Emacs, and quickly identified what I thought were two of DVCS’ killer scenarios: maintaining multiple versions of a product without dropping bug fixes, and being able to reliably develop new features of a product while always being ready to ship a bugfix in a heartbeat. I built the talk so that the audience would start with nothing, and build up to understanding how to realize those two workflows.

The result? Distributed Version Control System University, or: how do I actually use distributed version control in a meaningful way.

I’m actually really happy with how everything turned out. Even though the code I ended up using was silly, audiences responded well, and understood the potential power and flexibility that distributed version control provides. They asked intelligent questions, they were engaged, and a lot of them signed up for Kiln trial accounts and decided to start playing with Mercurial.

One of my goals in the future is to try to beef up hg init with some of the workflow examples in my talk. I also want to take some time to go into more detail about how we do workflow here at Fog Creek. But if you or a coworker has been wondering what the big deal is about distributed version control, watch the video. I think it provides a great overview in a way that any dev can immediately relate to and get excited about.

Finding Literate Programmers

2011-04-14T08:39:17Z

So we’re making candidates nervous by using C, and we’re not gaining anything by using C.

But are we losing anything by using C?

Yes. We’re losing any ability to test for literacy.

“Smart and gets things done” is a famous mantra, and one that I agree with and believe has served us very well. But even at Fog Creek, we don’t go entirely by “smart and gets things done” when it comes to developers. If we did, we’d happily hire anthropology majors who had gotten a few papers published. After all, they’re smart and get things done, right? But we care, at a bare minimum, about one other thing, which is passion for software development. Intern candidates especially may not be the best coders we’ve ever seen, but we expect them to demonstrate at least some proficiency in basic programming tasks in addition to being smart and getting things done. The threshold may not be super-high, but it’s there.

What does this have to do with interviewing in C? Software development, like a few other fields such as mathematics, English, and art, is something you can practice as a hobby even at a very young age. The best coders have to practically fight not to code in their spare time. My girlfriend routinely asks me to get the hell off the computer, I am not kidding, why are you still coding even though work’s over, dammit why does this computer keep working even after I’ve deliberately poured a bucket of water on it, what do you mean you can waterproof things like this because you said you couldn’t when it was my camera, I am leaving you and taking the cats with me, and so on. You pretty much have to pry the machine away from our fingers.

So even young coders, coders with very little experience who make piles of “dumb” errors, have generally written at least a smattering of little hobby projects. They may not be that interesting, and they may be duplicating something that already exists, but they’re there.

As they program more, something magical happens: they start to really get a language. They start to learn the common parts of its libraries, the common idioms used when writing in it. It’s like learning to write English well: you start with the basics, but then begin building an ever-better lexicon as you need to make more nuanced distinctions between concepts. You begin making allusions to other works you’ve read that made an impression. Your writing just becomes more refined, and, eventually, good. Part of your oeuvre.

That’s what I meant by “we aren’t testing for literacy.” If you’re trying to solve problems, especially if you’re a new programmer, you almost have to pick up at least the basics of the library of the language you’re working in. Especially if you’re a college kid, because, let’s face it, college kids are lazy. Any line you don’t have to write is one you can’t have points deducted for, so you’ll lean on that standard library as heavily as you possibly can. You build up a vocabulary of functions and classes and methods that allow you to get more done with less code. You become literate in the language.

If I let you program in the language of your choice, you have an opportunity to demonstrate that literacy. And, because the better you understand the language, the more you can accomplish in a short period of time, the more interesting and complex my interview question can be, too. No longer am I restricted to abstract data-types, like I am in C or some hypothetical Minimal Language You Learn For The Interview. Suddenly, asking a student to sketch an outline of a 6502 emulator becomes a totally legitimate request for a 60-minute session.

Let me take a break here and make clear what I am not advocating: I do not think candidates should have to know their language libraries inside and out. Someone really enthusiastic about programming, but who’s only just getting started, may not understand strcmp even if they’re destined to hack on the Linux kernel for fun, and the next _why the lucky stiff may not yet understand how to make a method take a block. You have to recognize that you’ve got a diamond in the rough there, and evaluate them in other ways.

But for those few who come in and who are super-literate at a language? It’s so fun to do those interviews all of a sudden! I get a chance to be totally blown away by how well you know your craft, you have a chance to be totally creative, and we both get out of the yet-another-data-structure interview question hell (or its lovely variant, how-do-I-get-three-beings-across-a-bridge-in-a-limited-amount-of-time-when-they-all-want-to-eat-and/or-kill-one-another question). Forcing everyone into a language they’re equally (un)familiar with may be “fair,” but it also needlessly penalizes some of the best craftsmen you’ll see.

I’m not abandoning “smart and gets things done” for “smart and gets things done and writes goodish.” But I’d like to abandon it for “smart and gets things done, and is optionally simply awesome.” It’s hard to do that if you restrict your interviews to a single language that most candidates won’t even have used before.

But if you let everyone use their native tongue?

You might be surprised what you’ll find.

Drowning in a C of Interviews

2011-04-13T08:12:59Z

The Guerrilla Guide. It is the be-all, end-all of how to look for candidates.

Except for one paragraph.

And I’ve come to realize that I agree with that paragraph; just not Fog Creek’s implementation of it.

In The Paragraph In Question, Joel writes:

I’ve come to realize that understanding pointers in C is not a skill, it’s an aptitude.

The paragraph goes on. But for Fog Creek, it stops right there. In fact, it stops even earlier. The version that Fog Creek has implemented, in practice, reads:

I’ve come to realize that understanding C is not a skill, it’s an aptitude.

Since literally before I’ve joined, Fog Creek has conducted all coding interviews in C. Period. If you want to work at Fog Creek writing nothing but pure JavaScript apps using Node.js, you will do your interview in C, and you will like it. In this one little sense, I became a heretic a few months ago. I quit giving my interviews in C.

And you know what?

Nothing changed.

Well, that’s not true. Candidates were a lot more relaxed, and asked a lot fewer nervous questions about how strict I was going to be about syntax. But my confidence that I could tell whether they would be good programmers, and more importantly, my acceptance rate, didn’t change whatsoever. I certainly didn’t feel like I was losing any data by not conducting my interview in C.

How is that possible? Let’s read through what Joel actually wrote after that introductory sentence:

In first year computer science classes, there are always about 200 kids at the beginning of the semester, all of whom wrote complex adventure games in BASIC for their PCs when they were 4 years old. They are having a good ol’ time learning C or Pascal in college, until one day the professor introduces pointers, and suddenly, they don’t get it. They just don’t understand anything any more. 90% of the class goes off and becomes Political Science majors, then they tell their friends that there weren’t enough good looking members of the appropriate sex in their CompSci classes, that’s why they switched. For some reason most people seem to be born without the part of the brain that understands pointers. Pointers require a complex form of doubly-indirected thinking that some people just can’t do, and it’s pretty crucial to good programming. A lot of the “script jocks” who started programming by copying JavaScript snippets into their web pages and went on to learn Perl never learned about pointers, and they can never quite produce code of the quality you need.

If you read the whole thing, you realize that Joel’s not advocating the idea that a candidate must know C; rather, he’s advocating the idea that C forces you to understand concepts that you may not otherwise understand.

Inventing questions that force candidates to understand pointers without using C isn’t too hard. Nearly any question that forces candidates to invent a data structure (e.g., a hashtable, an AVL tree, or the like) will test how they handle indirection, the idea that having a thing is different from having a pointer to that thing. So I’ve picked a question that forces candidates to design a data structure. And, sure enough, I see candidates who have a lot of programming experience, but who don’t “get it”, completely bomb out in my interview.

I think there’s a tremendous amount of value in Joel’s original point. Understanding indirection, and pointers in particular, seems to be innate. But doing interviews in C doesn’t really test for that; it tests for the candidates’ understanding of C. And I think, in our dogged pursuit of that, we’ve probably erred. We’re making candidates nervous about how well they know C, and focusing too much on how well they understand C’s semantics, but not really getting any meaningful indicators back in return.

For the moment, I’ll continue conducting my interviews in the language of the candidate’s choice as an experiment, but I think it’s inevitable that the Wall of C will fall.