bitquabit - Programming

Why how is boring and how why is awesome

2011-04-28T11:48:18Z

Last fall, Joel came to me and said, “Congratulations! We’re doing another World Tour. Also, we want to teach distributed version control. That’s your job. Make it happen.”

This sounded totally awesome. Not only would I get to one-up George Clooney in flight time; I was made for doing something like this. In high school, I was in the NFL, which, sadly, means the National Forensics League, which means the National People Who Talk Good and Wanna Learn To Do Other Stuff Good Too, and not the National Football League. My specialty was original oratory, where you give an original speech you’ve rehearsed ahead of time. Add in there that I love to teach, and that I’m the one who introduced Fog Creek to Mercurial in the first place, and I should be the perfect person to do this.

I of course said yes, went back to my office, and fired up Emacs, ready to go!

And then promptly suffered through two weeks of intermittent writer’s block.

Here’s the problem: yes, I love using distributed version control. Yes, I love teaching. And yes, I believe I have a mission to teach people about DVCS. But the talk I had to give, by definition, was an introduction to the basics of distributed version control.

I’ve been using distributed version control for over half a decade, first as arch and Monticello, then Darcs, then super-early versions of Mercurial. The basics of how do you do this? were super old-hat to me, and, well, boring. There was nothing exciting to me about how to commit a file, or do an elementary merge, but I had to cover that material.

Further, if you only cover the basics, you can’t possibly demonstrate why you want to use a DVCS. There are minor benefits you can note—hey, you can work offline!, or: hey, things are actually fast now!—but you don’t really get to any of the actual meat, the things that teach you why do you want to make this transition.

After two weeks of running in circles, I sat down with my friend Jason (who now works for the Khan Academy, named after Genghis Khan, who invaded the Enterprise in Star Wars II to teach Captain Kirk calculus), and asked him how to get out of this mess. Over half an hour of conversation, I realized that I already had the answer.

Yes, I needed to cover the basics. No, those would not be super-interesting. But I could keep that as short as humanly possible, and as soon as I’d done that, I could spend the rest of the talk discussing why you should care. We could cover workflows that it took lots of trial and error to discover. We could show merge scenarios that caused traditional tools to barf. We could show how Mercurial made it really easy for us to ship lots of versions of FogBugz without dropping bug fixes. In other words: spend half the talk with the basics of basics, just enough to understand how things worked, but then jump straight ahead to answering the question of what did distributed version control ever do for you.

I went back to Emacs, and quickly identified what I thought were two of DVCS’ killer scenarios: maintaining multiple versions of a product without dropping bug fixes, and being able to reliably develop new features of a product while always being ready to ship a bugfix in a heartbeat. I built the talk so that the audience would start with nothing, and build up to understanding how to realize those two workflows.

The result? Distributed Version Control System University, or: how do I actually use distributed version control in a meaningful way.

I’m actually really happy with how everything turned out. Even though the code I ended up using was silly, audiences responded well, and understood the potential power and flexibility that distributed version control provides. They asked intelligent questions, they were engaged, and a lot of them signed up for Kiln trial accounts and decided to start playing with Mercurial.

One of my goals in the future is to try to beef up hg init with some of the workflow examples in my talk. I also want to take some time to go into more detail about how we do workflow here at Fog Creek. But if you or a coworker has been wondering what the big deal is about distributed version control, watch the video. I think it provides a great overview in a way that any dev can immediately relate to and get excited about.

Finding Literate Programmers

2011-04-14T08:39:17Z

So we’re making candidates nervous by using C, and we’re not gaining anything by using C.

But are we losing anything by using C?

Yes. We’re losing any ability to test for literacy.

“Smart and gets things done” is a famous mantra, and one that I agree with and believe has served us very well. But even at Fog Creek, we don’t go entirely by “smart and gets things done” when it comes to developers. If we did, we’d happily hire anthropology majors who had gotten a few papers published. After all, they’re smart and get things done, right? But we care, at a bare minimum, about one other thing, which is passion for software development. Intern candidates especially may not be the best coders we’ve ever seen, but we expect them to demonstrate at least some proficiency in basic programming tasks in addition to being smart and getting things done. The threshold may not be super-high, but it’s there.

What does this have to do with interviewing in C? Software development, like a few other fields such as mathematics, English, and art, is something you can practice as a hobby even at a very young age. The best coders have to practically fight not to code in their spare time. My girlfriend routinely asks me to get the hell off the computer, I am not kidding, why are you still coding even though work’s over, dammit why does this computer keep working even after I’ve deliberately poured a bucket of water on it, what do you mean you can waterproof things like this because you said you couldn’t when it was my camera, I am leaving you and taking the cats with me, and so on. You pretty much have to pry the machine away from our fingers.

So even young coders, coders with very little experience who make piles of “dumb” errors, have generally written at least a smattering of little hobby projects. They may not be that interesting, and they may be duplicating something that already exists, but they’re there.

As they program more, something magical happens: they start to really get a language. They start to learn the common parts of its libraries, the common idioms used when writing in it. It’s like learning to write English well: you start with the basics, but then begin building an ever-better lexicon as you need to make more nuanced distinctions between concepts. You begin making allusions to other works you’ve read that made an impression. Your writing just becomes more refined, and, eventually, good. Part of your oeuvre.

That’s what I meant by “we aren’t testing for literacy.” If you’re trying to solve problems, especially if you’re a new programmer, you almost have to pick up at least the basics of the library of the language you’re working in. Especially if you’re a college kid, because, let’s face it, college kids are lazy. Any line you don’t have to write is one you can’t have points deducted for, so you’ll lean on that standard library as heavily as you possibly can. You build up a vocabulary of functions and classes and methods that allow you to get more done with less code. You become literate in the language.

If I let you program in the language of your choice, you have an opportunity to demonstrate that literacy. And, because the better you understand the language, the more you can accomplish in a short period of time, the more interesting and complex my interview question can be, too. No longer am I restricted to abstract data-types, like I am in C or some hypothetical Minimal Language You Learn For The Interview. Suddenly, asking a student to sketch an outline of a 6502 emulator becomes a totally legitimate request for a 60-minute session.

Let me take a break here and make clear what I am not advocating: I do not think candidates should have to know their language libraries inside and out. Someone really enthusiastic about programming, but who’s only just getting started, may not understand strcmp even if they’re destined to hack on the Linux kernel for fun, and the next _why the lucky stiff may not yet understand how to make a method take a block. You have to recognize that you’ve got a diamond in the rough there, and evaluate them in other ways.

But for those few who come in and who are super-literate at a language? It’s so fun to do those interviews all of a sudden! I get a chance to be totally blown away by how well you know your craft, you have a chance to be totally creative, and we both get out of the yet-another-data-structure interview question hell (or its lovely variant, how-do-I-get-three-beings-across-a-bridge-in-a-limited-amount-of-time-when-they-all-want-to-eat-and/or-kill-one-another question). Forcing everyone into a language they’re equally (un)familiar with may be “fair,” but it also needlessly penalizes some of the best craftsmen you’ll see.

I’m not abandoning “smart and gets things done” for “smart and gets things done and writes goodish.” But I’d like to abandon it for “smart and gets things done, and is optionally simply awesome.” It’s hard to do that if you restrict your interviews to a single language that most candidates won’t even have used before.

But if you let everyone use their native tongue?

You might be surprised what you’ll find.

Drowning in a C of Interviews

2011-04-13T08:12:59Z

The Guerrilla Guide. It is the be-all, end-all of how to look for candidates.

Except for one paragraph.

And I’ve come to realize that I agree with that paragraph; just not Fog Creek’s implementation of it.

In The Paragraph In Question, Joel writes:

I’ve come to realize that understanding pointers in C is not a skill, it’s an aptitude.

The paragraph goes on. But for Fog Creek, it stops right there. In fact, it stops even earlier. The version that Fog Creek has implemented, in practice, reads:

I’ve come to realize that understanding C is not a skill, it’s an aptitude.

Since literally before I’ve joined, Fog Creek has conducted all coding interviews in C. Period. If you want to work at Fog Creek writing nothing but pure JavaScript apps using Node.js, you will do your interview in C, and you will like it. In this one little sense, I became a heretic a few months ago. I quit giving my interviews in C.

And you know what?

Nothing changed.

Well, that’s not true. Candidates were a lot more relaxed, and asked a lot fewer nervous questions about how strict I was going to be about syntax. But my confidence that I could tell whether they would be good programmers, and more importantly, my acceptance rate, didn’t change whatsoever. I certainly didn’t feel like I was losing any data by not conducting my interview in C.

How is that possible? Let’s read through what Joel actually wrote after that introductory sentence:

In first year computer science classes, there are always about 200 kids at the beginning of the semester, all of whom wrote complex adventure games in BASIC for their PCs when they were 4 years old. They are having a good ol’ time learning C or Pascal in college, until one day the professor introduces pointers, and suddenly, they don’t get it. They just don’t understand anything any more. 90% of the class goes off and becomes Political Science majors, then they tell their friends that there weren’t enough good looking members of the appropriate sex in their CompSci classes, that’s why they switched. For some reason most people seem to be born without the part of the brain that understands pointers. Pointers require a complex form of doubly-indirected thinking that some people just can’t do, and it’s pretty crucial to good programming. A lot of the “script jocks” who started programming by copying JavaScript snippets into their web pages and went on to learn Perl never learned about pointers, and they can never quite produce code of the quality you need.

If you read the whole thing, you realize that Joel’s not advocating the idea that a candidate must know C; rather, he’s advocating the idea that C forces you to understand concepts that you may not otherwise understand.

Inventing questions that force candidates to understand pointers without using C isn’t too hard. Nearly any question that forces candidates to invent a data structure (e.g., a hashtable, an AVL tree, or the like) will test how they handle indirection, the idea that having a thing is different from having a pointer to that thing. So I’ve picked a question that forces candidates to design a data structure. And, sure enough, I see candidates who have a lot of programming experience, but who don’t “get it”, completely bomb out in my interview.

I think there’s a tremendous amount of value in Joel’s original point. Understanding indirection, and pointers in particular, seems to be innate. But doing interviews in C doesn’t really test for that; it tests for the candidates’ understanding of C. And I think, in our dogged pursuit of that, we’ve probably erred. We’re making candidates nervous about how well they know C, and focusing too much on how well they understand C’s semantics, but not really getting any meaningful indicators back in return.

For the moment, I’ll continue conducting my interviews in the language of the candidate’s choice as an experiment, but I think it’s inevitable that the Wall of C will fall.

Supporting the Giants

2011-04-05T08:32:00Z

One of the things that I love about Fog Creek is that we give back. Kiln can only exist because of the amazing foundation provided by the Mercurial distributed version control system, so we try to help them out whenever we can. In the past, we’ve done that by making Fog Creek one of the top Mercurial sponsors. Given how small Fog Creek is, I can’t tell you how proud I am to see that we’re placing up there amongst Google and Microsoft for supporting open-source software.

This year, we’ve decided to help contribute in an even more hands-on way: by devoting developer resources. Not only will Fog Creek be helping to sponsor the upcoming Mercurial 1.9 code sprint; we’re also sending one of our awesome Kiln developers, Kevin Gessner, to help contribute. And, in the weeks leading up to the sprint, we’ll be using Kiln team developers to help contribute bug fixes and features to the Mercurial project. In other words, we’ll be supporting Mercurial both financially and with developer resources.

Kiln stands on the shoulders of giants, so I’m happy to support those giants in return.

After all, the higher the giant gets, the taller Kiln stands.

Have a mission

2011-03-14T08:23:36Z

You know why I love working on Kiln every day?

Because we’ve got a mission, dammit.

Mission, not mission statement. The Kiln team doesn’t have a mission statement, and I’ll fight to keep it that way. Mission statements, no matter how well intentioned, become these trite little soundbites that you parrot indefinitely until they just become a meaningless jumble of syllables—kind of like what happens if you just say the word “marmalade” about 40 times in a row.

But that’s totally different from having a mission. Everyone who works on Kiln knows what we’re trying to accomplish. They’ll phrase it in their own way, but no matter who you talk to on my team, we’ll all tell you why Kiln exists: we believe that distributed version control is so much more awesome than old-school centralized crap that we have an obligation to bring that awesomeness to as many developers as possible. Every time I write a line of code, I have in my head that I’m trying to further that goal.

Distributed version control is having trouble making inroads at companies, you say? Fine. We’ve make Kiln an awesome system for companies: we’ve made a kickass project-centric workflow, given teams a way of handling large numbers of repositories, added asset management, permissions models, code reviews, rich APIs, and piles more. People having trouble leaving their old system behind? No problem: we’ll draft Joel into writing awesome tutorials, we’ll give talks on distributed version control all over the world, we’ll provide training services to help bring people up to speed, and we’ll make an importer to help you get your existing code moved over. We’ll even create a StackExchange just for helping people figure out Mercurial, Kiln, and distributed version control, so that there’s a good, central location to get answers about tools, workflows, and everything else. These things probably all help Kiln, but we’re not doing them because of that; we’re doing them because we have to do them to achieve what we believe we need to do. The rest is a footnote.

I feel like I can tell you any good product’s mission really easily for similar reasons. iPhone? Make using a smartphone fun. The App Store, iBooks, and so on are just fall-out.

Kindle? Make digital books completely pervasive. And you know that’s the goal, because you can read Kindle books on practically any device in wonderful, intuitive applications.

Google? Make it possible to find good, relevant data, quickly.

Facebook? Keep people from losing touch.

There’s a common theme here: in all of these cases, the mission isn’t about the product as such. The mission is “just” something that, if you’re really serious about, requires that the product become absolutely amazing, so you make that happen. If you prefer: these missions are all the answer to, why did we just add that new feature?, not, what new feature are we adding?

What the hell is Windows’ mission in life? The best I can come up with, at this point, is to keep Windows from getting too dated, but not breaking backwards compatibility, so that Microsoft can make money while they find something that actually sticks. That’s…pretty pathetic. Not to single out Microsoft here; I’m not really convinced the OS X team has a terribly coherent mission right now, either. They have a clear goal—they want to bring the experience of the iPhone and iPad to OS X—but I don’t quite grok why they’re doing it, and I would argue that the somewhat haphazard mishmash of app UIs in the Lion previews (compare Address Book with the new Mail, for example) is a pretty good indication that they don’t quite know, either.

When you don’t really know why you’re doing something—when it’s just a goal that you’ve got and you’ve lost track of your original motivation—it’s really tough to have a good product, or to be excited when you come into work every day. Yes, someone has to fix Windows. But why me? Why can’t I be making something awesome? So you’ll do the minimal amount necessary to achieve your best understanding of the goal, rather than everything required to maximize your chance of actually achieving your mission in life.

Don’t just latch onto something because it’s neat. Latch onto it because you really fucking believe that you have something to accomplish here, and since no one else is going to do it, it’s going to have to be you. That’s your mission.

Make it happen.

When things go well

2011-03-10T11:07:13Z

It was a lot of fun to vet dirty laundry in my last post on how one of our deployments went really wrong. But part of why that incident stuck out so strongly in my mind is that things so rarely go wrong.

Why is that? I think it’s because we do a lot right: we make it extremely easy for ourselves to keep features from going out to customers until we’re ready, and we give ourselves a lot of time to bang on the exact same version of the software, on the exact servers, that we’ll be pushing out to all of our customers.

How do we do that? Let’s say that I decide that Kiln totally needs an awesome WebGL flyover view of the Electric DAG. I code it up over lunch in about 15 minutes. Now what?

First, as bad-ass as my new feature is, I recognize that (unlikely as this is) I just maybe have a bug or two. So rather than pushing it directly to all my other teammates, I push it to a feature branch called flitesim or the like. If the feature totally stinks (who the feh doesn’t like flyovers? I mean come on!), it’ll never make it out of the branch, and never end up in the main code base. No one except devs working directly on the feature need to care about it at this point.
I don’t know OpenGL from an ice cream sundae and coded the feature up mostly via NeHe copypasta, so next up, I’ll use Kiln’s code reviews to ask my team members to review what I wrote. They’ll look for XSS exploits, check for sane coding conventions, check for possible performance bottlenecks or threading errors, and ask me what on Earth I was drinking when I invented this feature. Minus the last bit, it’s kind of like having a friend read over your essay before you hand it in. (Actually, with the last bit, it’s still like having a friend read over your essay before you hand it in.)
At the same time this is going on, I’ll use a feature of our continuous integration system, Mortar, to cut builds off the flitesim branch for the QA team. About five minutes later, they get a licensed (i.e., install-on-your-local-machine) version of Kiln with my spiffy flight sim feature that they can bludgeon the crap out of, develop tests for, and so on. Because they’re licensed copies, they can install on VMware snapshots, which makes diagnosing schema migration issues and the like ridiculously trivial, and nearly guarantees we can get good repros when things go wrong.
At some point, my team approves the code reviews (I hope), and QA agrees the feature is working to specification (weird specification though that may be). Kiln will be getting its own little DAG-oriented flight sim! Take that, $COMPETITOR! So at this point, I merge it into our devel branch, which we use for integration testing. Features merged into devel are destined for eventual deployment, and the devel branch is the one that all developers grab at the beginning of the day to base their work on. So as of now, the rest of my team will see my badassery, and begin playing with it and fixing bugs as they notice them.
On Tuesday morning, provided that the developers feel good about everything that’s currently in the devel branch, we merge devel into dogfood, and, again using Mortar, deploy this version of Kiln to our own servers. At this point, the entire company is hitting on Kiln and flying all over our source code in amazing 3D-accelerated goodness, and probably reporting bugs. Meanwhile, the QA team begins hammering Kiln with their full test suite to make sure everything looks okay and we don’t have any regressions.
Once we’ve stabilized the couple of bugs that inevitably appear at this point (e.g., “I flew before the beginning of time, halp?”), and developed tests to make sure they don’t reappear, we begin deploying to Fog Creek On Demand. This goes in several phases:
1. In the first phase, we leak only to so-called “test” accounts, which are purely for QA. If something went horribly wrong, we could always just nuke these accounts.
2. In the second phase, we leak to alpha accounts. Alpha account holders are exclusively current and former Fog Creek employees who understand the risks.
3. If things go well there, we can optionally leak to beta customers, who are very nice clients of ours who have agreed to test versions before they go live so that they get new features first.
4. Finally, provided that everything has been stable for a whole week, we push the leak up to 100%. Fly the rainbow!™

A couple of things that I think work really well with this system:

Features are incredibly heavily tested before they reach customers. They’ve already been tested in isolation, again at integration, again on our corporate Kiln install, and again on our exact On Demand infrastructure. The exact point release that eventually goes out has already been live for days by the time our customers get it, on the exact servers that our customers use. All we’re toggling is who can see the new version.
Our deployment process is fully automated. It takes one button-click in Mortar to tag a build, and another click to deploy to our own servers. Deploying to On Demand is slightly more involved, since Mortar doesn’t currently allow arbitrary input and we want to be able to specify which accounts to upgrade. But at the end of the day, it’s still just a single script invocation that takes a list of accounts and a generation to deploy them to, and takes care of the rest.
Any individual customer can be upgraded (or downgraded) at will. One of the features that we can do with Fog Creek On Demand that is comparatively rare in other systems I’ve seen is that we can upgrade any individual customer to any version of Kiln and FogBugz that is currently hot (i.e., a valid deployment target). Many other systems I’ve seen can only upgrade at server-level granularity, which can be problematic if you’re trying to help a single customer out with a bug fix that you’re not ready to deploy to your entire user base.

The process isn’t perfect, but it’s one I’m very happy with, and one I’d like to emulate on other projects.

Midnight deploys are for idiots, and other revelations at 12:30 AM Monday morning

2011-03-08T07:59:09Z

Every once and awhile, you have an idea that you totally think is the bee’s knees but which ends up being the bee’s ass.

When Kiln and FogBugz went to a weekly release cycle a few weeks ago, we had what we thought, if not a brilliant idea, was a pretty good one: we’d upgrade Fog Creek On Demand at midnight on Saturday, when all the normal people were asleep. It’s pretty common practice, it minimizes customer impact, overall a good thing. Right?

By 12:30 AM on Monday, I had developed a very different opinion of the whole matter. In particular, I think the idea of doing a midnight deploy on the weekend has got to be the absolute dumbest thing I’ve ever heard of, right up there with Capris and the Nintendo Power Glove. Because, you see, you know how no one else works on the weekend? You know how that’s why it seems so alluring? Well, that applies to you, too. So, for the same reason that your customers are less likely to be hitting your website, you have a drastically decreased ability to handle things if something actually does go wrong.

This got drilled into my head very painfully this past weekend. Our build manager Rock did the deploy to production at 10 PM on Saturday, checked out that everything looked good, and then enjoyed his weekend. At about the same time, I checked my test account, saw the update had gone out, and verified that all seemed in order. So we both decided the deployment had been a great success, and had wonderful evenings, during which I may possibly have made a total ass of myself in front of a Kinect playing Dance Central, but that’s for another post.

But while we enjoying ourselves, Fog Creek On Demand had a deep, dark secret:

The deployment had failed.

You see, Kiln actually has three components: Kiln itself (what you hit when you go to e.g. https://mirrors.kilnhg.com/); FogBugz (which is used for a variety of purposes even if you don’t actually use FogBugz in your On Demand account); and a plugin for FogBugz which allows the two to communicate. Unbeknownst to us, but beknownst to the servers, the version of the Kiln Plugin we deployed did not actually work, in a very subtle way: all the APIs were valid except one, and that one call would fail. The real kicker, though, was that we anticipated that FogBugz might drop out for a moment when we originally designed the system, so we programmed Kiln to ignore being unable to communicate with FogBugz as long as possible. The failed API call turns out to be one that’s trivially cached for a very long time, and so is one that Kiln would allow to fail without actually dying. So even though everything looked just fine, and would work just fine for awhile, and you could even use Kiln and FogBugz and not notice a problem, Kiln accounts were doomed to start flicking off as they finally had to invalidate their cache and would be forced to disable themselves until FogBugz came back online.

The first people to discover this were exactly the people you’d expect, given our logo: Kiwis, since, due to the all-powerful Theory of Relativity, or perhaps merely Magic, they have managed to have their Monday mornings at about 7 PM on Sunday evening. So one of our Kiwi customers went to go hit their Kiln install, and were greeted by an infinite redirect loop, as a very lonely Kiln installation desperately attempted to find any FogBugz installation out in the world who would talk to it, and, finding none, simply tried again indefinitely. Tim called Tyler and me, who in turn called Rock, and just for good measure another FogBugz developer, and we all got the problem taken care of, so everything was perfectly fine.

No, wait! That’s not what happened at all!

Because it was Sunday, it took Tim about half an hour or so to actually get in touch with Tyler and me, during which our customers were stuck on a busy weekday, and Tim could make no forward progress on resolving the issue.
Once he did finally get in touch with us, it took us a long time to spin up on solving the problem, since we were forced to get to our apartments, fire up wimpy laptops, and get into our office computers via slow RDC-via-VPN connections.
Once we finally figured out the problem, we realized that, even though we knew exactly what was wrong, we had absolutely no idea how to fix it, because the issue was in the bowels of how FogBugz On Demand deploys its plugins. I therefore called Rock, who, in addition to being our build manager, also does an amazing job running FogBugz in his spare time. Rock unfortunately told me he couldn’t be available for half an hour, but if I wanted to take a stab at fixing things, I just needed to 齰蝌齰蜡. I said thank-you and then cried a little inside after he hung up.
I then tried to find a FogBugz developer to explain to me how exactly one did…whatever it was I was told to do. So I started calling FogBugz developers at random. The first one I tried to call was the plugin guru, but he was busy. So then I called our Canadian guru, Dane, who understood how to do the thingie, which ended up roughly translating to “load random assemblies into a running AppDomain.” This idea ends up not working, so let’s skip ahead.
About an hour later, Rock was able to join us, and we are finally able to try to cut new builds. Because the QA team does not generally work Sunday evenings, we were forced to do a minimal very minimal amount of QA before just opting to pull the trigger and deploy to all Fog Creek On Demand customers. By this point, it’s already about 11:30.

So, for those keeping score: it took half an hour to reach anybody who could fix the problem; it took more than an hour before we were actually in a position to properly debug it; and it took another couple of hours to actually get in touch with everyone required to fix the problem. Further, throughout the whole evening, everything happened in slow motion: people would step away from their computer and miss a HipChat notification, they’d need to put kids to bed, they’d need to take dogs for walks, they’d need to outrun the cops, and so on. Even when everyone was fully on-deck, I think conservatively that things happened at half the speed as normal at best.

Conversely, if we had simply deployed first thing on a workday, and things had gone wrong, we’d have had everyone right there ready to take care of it immediately. We’d have had a real QA team to make sure the build was high-quality before we unleashed it. We’d have been able to pile into a single office to discuss the problem in a very high-bandwidth maneuver, and I think we literally might have been able to fix the problem in 30 minutes, rather than five and a half hours. And despite our midnight Saturday deploy, customers were still adversely affected, and so ended up negatively impacting customers, and then being extremely slow to respond, compared to what we’d have been capable of with a midweek deploy.

Well, screw that.

New policy: Kiln and FogBugz will deploy midweek. We’ll still do it late at night, but in the middle of the week, we’ll have everyone on-deck first thing in the morning in case anything blows up, and our chance of being able to reach people on a Wednesday is dramatically higher than on a Sunday night even if things do have an immediate negative impact.

Weekend deployments are for chumps.

If you want the full post mortem, plus some more details on what went wrong, please visit my write-up on the Fog Creek status blog. I’ve also spelled out how we’ll be avoiding getting into this situation in the future, Wednesday deploys or no.

Making Your Interns Addicts: a How-To Guide

2011-05-27T08:22:15Z

I was thinking back last week on why I started working at Fog Creek. If you don’t know, I got started on this thing called Project Aardvark, which eventually ended up becoming Copilot, the project I worked on for my first couple of years at Fog Creek. I don’t generally reminisce much about that time, simply because that was a very different point in my life, back before I found fashion, yet after I figured out how to end up in front of cameras constantly.

So my friends are usually quite good at reminiscing about that time on my behalf, which is plenty. But with this summer’s batch of new Fog Creek interns arriving in just over a week, I’ve been thinking about it a lot. What was good, what was bad, what about the experience made me need to stay at Fog Creek.

I’ve realized there are quite a few parallels between what got me into Fog Creek and what gets people hooked on hard-core drugs.

Not in the bad way, mind you. I have yet to see an intern maniacally chase source code into a dirty toilet, for example.¹ What I mean is, I almost feel like our internship is custom-tailored to pound the crap out our interns’ sense of awesome so that they just have to come back for more. Just, unlike with drugs, the effect is that they have a great summer, and they want to work for us, and I honestly feel they end up being way more productive in the process. What I find remarkable is that while nearly every part of Fog Creek has changed since my initial internship, the actual way we run our internships continues almost unedited.

I really want all the interns in the world to come work for us, but I also want all the interns in the world to have amazing summers, and the best way to do that is to tell you what we’re doing to get our interns hooked. So without further ado, I’m happy to begin my first and last guide to deliberately getting someone addicted.

Step 1: Get Them Excited

The first thing that matters is getting them psyched just to be there. When I was an Aardvark, that was easy: we knew we were going to be in a movie. Being in a movie is exciting. It’s even more exciting when the film-maker comes to your house before the summer starts just to interview you and film you doing a really botched version of a Chopin nocturne that he manages to edit so you sound competent. I mean, who could say no to that?

Our interns this year may not be excited because they’ll be in a movie, but they can be excited for a different reason: they’re going to be working on a young project that still has thousands of customers and manages over half a terabyte of data across nearly 25,000 active repositories. They’ll get a chance to be making a difference.

Find something that you yourself are excited about, and get your interns excited in that. Or, if you don’t have anything at your job that excites you, find another job. We’re hiring, after all.

Step 2: Get Them Doing Real Work

Of course, if the thing you’re getting your interns excited about is playing with a widely used application, it does require you let them, you know, touch the code. My first coding internship, at a company that shall remain nameless,² involved me writing a tremendous amount of Ruby code. That could be awesome, and in a way, it was. Except that the Ruby code I was writing were scripts to do things like automatically upload new advertisement photos to the intranet, collated by the date the photograph was taken. Or, when I wasn’t so lucky, it was to make a blog engine that could hold exactly one blog post, and then display a little red icon if the blog post it was showing at the time was Bad For The Company™. Not exactly central to the product, you know?

So at Fog Creek, we always make sure our interns are doing real stuff for the project they’re working on. When I was an intern on Aardvark, I wrote most of the Windows helpers. Other intern classes have written a new wiki for FogBugz, and given Kiln its API and its large files support. Hell, an intern class was integral to getting the first version of Kiln done in the first place. It feels so much better to actually be contributing code to something you know is going to make a real difference to the product.

Or, you know, have them write a one-post blog engine or whatever. It’s cool. Just make sure they know at the end of their internship that we’re still hiring.

Step 3: Give Them Instant Feedback

You know what sucks about automating some random task that you’ve been assigned to do as an intern? Getting absolutely no feedback on what anyone thinks about it. Is it useful? Does it stink? Who knows. It’s a fricking compilation of advertisements on a corporate intranet. For all I know, no one even reads this at all.³

With one exception,⁴ Fog Creek interns always get to see Creekers using their code immediately, because we code right alongside them. We’re running their changes on our development systems, and, in many cases, running that code on our dogfood systems just a day or two after it’s written. Not only does this generally mean that our products are better, because we’re using new features heavily ourselves long before they ever reach our customer’s hands; it means that our interns are getting feedback almost immediately, both from within and outside their team, on what worked and what didn’t.

Or you can send feedback three weeks after the internship ends. By email. That works, too.

Step 4: Make Them Feel Awesome

People like feeling good. And the simple fact is, if you’re using your interns’ code almost immediately, and giving them rapid feedback, you’re going to end up having your interns produce really great stuff. So: tell them that.

Not in a fake way, mind you: some things just honestly take a lot of revisions before they’re good, no matter how awesome the people involved are, and your interns are going to know you’re offering them bull if you tell them something’s awesome that they know isn’t. But there’s all those intermediate steps, the little victories, the occasional big one, where you owe it to your team to acknowledge that, yup, they did well.

Saying “good job” is one way to show you care, but actually getting to know your interns is even better. At Fog Creek, we hang out with our interns outside of work by taking them to documentaries like Harry Potter 7, to Yankees “games”⁵ at the new stadium, and, my personal favorite, to top-shelf-martini-powered bowling. If you don’t have these available in your town, take them to the airport so they can bond over a TSA freaky frisk-fest. The point is, do something with your interns so you actually get to know and like them as something other than a coding automaton.

Step 5: Ship, dammit!

But all of that comes to naught if you don’t actually ship stuff. Sure, maybe you worked on Kiln, and sure, maybe Kiln makes a difference to the company, and sure, maybe your boss likes you. But what about your own stuff? Do customers care? Does it just go into a bit bucket at the end of the summer, never to be seen again?

Our interns won’t have to wonder too much about that: thanks to Kiln’s fast release schedule, they’ll be seeing their code go out to customers multiple times this summer, get feedback from customers about their changes, and probably even do a second round or two that integrates that feedback. So while they’re busy being excited, writing production code, getting feedback, and doing fun stuff, they’ll also be knowing, even during their internship, exactly what a difference they’re making.

You see what I mean? It’s exactly like an addiction. Just because our interns are awesome, work on great products, and make a real difference, they get this crazy positive feedback loop that means they not only have a great summer, but that they want to keep coming back, again and again. We’ve literally had freshman interns who come back every single summer because they love it here so much. We’re doing something right.

And I hope that, if your interns aren’t already feeling the same way, this guide can help you get there. I remember what it was like to feel like an undervalued intern, and I remember what it was like to feel like I was king of the world.

Help your interns have their own king-of-the-world moments, and everyone wins.

I hate Trainspotting so hard. ↩
The East India Company. ↩
Actually, I know people read it, because I got bored and figured out how to use “DHTML” to log who was viewing the page. But you get the point. ↩
It turns out that, try as we might, even Fog Creek can’t achieve perfection. Just reallygoodtion, with an eye towards learningfromyourmistakesery. ↩
My editor informed me that calling them “Red Sox beat-downs” was not appropriate if I didn’t want to offend my audience. ↩

Cache me if you can

2011-06-10T12:03:12Z

I remember, when we first launched Kiln, that I was desperately hopeful that it’d actually make a dent. I was honestly quite scared that we might end up in a situation where, after we’d put in all of this effort, spent all of this time designing and coding and testing and writing really arcane and annoying crap like billing code, that no one would want Kiln, and we’d be forced to close shop.

Good news! That’s no longer the problem.

The problem now is that we’re too successful.

What?! I hear some of you ask. Don’t you want to be too successful?!

I think I speak for my entire team when I say: hell yes! We want to keep on being too successful. If anything, we want to be even more too successful than we currently are.

But success does present problems. It’s awesome to have thousands of customers and terabytes of data, but then you start dealing with the boring details of questions like where do you put those terabytes of data, and how do you get that data to the customers in a timely manner. And that can be a really tough cookie to crack.

I’m going to be talking about several different aspects of how we’re handling scaling Kiln over the next few weeks, but today, I want to focus on one single narrow thing: caching.

The WISC Stack

The main part of Kiln that you all know and love—the website—is a fairly typical WISC application. We have a couple of web servers running IIS and the Kiln website, which talk to several SQL Server boxes. The nice thing about well-engineered WISC stacks is that, like LAMP, you can scale the databases and the web servers independently. This is the bread-and-butter of designing a scalable application. So far, so good.

The thing is, just adding more SQL boxes isn’t always the answer. If you have complex queries that take a long time to run, then adding another box won’t help anything. It just gives you another box to run your complex query on at the same slow speed. Even if you’re only doing simple queries, adding more database boxes isn’t necessarily the answer. Good SQL boxes are expensive—doubly so if you’re using a big commercial DB package such as SQL Server or Oracle. While you might be able to afford buy more, you don’t want to if you can avoid it.

Instead, you should focus on just not hitting the database in the first place.

The S in WISC

It turns out that there are already some mechanisms we had in place to help with this. We prefetched certain data that we knew we needed nearly every request (like the full list of repositories), and then used that cache for any other lookups during the request. And LINQ to SQL does a bit of its own per-request object caching in certain situations (such as querying an entity by primary key), so we already had some actual data caching going on.

While that kind of stuff can help, what we really wanted to do was to try to avoid talking to SQL at all for common operations. Those complex queries that Kiln does—things like showing the amalgamated DAG for all related repositories—take a long time to run, but the resulting data doesn’t actually change that often. This is a clear and wonderful win, if we can pull it off.

Making it Happen

There were two problems we had to solve: where do you cache the data? and how do you get it there?

The first one was pretty easy: use Memcache. Memcache is used everywhere, is extremely well-understood, and has great implementations for both Windows and Linux. We ended up settling on Membase as the particular server implementation we liked, due to its easy-to-use administration console and trivial clustering support. For the client, we ended up going with the BeIT Memcached library, due to its easy customizability and simple design.

The second part was much more difficult. Kiln uses LINQ to SQL for its database access layer. That meant we had a problem: LINQ to SQL is a very complex beast, where objects have a database context that in turn is aware of all the objects that it’s managing. If you just grab a random LINQ object and throw it into Memcache, then it is not going to deserialize cleanly. Throw in that we have piles of custom logic in our LINQ-to-SQL-backed models, and you’ve got a recipe for pain.

We ended up solving this in two different ways:

We modified our models to allow for detaching and reattaching to the database context. Before serialization, the object is detached, so it has no controlling database context. On deserialization, we attach it to the current context. This isn’t as fast as grabbing an attached object out of a cache (such as the old per-request prefetch cache mentioned earlier), but ends up incurring minimal overhead for the common case.
We also had to modify our models to know that they might not have come from the DB. We rely heavily on signal handlers to make changes in a given model class propagate to all the parts of Kiln that need to be notified. These were firing erroneously as the deserialization code set piles of properties. The fix we came up with was to suppress signals for deserializing objects—which, since most of our model modifications are done by T4 templates anyway, was very easy to do in a DRY manner.

With these two changes, we were able to reliably store LINQ database entities in Memcache, get them back out, and work with them.

It was easy enough to verify that the number of queries was down, but would the caching code make a real difference?

The Result

I think this graph of what load looks like on one of our DB boxes, before and after the caching deployment, says more than I could in several paragraphs of text:

We’ve cut the amount of data we’re getting from SQL by 75%.

The benefits we’re seeing are already impressive. We have faster load times and less DB traffic. But we can still do a lot more: now that we have the outlines of a caching framework, we can continue to profile and to move more of our expensive queries into the cache. Based on what we’ve seen so far, this should yield immediate and real benefits to our Kiln On Demand customers.

Make Love, Not Flamewars

2011-06-28T08:23:08Z

I sincerely doubt that the statement “I like Mercurial” will catch anyone who reads this blog by surprise. I brought it to Fog Creek. I evangelized for it on the Fog Creek World Tour. I helped build a whole product around it. I’ve gone to a Mercurial coding sprint, I’ve sent a whizkid to a Mercurial coding sprint, and I’ve even written a few patches (mostly trivial) for Mercurial. So let’s agree that I like Mercurial an awful lot.

Reading crap like this pisses me off.

The question seems innocuous enough at first glance: Why is Mercurial considered easier-to-use than Git? A legitimate, simple question. And in a sane world, perhaps, a fair one.

But that’s not how things work amongst us developers, because we have these utterly inane religious flamewars. Emacs is for octopuses, Vim is for beep mode. Ruby is a language for potheads, Python is for BDSM fetishists. DOS is for PCP-using masochists…okay, that one may be legit, but it’s also kind of moot at this juncture. Point is, there are some topics where developers just cannot have a rational discussion anymore.

Mercurial versus Git is one of them. To save you time, here is every single Mercurial v. Git discussion I’ve read since 2005:

Harry: Mercurial is awesome, because it is easy.

Sally: It’s not easier than Git! You’re just too dumb to see the light!

Harry: Okay but like Git is written in Perl! And it doesn’t run on Windows! And I had my changesets fucking garbage collected once! And the man pages are like 900 pages!

Sally: It’s not, it’s written in C, and it’s ninety bajillion times faster than Mercurial! And I had to screw with its database once! Also Git does too run on Windows, and I lost nine weeks worth of stuff in Mercurial once due to enabling the MQ extension!

But don’t take my word for it; you can go watch the internecine fanboyism play out at Reddit, right now! In response to that crap question! Again! God! I live in Groundhog’s Day!

You know what?

Both sides are full of shit.

Yeah, I believe that Mercurial is easier to use than Git. But the best example anyone can provide on StackExchange is converting the author names in a pile of commit messages? Really? Because, okay, yes, I’ve done that. Maybe twice. And even if it completely stank, I’d just memorize how to do it and get on with my life.

What about the daily workflow? Is that easier? Why? Is it fewer commands? Is there more protection from you shooting yourself in the foot? Does the protection come with less power? Why do Git users swear by branching, if “it’s in Mercurial since <some low version>”? Why do Mercurial users swear by their branching system? Does Git have an emulation of it? Why or why not? What are the trade-offs here? Do they actually end up mattering?

These questions matter. And they’ve been answered, very eloquently, many times. But for whatever reason, that doesn’t resonate with people. They want to find the weak argument that they can refute, and then that gets traction specifically because it’s weak, and everyone can have the fight all over again.

This stops now.

I have a solution.

Instead of talking about why you are better than the other guy, let’s focus purely on why your system of choice rocks. That’s it. No whyfooisbetterthanx.com. Just whyfooisgreat.com.

Here, I’ll get us started. I whipped up I Love Mercurial, a site where you can talk until you pass out about why Mercurial is awesome. Just tweet with the #ilovemercurial hashtag, and we’ll pick it up and post it.

Note: why you love Mercurial. Not why Mercurial is better than X. Not why X causes brain tumors in lab rats and sterilizes your children. Just, things that Mercurial does that you love? Talk about ’em on Twitter with the #ilovemercurial hashtag. We’ll post ’em. Then, the next time, someone asks you why you like Mercurial, just point them to that site. Or the hashtag. I honestly don’t care. Point is, show them why Mercurial rocks, and not why the other guy sucks.

And then maybe, just maybe, just maybe, at least on this one little topic, the flamewar can die in a pile of love on why my tool is awesome instead of the 92835 deficiencies your tool has that mine doesn’t, and I don’t have to wake up to my alarm clock playing I Got You Babe ever, ever again.

Please remember to update manners to tip before tweeting.