bitquabit

The fighting's been fun and all, but it's time to shut up and get along

Wed, 10 Feb 2010 11:37:39 -0500

About once a week, I get an email in my mailbox that reads like this:

Hey, Kiln looks neat, but Git is totally the bee’s knees, so why the fuck are you using Mercurial?

Note that these emails are rarely (if ever) actually interested in why Kiln chose Mercurial; what they’re instead interested in is trying to piss me off enough that I get into a flamewar about why Mercurial is going to bring about Nirvana while Git causes people to eat babies using nothing but A1 sauce and a spork.

This is stupid.

Mercurial and Git are both DAG-based DVCSes. They use the same patch format. They both handle directories implicitly. They both can autodetect file deletions and renames. They both run on just about every platform I can think of. They both do a bunch of “cool” stuff, like rebasing, editing history, signing changesets, and serving as the inspiration for Ferris Bueller. They both have nearly identical performance characteristics, they both have social code sharing sites, they both are used by really big projects, they both have really good documentation, and they both got compared to James Bond or MacGyver or something like that in an analogy I didn’t really follow.

So why is there so much hating? I think what’s going on is that people are coming to these tools from Subversion or CVS, have their massive epiphany on how totally awesome DVCSes are, and then assume that only their tool can have this level of awesome, so they begin evangelizing. The problem, of course, is that the other guy feels the same way, and is also evangelizing, so the Git and the Mercurial guy end up in a locker-room-style temper-tantrum over whose tool has the best performance or whatnot, instead of how much more awesome their tools are than the competition.

This has to stop.

Mercurial’s enemy is not Git. Git’s enemy is not Mercurial.

Their enemy is Subversion.

For example, take a look at this tripe. Anyone who has seriously used Mercurial or Git for any length of time is going to spend most of the video alternating between laughing, peeing their pants, and trying to explain that they merely spilled a bunch of water on their crotch, honest. While there’s some truth to the video—a lot of people do ditch Subversion for off-line commits or for shelving or the like—the video also totally misses why people love and stay with DVCSes, which is that they make branching and merging actually work like they’re supposed to, and make source control so fast and seamless that you find you’re suddenly using it for everything.

But if you show that video to a Subversion user, they’re going to nod. And there’s really no reason for them not to: if you don’t grok what the DAG gives you, if you’ve never kicked an entire branch back-and-forth across a LAN, if you’ve never used a site like Bitbucket or GitHub, then the only tangible benefits you see from DVCSes are…well, partial commits, and the fact that their “checkouts” aren’t littered with .svn directories all over the place. And these are problems that Subversion’s upcoming versions actually might solve.

So it’s time to focus on the real enemy: the holdouts still using centralized systems. The way I see it, there are three parts to this:

Git and Mercurial need to do a better job handling one thing that Subversion is still better at: binary files. I know there are Git projects that are working on improving the transfer and storage of binary files within Git’s existing repository format, such as git-bigfiles. Mercurial is taking a slightly different tack through projects such as bfiles, which aim to deliberately move large binaries out of the store. We need to improve these workflows so that “DVCSes are totally better, unless” becomes simply “DVCSes are totally better.” (For our part, Fog Creek is helping fund development of bfiles via UCOSP, a semester-long student project.)
Git and Mercurial advocates need to remember that they need to be converting Subversion, CVS, and Perforce users, not each other. The fundamental evil here is centralized, branchless version control systems that effectively encourage all development to occur in trunk, and which make propagating bug fixes and features properly borderline impossible for all but the most disciplined shops.
Git and Mercurial advocates need to remember that anyone going to either system is a win for the both communities. It’s easy, in the yin/yang of Hacker News and proggit, to forget that most developers are not even aware of what DVCSes are or what they do. Yeah. Sounds crazy, I know, but trust me on this. The goal right now, if you honestly believe that the DVCS workflow is better—and I do—should be to get the mindset out there, to make more people aware of what DVCSes have to offer and why they should be using them. I for one definitely do not care whether you end up deciding that Mercurial is better than Git for you or not (well, I kind of do, because I want you to use Kiln, but otherwise…), but I get a warm fuzzy feeling knowing that, if I ever have to work with your code, I’ll be able to use a sane version control tool.

So let’s do it. No more fighting. Git, I hereby acknowledge you rock. And Mercurial, you rock so much I helped build an entire product around you. You’re both awesome. So you two shake hands…very nice. Now, see that other dude over there? The one with no real tagging or branching support?

Let’s get him.

Kiln's Evolution, Part 2: From Prototype to Beta

Wed, 10 Feb 2010 08:12:56 -0500

This article is a continuation of Kiln’s Evolution, Part 1: DVCS as Code Review.

In the fall of 2008, Joel was getting increasingly adamant that FogBugz needed source control integration, and most people in the company seemed to think Subversion would probably be the best SCM to make that happen. Tyler and I disagreed, believing strongly that we should use a DVCS instead, and that our code review tool gave a really compelling example of why DVCS was better that any software shop would instantly “get.” But to convince the rest of the company, we’d have to show them a version of our tool that was more polished and usable than what we’d submitted to Django Dash.

And so we began a skunkworks project.

Unable to use time at work on much besides Copilot, I instead used my week of Thanksgiving vacation cleaning up the prototype’s user interface and functionality, named the result Kiln, and gave it its first logo. Tyler spent evenings in December making Kiln a proper, pluggable Django application, made the UI actually usable, and fixed a pile of bugs that would have blown up in our face if we’d tried showing Kiln to anyone else. By January, 2009, Kiln was ready to demo.

Kiln’s interface, directly after the winter skunkworks changes.

After lunch on a cold winter day, Tyler and I dragged everyone out into the kitchen and demoed the current state of Kiln. We showed repository management and the FogBugz-inspired code review workflow, and then made the case that this, or something very similar, should be FogBugz’ source control system.

And an amazing thing happened: somehow, everybody basically agreed. Sure, some people thought Kiln should be in C# or Wasabi instead of Django, no one could agree on whether Kiln should be a direct part of FogBugz or be an independent product, and Tyler and I argued strongly for Kiln as a hosted-only solution to a bunch of people who knew their bread-and-butter came from licensed applications, but everyone agreed that the basic of idea of a Mercurial-powered SCM with DVCS-backed code review made for a compelling product. And so Kiln was born.

For Kiln to develop into an adult, though, we had to assemble a Kiln team. Tyler and I were still working on Copilot, and the newly appointed team lead, Ben Kamens, was busy with the FogBugz 7 release. Even if all three of us started work immediately, we couldn’t possibly turn the project from prototype to beta by our target date of August 2009, and starting immediately seemed…well, optimistic, at best.

But we work at Fog Creek, and if there’s one thing Fog Creek knows how to do, it’s how to help interns churn out awesome products over the course of a single summer. After all, Tyler and I started at Fog Creek by developing all of Copilot in the summer of 2005; why not go for broke and try for a repeat? What we therefore decided to do was to bet the farm and put all of our summer interns on Kiln. The three of us would try to wrap up the work we had to do on our current projects as quickly as possible, and, as we transitioned off, we’d focus purely on building up enough Kiln infrastructure that the interns could immediately be productive when they arrived. Meanwhile, until the three of us could start work on Kiln, we’d have our project managers figure out the details of the user experience so that, once we finally could work on Kiln, we’d be able to focus as much as possible on coding instead of decision-making meetings.

As we moved ever closer to June, we made several key decisions about the design of the product:

Kiln could launch hosted-only, but we’d need to ensure that its design was amenable to on-site installation.
Kiln would depend on FogBugz for user management and bug-tracking integration, but would otherwise be its own code base.
Kiln’s website would be written in C# and ASP.NET MVC, completely freeing it from the FogBugz legacy code base.
The part of Kiln that needed to talk directly to the DVCS would be a separate component so that we target different (or even multiple) SCMs without changing the website.
Code reviews on branches would be eliminated in favor of arbitrary discussions on files and changesets.

By the time the first interns arrived, we had a beautiful set of specs with lovely Balsamiq Mockups put together by Jason and Dan, and we’d managed to cobble together a basic framework that supported repository hosting and FogBugz integration, and that learned as much as possible from our best example of a known-good ASP.NET MVC code base, StackOverflow.

Our plan paid off ridiculously quickly. A week into the internship, the interns had already managed to get key pieces of Kiln limping along. By the end of the second week, they had enough ownership they were starting to challenge us when they felt the user specs or the engineering didn’t make sense. Their strong focus on core Kiln freed Tyler, Ben and me to focus on performance, billing, On Demand integration, and all the other things that absolutely must get done for a real product, but that no one would otherwise ever do.

Just over a month into the summer, Kiln might not win any speed or beauty awards, but nearly all of its features were working in one way or another, and it was usable for its intended purpose. In other words, we’d hit pre-alpha. With great fanfare, we decided that Kiln was ready for dogfooding, and Kiln development moved to Kiln itself.

There’s a slightly unfortunate thing about dogfooding, though: features that looked great on paper, and even worked perfectly in the prototype, end up not being what you want in the real product. Some interfaces end up not scaling the way you want. Some end up too complicated, or end up solving one particular problem at the expense of all others. It’s a testament to our PMs that we had comparably few of these occur, but sometimes the difference between the prototype and what we ended up shipping was massive. For example, compare the Balsamiq mockup of the code review system, which was actually used by the pre-alpha:

with the version that we ended up actually shipping:

Or take a look at the original specification for the Kiln Dashboard:

compared to the shipping equivalent, the Activity Feed:

(I apologize about using the mockups, rather than screenshots, for the earlier versions; trying to get Kiln circa June 2009 running at this point proved a royal pain in the butt, and I don’t honestly think that it makes a big difference.)

What’s not obvious in these two screenshots is that the change from the pre-alpha interface to the shipping interface frequently happened over the course of just a couple of weeks. Everyone was very vocal about what they liked and didn’t like, and the interns were happy to go through several iterations rapid-fire to find one that everyone liked. In that way, the weakest parts of Kiln ended up getting the most attention, and rapidly matured into some of its strongest features.

In a massive code sprint at the end of the summer, Kiln matured into something resembling a fully grown product. One of our interns made a beautiful JavaScript renderer for Kiln’s DAG, the novel repository management our PMs designed fully matured into beautiful JavaScripty goodness, our FogBugz/Kiln workflow became increasingly seamless, and we further loosened review requirements so that you could review arbitrary discontiguous changesets. We transitioned Copilot and FogBugz to be hosted on Kiln as well, got mostly positive feedback from the rest of the team, and worked on swiftly addressing their complaints. Despite these feature additions, Kiln’s performance went from tolerable, to better, to fast. We knew we had a winner on our hands. We prepared the FogBugz On Demand environment to become Kiln On Demand, made our first deployment, and turned the switch for our first batch of beta users.

And while you might expect this part of the story to be about everything going haywire and all hell breaking loose, what actually happened is that, against all odds, everything basically worked. The beta was quite boring: while there were a lot of bugs to fix at first, and some of them were extremely tough (for example, supporting very large repositories, or making history views faster, or legitimately supporting Internet Explorer) or really ticked off our customers (Kiln at once point let you rename and move repositories without breaking URLs, which sounded like an absolutely great idea when I helped hammer it through the design committee, but which completely blew up in one of our client’s faces a few weeks later), everything basically worked. Our beta testers seemed increasingly excited about the product. All of our gambles seemed to have paid off as we readied Kiln 1.0 for its November launch date.

Except, of course, that Kiln did not ship in November, 2009. That’s because, just a week or two before it was supposed to ship, we went to the Business of Software conference in San Francisco.

You see, that was where we realized we were doing it all wrong.

To be continued…

Firing Up Kiln

Wed, 13 Jan 2010 08:48:36 -0500

As Kiln draws ever closer to release, I realized that we have long since passed the point where I should move all of my personal projects to it.

So as of today, I have.

If you’re interested in grabbing the most recent version of FogBugz Middleware, my fork of Kiln Backup, or any of my other public projects, they’re now all at https://bqb.kilnhg.com. Log-in with the user guest and the password anonymous, and you’ll have full read-only access to the entire site.

Even if you’re not that interested in the stuff I’ve written, you still may be interested in checking out the site: by enabling read-only guest logins on my FogBugz account, I’ve made it very easy for you to take a look around a real, active Kiln and FogBugz install, without setting things up or filling out any annoying forms. If you just wanted to a get a quick feel for what Kiln looked and felt like, now’s your chance.

So go ahead and check it out. Feel free to post question or comments here, or (if appropriate) over at the Kiln StackExchange, and I’ll be happy to answer them.

On Being Good

Tue, 12 Jan 2010 21:42:29 -0500

Google’s motto is, “Don’t be evil.”

I’ve always found that motto disturbing for two reasons. First, a company that can differentiate itself—successfully, no less—from its competitors merely by promising not to be evil implies that the average company is ridiculously corrupt. A person who announced, “My motto is, ‘don’t shoot people’” would be notable because no one thinks you should shoot people, making the promise weird and redundant—not because the promise represented some great sacrifice. Yet Google’s promise to do no evil somehow hits people, especially those in the tech industry with fresh memories of Microsoft in the 90s and the specter of Oracle in the 2000s, as a breath of fresh air. Great for Google, but pathetic for our industry.

But the second reason, and the more important one for me, is that “Don’t be evil” is not the same as “Do the right thing.” A person who watches idly while a bully beats someone up isn’t being evil, but they are being a coward, and they are not doing the right thing. Their interference could save a poor victim a world of pain and suffering, probably at minimal risk. Instead, they simply watch the bully, knowing that they themselves would not do the same thing. This may not be doing evil, but it’s also not the moral high ground. Knowing you would never beat someone up is not the same as protecting those weaker than you.

Google chose its motto carefully; if its motto were instead, “Do the right thing,” then it would have no presence in China. For all the corruption that people accuse our government of perpetrating, our government does not censor the Internet, does not shoot and incarcerate those who disagree with it, does not deny its citizens the right to vote, and does not persecute religious minorities as a matter of state policy. China does. And until today, while Google may not have been evil in China, they certainly enabled evil to go about its business by running a censored search engine there. They were unequivocally better than Yahoo, who handed over the names and email addresses of dissidents, but they weren’t doing the right thing, either. They weren’t standing up to an autocratic, dictatorial regime.

As of today, that has changed. Google has announced that they will no longer censor their Chinese search results. While you could argue that Google’s doing this out of anger that their resources have been hacked, rather than out of a genuine desire to protect its users, their result of their actions is beyond dispute: they are taking the moral high ground. And potentially at great cost: while China has certainly failed to materialize as the unstoppable threat to the West that pundits were claiming it would become two decades ago, it’s nevertheless home for nearly a billion people, and shows no sign of stopping its economic growth in the near future. For Google to make a move that will almost certainly sacrifice any chance they have of winning the Chinese market is an economically painful move.

But it’s the right move.

So, at least for today, at least this once, look at Google as a company which is not merely avoiding perpetrating evil. Google is doing the right thing, at great cost. And they deserve to be lauded for that.

Microsoft and Yahoo: this is your turn to follow in Google’s footsteps. Do the right thing. It won’t make you money. In fact, it’ll cost you. But it’s the right thing to do.

Google: where it’s not don’t be evil. It’s, “Do the right thing.”

Congratulations.

The Amazing Spammable Marketplace

Thu, 07 Jan 2010 16:05:22 -0500

Whenever I browse the Android Marketplace, I’m utterly amazed by how many “app reviews” are nothing but spam. The problem is so pandemic that I have to conclude that Google has thus far done absolutely nothing to combat the problem. Shopping in the Marketplace ends up feeling like going through a dirty bazaar, surrounded by panhandlers and con artists looking to make a cheap buck. I don’t care how good the deals may be; if shopping ends up being an annoying experience that makes me feel dirty, I’m unlikely to bother going in the first place. In the Marketplace, that translates to finding nothing but crap reviews, making shopping for any given application basically a crap shoot. Plus, the apps themselves end up looking like schlock you’d find on a spyware site.

If Google’s serious about trying to compete with the iPhone App Store, they need to get off their feet and fix this problem right now.

Droid Update Makes Droid Not Suck

Fri, 11 Dec 2009 11:07:59 -0500

Well. At least it makes it suck less.

I bought a Droid the day it came out. While it was a tremendous improvement over my BlackBerry, I’ve been disappointed with the phone overall. The battery cover comes off constantly ([2], [3]), the phone’s proximity sensor was extraordinarily finicky (usually resulting in me hitting the “mute” button with my cheek in the middle of a call), the camera was all but useless, and, for reasons I did not really understand, my Android developer phone running Android 1.6 provided a much smoother user experience than the vastly-more-powerful-on-paper Droid. In other words, the Droid was a solid upgrade from what I had, but still disappointing. I have to agree with Dave Winer’s now-famous rant on why the Droid sucks.

Last night, Motorola and Google unleashed Android 2.0.1 as an over-the-air update. While the update does little about the battery cover, it seems, at least so far, to resolve nearly all of the software issues. The proximity sensor’s logic seems improved, though not perfect; many operations are visually smoother (although oddly still not universally as smooth as the G1); the camera’s usable taking pictures, rather than mocking the incompetence of Motorola’s engineers; and there have even been some very nice visual refinements to fonts and color schemes. Best of all, and unusual for the first update to a new device, nothing broke: sudoku, SpacePhysics, Twidroid, TripIt, and other applications seem to still be working just fine.

So if you were previously hesitant about buying a Droid for software reasons, and don’t really have a problem using $2 double-sided tape to compensate for Motorola’s QA team having the same skill as an inebriated eight-year-old, I think you’ll be much happier with your purchase now than you’d have been a month ago. Otherwise, you might want to wait for the next Android-powered phone on Verizon and see if it works better. It’s certainly unlikely to be worse.

Kiln's Evolution, Part 1: DVCS as Code Review

Tue, 10 Nov 2009 08:41:19 -0500

One of the things that really sucks about doing online code reviews is that, in all the systems I know, your code reviews do not integrate with your source control. If the code reviews are versioned at all—and they’re frequently not—then they’re in an entirely different system than your real VCS. For larger reviews, where you’re talking about a major piece of functionality, that means that your source control system will end up lacking the history of how a feature came to be. In other words, the more you use code reviews, the less actual history you have in your VCS.

That’s totally broken. You’re being punished for doing the right thing.

A little over a year a year ago, Tyler came to me and asked me to join him in the Django Dash, a weekend code sprint. Tyler and I had been talking about the code review problem, and had been thinking of writing our own that lacked these issues. Django Dash seemed like the perfect time to try to actually do that.

That opened up a question: how do you actually achieve better code review? If you accept that a huge part of the problem is that the code review history is out-of-stream with your VCS, then it follows that you have to somehow store the in-process code in the VCS.

In most systems, that means using branches. But branches in almost any system suck. Everyone has a horror story about trying to do a merge in Subversion, or CVS, or Perforce—and these are usually not meaningfully large merges; just small feature branches. Trying to use branches for long-running code reviews in these systems simply isn’t viable.

But DVCSes are great at handling this kind of problem. To be distributed, they have to have extremely robust branching and merging systems. Because their systems are so good, it’s very common in DVCSes to do quick experiments and features in their own branch, then merge when complete.

Tyler and I were both big fans of Mercurial (in fact, we convinced all of Fog Creek to switch to Mercurial from Subversion), so using Mercurial as our DVCS base seemed like the best bet. After some discussion of the technical details for making the system work, we got a good night’s sleep, woke up early, threw a nice breakfast on the table, and started coding.

Forty-eight hours later, we had our first prototype. When users wanted to contribute code to a repository, they would fork the repository, push all of their changes to the fork, and then request a review on the fork. Users would see the exact diff of what they would be approving to the repository; no more, no less. Code could not be approved unless it had already merged in the trunk, ensuring that the user who wrote the code had taken care of the merge. When the review was approved, it’d be seamlessly merged into trunk (guaranteed seamlessly, due to the previous rule), with full history.

The design was inflexible and unintuitive, and would have had serious issues in a shipping project, but we achieved what we set out to do:

Approving a code review was the same as pushing it
Which meant that we could fully separate the concepts of code author and code approver
And which meant that the full history of reviewed code was completely preserved

A screenshot of the prototype that would later become Kiln

Even in its nascent form, the tool was already impressive enough, and unique enough, that we won the Django Code Dash.

Tyler and I talked of making our code review tool into a real product, but we were knee-deep in Copilot work, so it had to wait. But we had proved, if only to ourselves, that using a DVCS, even in a centralized model, provided some very unique capabilities that simply were not possible in other systems.

When Joel announced a few months later that FogBugz needed a source control system, we’d be ready.

A Typographer's Captcha

Thu, 15 Oct 2009 18:07:23 -0500

I’m not entirely sure that this qualifies as a reasonable captcha for mere mortals.

Perhaps on a typographer’s site…

The Launch of a Secret Product

Wed, 14 Oct 2009 09:12:34 -0500

For the past year, an odd thing has happened, if you’ve followed my doings. My work on Fog Creek Copilot seemed to dwindle, I became tight-lipped about what I was working on, and I started getting really excited about an upcoming product release. Also around this time, my knowledge of Mercurial, Python, C#, and ASP.NET MVC all seemed to dramatically increase, even though my free-time code output shrank to nothing. What was going on?

Oh, the usual. I was working on a top-secret brand-new project. And now, it’s released to closed beta.

I’d like to introduce to you Kiln, a brand-new source code hosting and code review tool from Fog Creek. Kiln introduces what I believe is a truly novel take on code reviews that integrates the strengths of Mercurial and FogBugz to provide rejectable code review after commit. We also have some unique takes on Mercurial features, such as the ability to preview what will go into a merge beforehand, really awesome branch management, the most beautiful DAG view I’ve seen in any DVCS product, and lots more.

Over the next few days, I’ll be providing a couple of blog posts detailing how Kiln was developed. In the meantime, go check out Kiln and sign up for the beta. We’re approving new people for the beta on a regular basis, so if you don’t get an invite immediately, don’t worry; we’ll get to you sooner rather than later.

hg log -R tips_and_tricks

Fri, 09 Oct 2009 07:54:32 -0500

I was delighted to find that Steve Losh has begun making a website called hg tip—a site updated on a regular basis with Mercurial tips for both beginner and expert users. The site’s beautifully designed and a pleasure to read. If you use Mercurial, do yourself a favor and go take a look.

(My favorite tip, incidentally, is definitely the tutorial on making a command called nudge, which allows you to push only the current head by default, rather than all of them. I’ve been using a variant of that, combined with bookmarks, to have Git-style lightweight branching in my Mercurial work when appropriate.)