When you click it, you can enter a comment in a new dialog:

which then shows up in in the editor:

The plugin uses some javascript hackery to modify the timesheet editor since there is no plugin interface to implement to change its functionality. As such, it works with FogBugz 8.7 and is not tested and approved for FogBugz On Demand. You can download the plugin to run on your own local FogBugz server, and get the source code from my post here on the FogBugz Knowledge Exchange.

We announced that we were open-sourcing WebPutty back at the end of February, and now its source is open; out there wild and free for all to see!

Just check out WebPutty on GitHub, follow the instructions there, and you should have your own sparkly new copy of WebPutty up and running in a matter of minutes. Sweetness!

Important reminders about www.webputty.net:

• WebPutty.net (and any stylesheets hosted there) will be available until December 31, 2012. Past that point, you will need to migrate to your own WebPutty installation, or find another provider.
• You can export any of your stylesheets hosted on WebPutty.net by opening those stylesheets in WebPutty’s editor and clicking the “Export” button in the upper-right-hand corner of the editor.

View Designing and Developing Trello for Multiple Devices on Scribd

First, the short version:

• WebPutty’s going to go open-source! If you want to host your own installation, you’re going to be able to.
• While the official WebPutty.net site has had a wonderful reaction from the community, we can’t justify keeping it around as a Fog Creek product. WebPutty.net (and any stylesheets hosted there) will continue to be available until at least December 31, 2012. Past that point, you will need to migrate to your own WebPutty installation, or find another provider.
• If you’re concerned about any of the above, you can export any of your stylesheets hosted with WebPutty by opening those stylesheets in WebPutty’s editor and clicking the “Export” button in the upper-right-hand corner of the editor.

Questions? Concerns? Interested in taking over the hosting? Please contact us at webputty@fogcreek.com.

And now the more detailed version…

WebPutty has been unique for Fog Creek from day one. The initial idea for WebPutty came up during a one-off brainstorming session Tyler and I held with Jason, a former Creeker (and good friend). Tyler and I had already spent a few weeks brainstorming ideas for new products with Joel and Michael, but when the idea for WebPutty (code-named “CSSFiddle” at the time) popped into our heads, we couldn’t resist getting a simple proof-of-concept version of it up and running.

So that’s what we did. We sat down in Tyler’s office one June day in 2011 and in just one single afternoon got the basic premise behind WebPutty, a live preview of as-you-type-it CSS, working on Google AppEngine. When we showed it to Joel and Michael, we all agreed to give WebPutty a shot. We had no idea whether it would be successful; we just knew it was awesome, and we wanted to try. On July 20th, a mere six weeks later, we launched WebPutty to the world in an announcement that was one of our most popular blog posts ever!

Needless to say, we were very happy to have received such a warm response, and after coming down from the high of shipping, Tyler wrote up a more detailed summary of how WebPutty came into being in just six weeks.

Fast-forward seven months. Where is WebPutty now? Well, despite us doing effectively zero marketing for WebPutty, we’ve gotten some pretty respectable numbers:

• Over 25,000 people have logged in and taken a look around
• Over 9,000 sites (collections of stylesheets) have been created
• Millions of CSS requests have been served by our WebPutty CDN
• Thousands of happy tweets from customers using WebPutty have graced our feeds

Not bad!

…but also not good enough. Despite these promising numbers, WebPutty has not seen the broad user adoption we would have liked. Given Fog Creek’s current size, we do not plan to continue developing WebPutty. Instead, we’re going to open source the code and see where the community takes it.

But don’t worry! We will continue to host WebPutty.net until at least December 31, 2012. We’ll be blogging again with more details after we’ve had a chance to clean up the code base for public release, including providing documentation for getting your own copy of WebPutty up and running.

WebPutty has been a giant experiment from day one. It’s the first product we’ve come up with in this manner, the first product we’ve launched so quickly, the first free-right-off-the-bat product we’ve launched, and now will be the first complete product we’ve ever open-sourced. WebPutty’s taught us a lot, and we’re proud of how it’s been received and the things it’s helped others question when it comes to making website design and development easier. Did it change the world? Not quite. But did we have a blast working on it and make the lives of others at least a little easier in the process? You bet we did.

We appreciate everyone who’s taken a look at WebPutty and hope that at least some of those same people will consider contributing to WebPutty’s code base once it’s been open sourced. We’re excited to see where the ideas behind WebPutty can be taken once a broader audience of people are able to contribute to it.

One of the many ways to spawn cases in FogBugz is via a mechanism we call BugzScout. BugzScout is a simple interface for collecting crash reports from your software in the wild. For example, you might add an exception handler to your product which hooks into BugzScout so that when your customers encounter errors, you receive a case in FogBugz with details about the problem. Unlike the FogBugz XML API, which can also be used to serve this use case, BugzScout doesn’t require that you embed any type of FogBugz security token in your deployed application, and BugzScout will automatically collate all of the bug reports into cases based on titles of the reports.

We use BugzScout extensively throughout our own infrastructure — when FogBugz accounts misbehave, if an unparseable email gets stuck in an account’s mail queue, if an account’s full-text search index gets corrupted, etc., we receive cases in our FogBugz environment detailing the problem and often including key diagnostic information such as stack traces and assorted report-specific meta-data.

BugzScout has been around for a while, long enough to get a little crufty, and we’ve felt some BugzScout pain as we’ve scaled up. Pain can lead to cases:

We generally agreed with these suggestions, and we’ve definitely known the frustration of some small process misbehaving and sending literally thousands of BugzScout reports our way.

Better.

The first thing we came up with was what we all thought would be a completely amazing BugzScout-esque product which could solve the BugzScout use case perfectly. But we didn’t build that… its scope was slightly larger than the part-of-a-week we wanted to spend on it. Plus, building a new product wouldn’t have solved our immediate problems fast enough. Our driving force behind the improvements became the following insight:

Sometimes you get a human-readable number of BugzScout reports. In these cases, you absolutely want the full error text from each occurrence, and it’s reasonable to route the event through all of the standard notification hooks. If someone doesn’t want to hear more about this error, they can direct BugzScout to stop reporting via editing the case.

Other times, you get so many reports that no human will read them. The case is spammed with stack traces and error notifications. After some point, additional stack traces are of no use — you’re not going to read through them anyway, they just make the case cumbersome. At this scale, you need BugzScout to do a few things:

1. Shut up and let me fix it. You’ve already gotten a bunch of emails about the problem- one additional email per occurrence is of no use to anyone, but can legitimately hinder your efforts to actually deal with the problem.
2. Help me know I’ve fixed it. It’s always been possible to determine if more reports are coming in by either watching the occurrence count of the BugzScout case or by monitoring when the case was last modified by BugzScout. Unfortunately, if you wanted to accomplish #1 by setting BugzScout to ‘Stop Reporting’, you’d also stop getting updated timestamps from when new reports came in.

We decided that reconciling #1 and #2 would be a great improvement for BugzScout and would be well within the scope of a few days of development. We felt confident that this type of change would solve our pain point, but before we went any further we wanted to make sure we weren’t operating in a no-fact zone. We reached out to our SaaS platform and gathered some anonymous data on how people were using BugzScout. The data confirmed our guess: Our use cases are valid and a significant amount of people are using BugzScout-based error reporting.

We settled on:

• BugzScout will now automatically place itself into “Stop Reporting” mode after 50 occurrences of a report.
• Last Occurrence is a new case field which is stored alongside all BugzScout cases and is fully searchable, filterable, etc. Last Occurrence is the timestamp of the most recent BugzScout report, regardless of whether that report was recorded as a case event.

Resulting in:

• Human-readable amounts of BugzScout reports should be unaffected.
• Human-unreadable amounts BugzScout reports will automatically cut off after 50. This means that the bloody tide of email will be curbed; BugzScout cases are limited to having a workable number of events in them.
• If it comes time to collect more reports, you can do this by editing the case and instructing Scout to ‘Continue Reporting’.
• Last Occurrence is a new and useful filterable value.

Shipped.

We’ve been dogfooding the new BugzScout in our FogBugz environment for the last month and have been quite happy with the improvements. We’ve found that the automatic shutoff of case event generation after 50 reports has allowed us to function much better when dealing with issues at scale.

Here’s an example of how our team likes to monitor the incoming BugzScout reports using the new Last Occurrence column:

This isn’t the only way we triage our BugzScout reports, but this simple filter lets us quickly spot any issues that are happening right now and know if they’ve happened with enough frequency to warrant immediate attention.

These changes went live to FogBugz On Demand two days ago, so we hope that you’ve found them useful. Licensed FogBugz customers will get these changes in our next licensed release.

But we also have an iPhone app. It’s pretty great if I may say so. If you are a Trello user with an iPhone, you should download it now. It’s free. Now that there is a great app, why would we still focus on a mobile web app? One reason is that focusing on mobile makes Trello better as a whole.

Everything we do for mobile translates back to a better desktop experience. This wasn’t the first reason we wanted to implement a responsive design, but it turned out to be the most important. Keeping mobile in mind focuses us on creating a fast and easy-to-use interface. Trello gets data super fast, but the page must also render quickly. That stops us from using complex interactions and some of the more whiz-bang CSS features. This translates back to rendering speed on your desktop, which isn’t likely to be an overclocked gaming rig with 32GB of RAM.

All the interface elements are mobile-friendly, which means they are also more desktop-friendly. Buttons and menus have big, friendly hit targets. Interactions are straight-forward. We don’t rely on hidden hover effects and if we’ve got a complex interaction, it will have a fallback for touch devices.

There are no redirects. Let’s say you’re on your phone and somebody links to trello.com from Twitter. You’re able to watch the video, read the pitch, sign up, and try it out. You are left with a good impression, no nonsense. No screaming “DOWNLOAD THE APP!” page, no switching out of whatever app your using, and you won’t be redirected five times and land on the m.trello.com homepage. You just get the information and it works.

Scaling, zooming, and resizing work seamlessly. Sometimes you want to use a small browser window on a desktop. Maybe you’ve got a side monitor with a Trello board and your mail app open while your text editor or photo editor are up on your main screen. The horizontal board view won’t work at smaller window sizes, but Trello will adapt to a view that does. If you need bigger or smaller text, you can zoom in (or out) as much as you need and Trello will use a view that works. Got a huge monitor and want to project your Trello board for all to see? Trello will work for that as well. Is it way far back in the office? Just zoom in and text will be readable.

We can deliver updates to all devices seamlessly. We have a single codebase and one place to deploy, which means updates are easy. We don’t have to retro-fit new features to a separate codebase, worry about new workflows or interactions, or think about how some URL is going to work. This lets us develop and ship features faster.

So how does it all work?

Here are some of the tools and tricks that made developing a responsive interface much easier.

Use a limited library of mobile-optimized, reusable components. Each component can be collapsed into a single column or otherwise adjusts for smaller screen sizes. The same layout elements used for the back of cards are used in the organization profile and a bunch of other pages. Our context menus (those small pop-ups used to do things like assign members and select labels) are narrow enough that they will fit on any screen. We have some one-offs like the landing page and the board, but they are few and far between. For the most part, we don’t have to ask how a new feature is going to work on mobile because any component we use will already be mobile-optimized.

Think twice about navigation. The card sidebar typically has navigation and buttons for voting, assigning, and the like. It collapses below the main column with a smaller window, which means you would have to scroll and scroll to get to that vote button you were looking for. We added a card menu to the header on the back of cards that lets you easily vote, add due dates, assign members, and everything else without wearing out your thumb.

Use a vector-based image editor like Illustrator to produce icons. We wanted to make sure icons looked sharp on devices with a higher pixel ratio like the the iPhone 4. We use a CSS sprite sheet that has every icon used on the site. We can easily export a higher resolution version using Illustrator, and serve it to capable devices using some simple CSS media queries.

That being said…

We’re developing for the browsers and devices of today and tomorrow. We don’t have spare development time, so we can’t spend any on browsers and devices that won’t be around in two to three years. Trello won’t work on the RAZR, for instance. It also doesn’t work on every ‘smart’ device. Internet Explorer 8 doesn’t have the technical capabilities to run Trello. The Windows Phone 7.0 browser is based off of IE8, so it won’t work either. But it will work on Android 2.3+, iOS 4.0+, Windows Phone 7.5 (Mango), and others. Are we neglecting would-be happy users? Perhaps. But we can provide more value to more people by shipping features faster and that’s better in the long run.

And we still love native apps! Apps provide things we can’t get out of the web: better speed, offline support, smooth animations, push notifications, and a native look and feel. Native apps will provide a better experience to a broader reach. We just don’t think the mobile web should be ignored because of them.

Designing for multiple devices has deeply influenced our design decisions and made for a better, faster, and easier-to-use product. There are plenty of improvements to be made, but we’re happy with the foundation we’ve set. Sign up now and check it out for yourself.

The Initial Trello Mockup

That led us toward a single-page app that would generate its UI on the client and accept data updates from a push channel. This is pretty far from any of the work we’ve done before at Fog Creek, so from a technical perspective Trello has been an adventure.

Initially, we were wondering how interesting and far-out the stack could be before management got nervous, but our concerns were addressed in an early meeting with Joel, when he said “Use things that are going to work great in two years.”

So we did. We have consistently opted for promising (and often troublesome) new technologies that would deliver an awesome experience over more mature alternatives. We’re about a year in, and it’s been a lot of fun.

CoffeeScript

Trello started out as a pure JavaScript project on both client and server, and stayed that way until May, when we experimentally ported a couple of files to CoffeeScript to see how we liked it. We loved it, and soon converted the rest of the code over and started coding CoffeeScript exclusively.

CoffeeScript is a language that compiles to readable JavaScript. It existed when we started Trello, but I was worried about the added complexity of having to debug compiled code rather than directly debug the source. When we tried, it, though, the conversion was so clean that mapping the target code to the source when debugging in Chrome required little mental effort, and the gains in code brevity and readability from CoffeeScript were obvious and compelling.

JavaScript is a really cool language. Well-written CoffeeScript smooths out and shortens JavaScript, while maintaining the same semantics, and does not introduce a substantial debugging indirection problem.

The Client

• Backbone.js (client-side MVC)
• HTML5 pushState
• Mustache (templating language)

The Trello servers serve virtually no HTML. In fact, they don’t serve much client-side code at all. A Trello page is a thin (2k) shell that pulls down the Trello client-side app in the form of a single minified and compressed JS file (including our third-party libraries and our compiled CoffeeScript and Mustache templates) and a CSS file (compiled from our LESS source and including inlined images). All of that comes in under 250k, and we serve it from Amazon’s CloudFront CDN, so we get very low-latency loads in most locations. In reasonably high-bandwidth cases, we have the app up and running in the browser window in about half a second. After that, we have the benefit of caching, so subsequent visits to Trello can skip that part.

In parallel, we kick off an AJAX data load for the first page’s data content and try to establish a WebSocket connection to the server.

BACKBONE.JS

When the data request returns, Backbone.js gets busy. The idea with Backbone is that we render each Model that comes down from the server with a View, and then Backbone provides an easy way to:

1. Watch for DOM events within the HTML generated by the View and tie those to methods on the corresponding Model, which re-syncs with the server
2. Watch the model for changes, and re-render the model’s HTML block to reflect them

Neat! Using that general approach, we get a fairly regular, comprehensible, and maintainable client. We custom-built a client-side Model cache to handle updates and simplify client-side Model reuse.

PUSHSTATE

Now that we have the entire client app loaded in the browser window, we don’t want to waste any time with page transitions. We use HTML5 pushState for moving between pages; that way we can give proper and consistent links in the location bar, and just load data and hand off to the appropriate Backbone-based controller on transition.

MUSTACHE

We use Mustache, a logic-less templating language, to represent our models as HTML. While ‘harnessing the full power of [INSERT YOUR FAVORITE LANGUAGE HERE] in your templates’ sounds like a good idea, it seems that in practice it requires a lot of developer discipline to maintain comprehensible code. We’ve been very happy with the ‘less is more’ approach of Mustache, which allows us to re-use template code without encouraging us to mingle it with our client logic and make a mess of things.

Pushing and Polling

Realtime updates are not a new thing, but they’re an important part of making a collaborative tool, so we have spent some time on that layer of Trello.

SOCKET.IO AND WEBSOCKETS

Where we have browser support (recent Chrome, Firefox, and Safari), we make a WebSocket connection so that the server can push changes made by other people down to browsers listening on the appropriate channels. We use a modified version* of the Socket.io client and server libraries that allows us to keep many thousands of open WebSockets on each of our servers at very little cost in terms of CPU or memory usage. So when anything happens to a board you’re watching, that action is published to our server processes and propagated to your watching browser with very minimal latency, usually well under a second.

AJAX POLLING

It ain’t fancy, but it works.

Early Architecture Drawing

When the client browser doesn’t support WebSockets (I’m lookin’ at you, Internet Explorer), we just make tiny AJAX requests for updates every couple of seconds while a user is active, and back off to polling every ten seconds when the user goes idle. Because our server setup allows us to serve HTTPS requests with very little overhead and keep TCP connections open, we can afford to provide a decent experience over plain polling when necessary.

We tried Comet, via the downlevel transports for Socket.io, and all of them were (at the time) shaky in one way or another. Also, Comet and WebSockets seemed to be a risky basis for a major feature of the app, and we wanted to be able to fall back on the most simple and well-established technologies if we hit a problem.

We hit a problem right after launch. Our WebSocket server implementation started behaving very strangely under the sudden and heavy real-world usage of launching at TechCrunch disrupt, and we were glad to be able to revert to plain polling and tune server performance by adjusting the active and idle polling intervals. It allowed us to degrade gracefully as we increased from 300 to 50,000 users in under a week. We’re back on WebSockets now, but having a working short-polling system still seems like a very prudent fallback.

The Server

• node.js
• HAProxy
• Redis
• MongoDB

NODE.JS

The server side of Trello is built in Node.js. We knew we wanted instant propagation of updates, which meant that we needed to be able to hold a lot of open connections, so an event-driven, non-blocking server seemed like a good choice. Node also turned out to be an amazing prototyping tool for a single-page app. The prototype version of the Trello server was really just a library of functions that operated on arrays of Models in the memory of a single Node.js process, and the client simply invoked those functions through a very thin wrapper over a WebSocket. This was a very fast way for us to get started trying things out with Trello and making sure that the design was headed in the right direction. We used the prototype version to manage the development of Trello and other internal projects at Fog Creek.

By the time we had finished the prototype, we were good and comfortable in Node and excited about its capabilities and performance, so we stuck with it and made our Pinocchio proto-Trello a real boy; we gave it:

• a real DB and Schema (node-mongodb-native and Mongoose)
• basic web tech like routes and cookies (Express and Connect)
• multiple server processes with zero-downtime restarts (Cluster)
• inter-process pubsub and structured data sharing via Redis (node_redis)

HAPROXY

We use HAProxy to load balance between our webservers. It balances TCP between the machines round robin and leaves everything else to Node.js, leaving the connections open with a reasonably long time to live to support WebSockets and re-use of a TCP connection for AJAX polling.

REDIS

Trello uses Redis for ephemeral data that needs to be shared between server processes but not persisted to disk. Things like the activity level of a session or a temporary OpenID key are stored in Redis, and the application is built to recover gracefully if any of these (or all of them) are lost. We run with allkeys-lru enabled and about five times as much space as its actual working set needs, so Redis automatically discards data that hasn’t been accessed lately, and reconstructs it when necessary.

Our most interesting use of Redis is in our short-polling fallback for sending changes to Models down to browser clients. When an object is changed on the server, we send a JSON message down all of the appropriate WebSockets to notify those clients, and store the same message in a fixed-length list for the affected model, noting how many messages have been added to that list over all time. Then, when a client that is on AJAX polling pings the server to see if any changes have been made to an object since its last poll, we can get the entire server-side response down to a permissions check and a check of a single Redis value in most situations. Redis is so crazy-fast that it can handle thousands of these checks per second without making a substantial dent into a single CPU.

Redis is also our pub/sub server, and we use it to propagate object change messages from the server process making the initiating request to all of the other server processes. Once you have a Redis server in place, you start using it for all sorts of things.

MONGODB

MongoDB fills our more traditional database needs. We knew we wanted Trello to be blisteringly fast. One of the coolest and most performance-obsessed teams we know is our next-door neighbor and sister company StackExchange. Talking to their dev lead David at lunch one day, I learned that even though they use SQL Server for data storage, they actually primarily store a lot of their data in a denormalized format for performance, and normalize only when they need to.

Trello Today

In MongoDB, we give up relational DB features (e.g. arbitrary joins) for very fast writes, generally faster reads, and better denormalization support — we can store a card’s data in a single document in the database and still have the ability to query into (and index) subfields of the document. As we’ve grown quickly, having a database that can take a fair amount of abuse in terms of read and write capacity has been a very good thing. Also, MongoDB is really easy to replicate, back up, and restore (the Foursquare debacle notwithstanding).

Another neat side benefit of using a loose document store is how easy it is to run different versions of the Trello code against the same database without fooling around with DB schema migrations. This has a lot of benefits when we push a new version of Trello; there is seldom (if ever) a need to stop access to the app while we do a DB update or backfill.
This is also really cool for development: when you’re using hg (or git-) bisect and a relational test DB to search for the source of a bug, the additional step of up- or downgrading a test db (or creating a new one with the properties you need) can really slow things down.

So we like it?

We like our tech stack. As Joel observes, we’ve bled all over it, but I’ve never seen a team make an interesting app without tool- and component-related bloodshed, and not everyone can say that they really like what they’ve ended up with. As is true of most applications, no component or implementation detail is necessary to its nature; however, we think that this excellent set of open-source projects has sped up our development, left us with a solid and maintainable code base that we’re eager to move forward with, and made Trello a more responsive and beautiful app. Thanks to everyone who has contributed to them; it’s a great time to be a programmer.

Sound neat? Try Trello! It’s free.

Just can’t get enough tech stack talk? Here’s a Prezi I made for a recent talk on Trello.

* The Socket.io server currently has some problems with scaling up to more than 10K simultaneous client connections when using multiple processes and the Redis store, and the client has some issues that can cause it to open multiple connections to the same server, or not know that its connection has been severed. There are some issues with submitting our fixes (hacks!) back to the project – in many cases they only work with WebSockets (the only Socket.io transport we use). We are working to get those changes which are fit for general consumption ready to submit back to the project.

Hold onto your hats. There’s a lot to like.

The Year in Review

We had a tough act to follow last year. In 2010, we launched Kiln On Demand and Kiln for Your Server, shipped piles of new features, and found ourselves home to tens of thousands of users’ source code. But Kiln was new at the time; most of those of you who came to us were already familiar with distributed source control in one way or another. Kiln’s mission is to bring distributed version control to as many people as possible, and that means appealing to more than just early adopters.

To make that happen, we made a giant list of what was keeping people from jumping to Kiln, and we spent nine months fixing every issue one by one. We rewrote the review system to be more user-friendly (by overhauling the UI and making it easier to make cases and turning reviews into multiparty discussions instead of one-on-one chats). We helped Kiln integrate tightly with lots of other services so you could keep your existing infrastructure. We allowed configurable diffs, we taught Kiln how to display images in your repositories, we beefed up the API, we contributed largefiles (based on kbfiles, our extension that makes it trivial to have large binary files in Mercurial) back to the main Mercurial project, and we added a full-blown groups permission system so you can have large Kiln installs without getting lost in a quagmire of user management.

The result? Kiln 2.7, which we think gets even those who don’t have a lot of experience with DVCS easily using distributed version control, rather than getting lost in a muddle of tool arcana.

What’s to Come

“First nine months,” I hear you say. “Fascinating. That might be an impressive list. But the year is twelve months long, so what have you done for us lately?”

We have a strong rule against publishing road-maps for products, for a simple reason: we don’t like to disappoint you. We hate promising you’ll have some awesome new feature in Kiln 45.7, like telekinetic abilities, and then smashing your hopes most excellently by announcing we’ve had to delay them until 45.8. Instead, we prefer to surprise you each release with piles of new goodies. A kind of monthly Christmas for Kiln, if you will.

Of course, the flip-side is that we know you can feel abandoned when we go dark for awhile, even if we’ve got a really good reason for doing that. And I think we’ve been too quiet lately, so some of you are wondering if we’ve been run over by a bus.

Good news: we’re still very much here! And we’re still working hard. Unfortunately, our current projects are harder than our old ones, so we haven’t had much to show for it for a few months.

To fix that, I want to welcome you behind the curtain, and introduce you to three features coming your way in the next few months: the Kiln Client for Mac OS X; SSH support; and vastly improved search and general performance across the entire app.

Kiln Client for OS X

I admit it: Kiln may run on Windows, and I may love writing code for .NET, but at home, everything I own runs OS X. So it frustrates me that the Kiln Client is only available for Windows.

That’ll be changing shortly. Coming soon to a Mac near you: Kiln Client for OS X.

While it looks and works similarly to the Windows client, since both are based on the excellent TortoiseHg, we’ve taken the time to make it work like a real Mac app. You install it by drag-and-drop, and it takes care of the rest. It’ll automatically put Mercurial on your path, it’ll automatically update itself when there are changes, and otherwise behave just like a good Mac citizen.

SSH Support

While we love Mercurial over HTTP, the simple truth is that it’s not perfect for everyone. HTTP-based pushes put you into HTTP timeout hell, where you have to make sure your timeouts are correct at every level of your network (load balancer, web server, web site in IIS, and so on). For Kiln On Demand, that’s okay, because we control the full infrastructure. For our licensed customers, this can be a serious problem.

Good news: SSH suffers none of these issues, and Kiln is adding SSH support. And not just for On Demand; it’ll be available for licensed customers as well, as a single, easy-to-use, turnkey solution.

Improved General Search

Search in Kiln is a big deal, and we love it and we know you love it, but we want to be even faster, and we want it to be better with non-Windows-compatible file names.

So we’re rebuilding Kiln’s search story around Elastic Search, an excellent NoSQL search solution. We’re currently still heavily in the development stage, but already, our search times are down from a second or two for very large Kiln installations to just a couple of milliseconds. We think this will be a big deal for our licensed customers, and a huge deal for our On Demand customers.

We’ll have this work on Kiln On Demand in a couple of months, and for Kiln licensed customers about a month later.

So we believe there’s a lot to love in the coming months for Kiln. And we hope that this roadmap of what we’re doing, and the rough timeline we plan to do it in, will get you as excited as we are.

In the meantime, happy Kilning.

But why?

This is a practice so deeply ingrained that almost everyone assumes that commissions are an unalloyed good, and that salespeople won’t work without them. I’ll return to that notion about work shortly, but it’s somewhat amazing that commissions are so widely lauded when they come laden with so many recurring problems. These issues pop up with distressing regularity.

There are all kinds of problems with commissions, for example, high turnover as salespeople shop jobs to get a slightly more lucrative commission system. Always attempting to maximize personal benefit which results in system gaming like making fake phone calls to hit call numbers, sandbagging deals into the next quarter, sniping new leads, and so on (the list here is actually endless).

The problems include infighting over who gets credit for accounts and sales. They include constantly comparing territories and account value to determine fairness between salespeople. They include an enormous amount of overhead as each salesperson sedulously tracks every transaction no matter how minute to make sure they get paid on it (by the way, they hate having to do this, and it’s a staggering waste of time. It’s also a place where weak salespeople like to hide out).

All of this is organizational dysfunction, and it’s a recipe for resentment and distrust among your team.

Management then tries to correct for these problems. They constantly drop or add ballast. They have to carefully structure the pay plan, the territories, the lead assignments. They have to referee disputes, tweak the various systems, and try to keep everyone happy. It’s like a spinning top and every time it starts to wobble, management has to try to nudge it back. It’s a large amount of effort spent propping up a system that we have all just assumed is necessary.

But it gets even worse.

In research by Dan Ariely and others it appears that higher incentives, actually reduce performance. That’s a perverse and counter-intuitive result, but in several different kinds of experiments, groups that were promised the largest amount of money as a reward for doing a task performed that task more slowly, and completed the tasks less often.

And yet the paladins defending commissions are everywhere.

They say that commission-based pay maximizes autonomy. Assume there are ten “units” of pay which, of course, can be divided up in all kinds of ways; eight units of salary and two units of commission, or one unit of salary and nine units of commission. How best to do this is the source of several different kinds of holy war.

This means, they say, that for every unit of base salary that gets added to the pay mixture in place of commission there is an increase in employer control, or put the other way round, a reduction in autonomy. More salary in the mix destroys the space for independence and kills the morale of the worker. It turns motivated independent free-agent sales people into wage slaves.

But it gets even worse, according to the defenders of commissions, and here is the crux of their argument.

They say that without the potential to make extra money for your efforts—that is without a one-for-one relationship between a piece of work and the resultant money—that there is no real motivation to work, or, god forbid, to do any extra work. How do you keep salespeople hungry if you don’t pay them commissions? This is the great inscrutable puzzle in sales pay. Too much salary and not only do salespeople lose their independence they become slugs on top of it.

But hang on a second. Isn’t all this deeply insulting to salespeople? Doesn’t it presuppose that salespeople are lazy and greedy and unethical and that it is only the sweet smell of more lucre that gets them to shed their pajamas each morning?

According to this view sales people are only motivated by a specific type of pay—the commission, and the magic is in finding, like Goldilocks, that place that’s just so perfectly just right.  Because of this we regard all the pathologies commissions create as just part of the price.

But could it be possible that we’ve got cause and effect reversed here? Is it possible that the stereotype of the slimy salesperson, and the derangement of the culture they so often get blamed for, are actually the result of the way they are paid instead of the kind of people they are? Could commissions, rather than the people, be the primary cause of dysfunctional sales cultures?

Think for a second: we don’t insist that other kinds of workers be paid on commissions. Only an amazing idiot pays a programmer by lines of code. We don’t assume that programmers will dog it if they are paid only salary. Actually, we think just the opposite; they will work hard because they care about the work they’re doing, and they won’t need lashings, or thumbscrews, or other popular forms of motivation. We assume they are internally motivated. This is one of the hallmarks of the enlightened software industry. Development isn’t (usually) conveyor belt work. It doesn’t (usually) suck the soul out of you. It’s creative, it’s mind work and it operates on an entirely different rhythm from traditional kinds of labor, so heavy external control tends to be counter-productive.

And this is true of other workers today—designers and writers and interaction experts, for example. They work on this different rhythm, and are trusted to do so, not coerced by the style of pay they receive, and in good work places, not too often by a manager either. A writer may impose a rigid schedule on herself, but any manager who attempts to impose a similar sort of external authoritarian control is suffering from a kind of insanity given how out of synch such management is with the demands of a lot of today’s work. And so it is with attempts to control with pay.

The different pay systems, then, leave us with two kinds of workers; the lazy salesperson who needs to be coerced with direct rewards, and programmers who can be trusted to go about their day and get good work done. It’s just weird when you think about it.

But, these are actually two different views of workers, not necessarily two kinds of workers.

The tension between these views of workers was described in the 1960s by Douglas MacGregor in his book The Human Side of Enterprise. He suggested that managers had two views of motivation, and that a manager’s theory of motivation determined company culture. The first view he called Theory X which assumes that people are lazy, want to avoid work and need to be controlled, coerced, punished, and lavishly rewarded in order to perform. Sounds like some sort of S&M dungeon to me. Theory X demands a lot of managerial control and tends to demotivate, generate hostility, and generally make people into sour pusses.

The second he called Theory Y which assumes that people are self-motivated, derive satisfaction from their work, are creative, and thrive when given autonomy.

With these two views in hand, we now have a way to describe why sales commissions create so much dysfunction. Commissions assume a Theory X world. It assumes salespeople are lazy. They need external motivation to shake off the moss. If you pay them on salary, they just won’t get things done. The commissions view then makes the mistake of presuming that this can be fixed with either larger overall commissions or a greater percentage of total pay coming from commissions.

But if Theory X doesn’t fit and is a degrading way to treat employees, then doing more of it gets you nothing but more degradation and misery, right?

The Theory X way of doing sales compensation, I now think, has habituated us into accepting deranged and dysfunctional sales behavior as if it’s just the cost of doing business.

We Get Rid of Commissions

So we got rid of commissions.

We thought about for a long time before we did it, but we finally switched about a year ago.

We did it because we were having a lot of the problems with commissions described above even though all of our salespeople are ethical and decent. Commissions just encourage certain kinds of behavior; dysfunction is built into the logic of the system.

The different kinds of  pay created divisions at Fog Creek. It was the fundamental thing that separated sales from everyone else. Divisions like that can be exceptionally corrosive to morale, and that’s something Joel and Michael (the founders) specifically set out to avoid when they designed the Fog Creek compensation system. Fairness was one of their main concerns, just as it was in the newer StackExchange comp system, and they felt that commissions just didn’t fit.

I’ll admit that I was skeptical about ending commissions mainly because commissions are such an established way of paying salespeople. I worried that we wouldn’t be able to hire anyone. I worried that some of our sales people might quit. I worried, good Theory Y votary that I was notwithstanding, that the salespeople would coast. I worried when my friends in sales management said this would never work.

To my great surprise, our salespeople really liked the idea. They especially liked being on the same salary plan as the rest of the company.

So we did it, and no catastrophes struck us. No earthquakes. No plagues, and no one quit. In the year since we dropped the commission system our sales have gone up. In fact, four of the last five months have been record months. We can’t reasonably say that our record sales were caused by this change, but we can reasonably say it didn’t hurt, and that’s worth having a hard think about in your own company. There is no guarantee that this will work for everyone, but it’s unlikely to be a disaster either.

Our salespeople all estimated that they were spending about 20% of their time just keeping track of what money was due them. There was constant horse trading. And, most worrying, we created a heavy disincentive to do all the service stuff that makes customer service shine. Why would you want a system that sets up after-sales service as competition against new sales, especially if you have a small sales team? Reputation and retention, after all, are both paths to revenue.

Removing commissions has changed the sales team. It has taken their focus off their compensation. They have all that administration time back for more useful things. They take a longer view of the value of a prospect, and are less worried about who is going to buy right now. They feel less stress about taking vacation. They don’t quibble among themselves over accounts. And best of all, they feel more integrated with the company.

As John, one of our salespeople said, “It’s made the team better. It’s removed the ‘me, me, me’ mentality. Now I want to share information with everyone on the team, and everyone is willing to pitch in because it doesn’t hurt me to help my colleagues.”

Getting rid of commissions lets us forget about policing the wobble. Now, it is not necessarily the case that commissions are always bad, or always fail, or are wrong for you. It’s just that they come with real problems, and you need to carefully weigh both your desired company culture, and the costs of policing your commissions system against the expected increase in performance, which is a very hard calculation to make.

For us, it’s been a great success, and at least from that perspective it might be time we punch the Theory X, commissions-based sales culture right in the nose. Real redemption might lie in removing the source of the derangement and treating sales people like we treat programmers and other workers that we implicitly trust.

There is always something that feels more pressing. Always some technical debt to pay down. Always a bug in some super wonky edge case, But It’s Happening To A Very Important Customer, and needs to get fixed. In short, we always can find a very reasonable and important reason to put off the design changes that pretty much everyone here wants to make.

But as everyone who has been near the internet or heard of Steve Jobs will tell you, design is just as important as features—possibly more so in some cases. In the same way that we’ve come to regard our website as a product, we also know that design is a feature that deserves equal billing with more pedestrian features. It’s not that we didn’t know this—we’ve always cared a great deal about it, and in fact, Joel wrote a book about it—it’s just that we haven’t had the time or the resources to devote to it in some time.

Tastes change. Technologies change. Sometimes even established practices and affordances change. It’s entirely possible to build up what we might call design debt in the same way that software can accumulate technical debt. And FogBugz has some design debt to pay down at this point. It’s time to scrape some of the mold off the bread.

But design changes are tricky. They alienate power users.  If things are mysteriously moved you have to be sent to a Microsoft re-education camp (to learn where all of your commonly used administration tools absconded to…again). You’re just as likely to make things worse as you are to make them better, especially if you’re doing a lot of guessing about what people want and how they behave with your software.

So this weekend we’re rolling out some design changes to FogBugz. We’ve tried to balance updating the look without requiring the cumbersome re-training camps. There are some very subtle changes on the gridview page but these are mainly making menu items a bit bigger, making fonts more consistent, and some color changes. You probably won’t even notice these changes.

The more interesting changes are on the case view page.

Here is a pic of the current (soon to be old) case view.

Here is a pic of the new case view.

There were a couple of things we tried to do. First, we just wanted to make it look a bit fresher and more modern, so we changed the icons, using some of the icons from the very beautiful Fugue Icon Pack.

Next, we felt it was important to make certain kinds of information stand out better. You might occasionally want to know when a case was assigned to someone, but more often the content of the case is much more important. Given that, we de-emphasized the information around administrative minutiae and tried to make the comments of a case more apparent.

If you send an email or have a case that comes in via email (in other words, the case has a correspondent), you’ll notice a nice little air mail bar along the top of that portion of the case. That makes it clear that you are dealing with or sending an email and distinguishes it from some other normal case edit. You’ll never accidentally send an email from FogBugz again. This one deserves a little more comment. At the risk of indiscreetly tooting our own trumpet, I’ll say that this was one of those occasional moments of perfect design inspiration. I didn’t even know it was going to be there, but the first time I saw it I understood instantly what it meant. Without the need for a text label, or an explanation, or some other heavy-handed means the function is totally clear. It’s an “I could have had a V-8” moment. Seems so obvious after the fact.

Cases have always contained status information, but you had to read text to know what it was. That’s OK, but we wondered if it would be better to give a visual cue so that this information could be absorbed at a glance. Now the status of a case is given a color in addition to the text. Green is active, blue is resolved, and gray is closed.

We’ve also done some odds and ends with color and more consistent font choices and the like.

Overall we’re pretty pleased with the balance we struck between freshness on one hand and consistency on the other.

We were able to do this because we moved a talented support tech over to the design team, and he made these changes over the course of a couple of weeks. But, alas, the commute was making life with his new child hard to manage and he left Fog Creek. Everyone on the FogBugz team was thrilled to have him, and we’re committed to making more changes to the FogBugz interface to make it more modern and responsive and generally more pleasing.

So this post ends with a request. If you’re a good designer, or know one who wants to work at a great software company in New York City, please send them our way. Check the Fog Creek careers page for the job posting.