Is this because:

• Nobody needs anything other than Google Website Optimizer?
• Startups don’t actually do AB testing, possibly because they don’t get enough traffic to get meaningful results, or maybe because they don’t have time?
• AB testing (including the statistical analysis to determine if results are valid) is so simple that everyone just bangs out their own?
• As a largely theoretical issue for most startups, scalability is more fun to talk about on the Internet?
• Everyone that is using AB testing is so happy that they are trying to suppress information about it so their competitors don’t start doing it too?

If everyone is secretly using some great framework please shoot me an email and let me know.

If you haven’t thought much about it before, here is a short paper on AB testing from some folks that made Amazon a ton of money.

• A clever proposal from Google: Shared Dictionary Compression over HTTP
• Cacheset – a tool for clearing the windows disk cache (useful for testing cold starts).
• Fun fact: the Tesla Roadster carries 3 milligrams of electrons when fully charged.
• The ultimate Airplane on a Treadmill debate resource: www.airplaneonatreadmill.com
• A 728-ton tuned mass damper in a skyscraper

But, I don’t think I’ve reached the critical mass of followers necessary to really unlock the Q&A potential of the site. Having a few hundred technical folks all following each other would be a tremendously useful resource for everyone involved. For example, I’m considering upgrading my desktop to 8 or 16GB of RAM. I’m going to need a new motherboard, processor, and RAM. My normal approach for this would be to spend a few hours on Newegg and the hardware review sites trying to figure out where the price/performance curve is and making sure I’m not getting ripped off. If someone else has done this same research it would be nice to use their information as a starting point, and twitter provides the kind of free-form conversation necessary for that kind of sharing.

To really make this work, you need to run one of the desktop apps so you don’t have to constantly reload the website (I use Twhirl).

Positive Feedback
Being told that I only spent 10% of my day doing work is good to know, but getting a low number might depress me rather than motivate me. I suggest a system that actually rewards me when I have a killer day or a great week. For example, I give the service $25 or $50 up front, and after I meet some sort of goal it buys me something off my Amazon wish list.

Wouldn’t that be neat? You’re having a good week, and suddenly a book you want shows up at your door. The key to this is making the rewards somewhat random:

Several studies have been conducted which targeted neural response to rewards. The results were unanimous in the fact that when one performed an action over and over again, and was given a reward randomly, dopamine levels rose. If the reward was given consistently, i.e. every four times the action was performed, the dopamine levels remained constant.

A slight variation that might work better would be for each contiguous block of productivity over a certain length, you have a chance of earning a credit towards a purchase. After N credits, the service automatically buys and sends you the item. Structuring it like this would make the feedback more rapid and allow for a little burst of dopamine each time you get an email saying you earned a credit. Isn’t this why MMORPGs are so much fun?.

A program to help you get addicted to work is either terrifying or a big win. Either way, it would be really neat to try.

Targeted Recommendations
Some software is just better than the default stuff that ships with Windows. For example, I like Textpad and Paint.net a lot more than Notepad and MS Paint. I’ve also been pleased with my switch from Bloglines to Google Reader, and from web based twitter to Twhirl. If a program spends all day monitoring my activity, it would be a cinch for it to recommend the tools and websites that are considered “best in class.”

There is obvious potential here in terms of sponsored recommendations, but it would be nice to see those separated out from community or editor controlled listings. Recommendations could be driven by some sort of wiki, which would make for all kinds of interesting fights over things like whether Google Reader is better than Bloglines. Any recommendation could also come with an estimate of the number of people that are currently using it, which would help the cream rise to the top.

Ultimately, it’s not just about how much time you spend slogging away – making good use of computer time is an important dimension of productivity as well.

A slight variation on this idea is to recommend something like LeechBlock if the user is spending too much time on the web.

Attention API
I know I’ve seen this idea somewhere before, but because things like RescueTime are actually in a position to make it happen, I’m going to mention it here. Interruptions are usually bad, but there are some times that they are worse than others. If I’ve been focused on Visual Studio and Windbg for 30 minutes with no breaks, I’m almost certainly in that fascinating “flow” state, and I’m going to be angry if I get an IM or (even worse) if some random app asks me to download a new version.

To deal with this kind of thing, it would be great to have a standard for publishing my current tolerance for interruptions, just like IM apps publish my presence. Both desktop apps and remote users could use this to determine if what they want to tell me is important enough to interrupt me. Of course, this only works if apps pay attention to it, but first we would need some apps that can accurately measure it. I’m not terribly good with naming things, but unless someone has something better, I’m going to suggest calling this value your “inTolerance”.

So there you go. One idea to help you spend more time being productive, another to help you make better use of the time you are actually working, and a third to keep you from getting interrupted. Anyone want to go and implement this stuff? I’ll be happy to beta test it for you.

But ultimately, better movie recommendations don’t matter a whole lot to me. I’m more interested in the fact that by providing a unique set of data and a prize, they’ve been able stimulate so much interest. The other day I was thinking about which companies are in a position to sponsor contests in other fields that might have a bigger impact on my life, and one thought jumped into my head – Google’s 411 phoneme collection service. Marissa Meyers says:

You may have heard about our [directory assistance] 1-800-GOOG-411 service. Whether or not free-411 is a profitable business unto itself is yet to be seen. I myself am somewhat skeptical. The reason we really did it is because we need to build a great speech-to-text model … that we can use for all kinds of different things, including video search.

The speech recognition experts that we have say: If you want us to build a really robust speech model, we need a lot of phonemes, which is a syllable as spoken by a particular voice with a particular intonation. So we need a lot of people talking, saying things so that we can ultimately train off of that

Presumably, Google has already done the heavy lifting to manually transcribe a large number of these samples so that they can train their own algorithms. Why not create a contest that lets teams submit an algorithm that gets trained on a subset of the data and then tested against the rest? Speech recognition is more complicated than movie recommendations, but making it easy to train and test an algorithm against an interesting number of samples would certainly lower the barrier to entry.

Google would benefit from this in hiring, if nothing else. It would give them a chance to realistically evaluate the work of all kinds of grad students and researchers, and demonstrate to the candidates the advantages of working for the company with the biggest databases.

Design the system to never need human interaction, but understand that rare events will occur where combined failures or unanticipated failures require human interaction.

Yes, designing the system to never need human interaction is a great ideal to shoot for, but when you are working for a startup with three guys and a dozen servers, you don’t have the resources or the justification to do it from Day 1. It is entirely likely that your business model will fail before you lose a single disk. And since backend refinements don’t pay the bills at a small scale, something with a pair of hands is going to be interacting with your system until you get enough people and servers to justify more automation.

These events will happen and operator error under these circumstances is a common source of catastrophic data loss.

That is a wonderfully simple and accurate summary of How Bad Things Happen in your datacenter. It starts when you lose a hard drive or MySQL crashes, and you have to promote your slave until you can check the master tables, or anything painful but routine happens. But then, as you are trying to fix things, you notice, for example, that you are almost out of disk space. When you start trying to fix more than one problem under pressure, you are entering a world of pain.

The big issue here as James points out is that you are going to do something wrong. You’ll probably use a much stronger word than “wrong” once it is all over, but let’s settle on “stupid” for right now.

It won’t feel stupid until after you hit “enter,” but when you are making unfamiliar decisions quickly under pressure, you are extremely likely to overlook something. Maybe you won’t shut down mysql before you start a myisamchk from the shell, or maybe you’ll reverse the arguments to “tar -cvzf” and wipe out something important. Or perhaps you’ll screw up a firewall rule and block ssh access to the machine you are frantically trying to fix. Accidently killing the ssh daemon is another favorite. The point is that during a stressful situation in the datacenter, the human operator is the biggest potential source of more downtime or “catastrophic data loss.”

Assuming you can’t automate everything, what can you do? Well, the absolute best thing you can do is practice. Corrupt some data on your dev master db, and see how long it takes you to get it restored from backups or a slave copy. Practice what would happen if you lost a database slave and had to activate a spare machine to take its place (I hope you have at least one spare machine). But of course, no one at a small startup has time to practice. Maybe once you hire a full time ops guy, it would be good to make sure he is practicing this sort of thing occasionally. But when practicing is going to take away from writing code, you aren’t going to practice.

Since you aren’t going to practice, what else can you do? The next best thing is to cultivate the attitude that you are the most likely source of problems. Don’t worry about hard drives, worry about bad decisions. Develop some humility about how you expect to behave when you get woken up at 4am to fix a database the morning of your launch or when a switch fails an hour before your big demo. From that mindset, here are a few things to do:

Script what you can
Off the top of my head, a good place to start would be writing scripts for some of the steps in setting up master/slave replication and manipulating firewall traffic (allowing or blocking external traffic, for instance).

Use the buddy system
It is not a bad idea to have somebody else there looking at what you are typing, or at least on the phone confirming things verbally.

Take your fingers off the keyboard before you hit enter
Are you in the right directory? Are you on the right machine? Are those arguments in the right order? Can you just rename this old stuff instead of deleting it? All of these are excellent questions to ask yourself or your coworker while you have your hands in your lap. This is also a good idea when you are doing something scary with SQL, like running any query that doesn’t have a where clause.

Slow things down
As soon as you make one mistake, no matter how minor, it is time to slow things down. Beyond the fact that making a mistake will fluster you, making one mistake demonstrates that right now, you are likely to make mistakes. That is a huge red flag. At this point, the safest thing may be to accept a slightly longer downtime just so you can slow things down, get some water, and relax. Trying to compensate for a little mistake by doing things faster can result in a much, much worse mistake. Unless you’ve just rolled a server cage down the stairs, there is always a worse mistake you can make.

Make it hard for people you work with to make mistakes
A quality server naming scheme is the easiest thing you can do here. No colors, deities, countries, snack foods, snakes, etc. I like $machineType-$number myself, but with distinct number ranges, even between different machine types. So, don’t have SQL-001 and Web-001. One day, some very sleepy datacenter employee may get things mixed up when you call and ask him to reboot Web-001. I’m sure you’ll get an apology, but you won’t get your uptime back. So make it harder for him to screw up: if your web machines start at Web-201, he’ll have to make 2 mistakes before he accidently reboots your primary database.

Talk about this stuff ahead of time
You probably have plenty of stuff to talk about at lunch with your coworkers, but here are a few convers ation starters if you want to sharpen your disaster recovery skills:

• “What happens if we lose power to one of our racks?”
• “How many of our switches could we lose and get the site back up?”
• “What is the smallest amount of hardware we could lose that would knock us 100% offline?”

This stuff isn’t theoretical. I woke up at 2am one weekend during FolderShare with a ton of text messages from our cluster. The kindly folks at the datacenter had been doing power supply maintenance. At some point, they powered down 2 of our racks. Then, they powered them right back up. It wasn’t tough to fix, but it was so unexpected that it took me a few minutes to even realize what had happened.

Use tricks to deal with the general class of “running a command on the wrong machine” problem.
Typing the right command on the wrong machine is obviously something to avoid. But when you have a sea of ssh windows open, what can you do?

• Use a different color background for your terminal to machines hosting master databases versus slaves
• Make sure the machine name shows up in the command prompt

Does anyone else have any good ideas or horror stories to tell? Post a comment and share your wisdom and/or pain.

• Am I getting more than I need? It pains me to add a multi megabyte DLL to a client download for a small amount of functionality.
• Will I spend more time learning the interface than I would writing the functionality I need myself?
• Is this an active project, and is there any documentation?
• If scheduling isn’t an issue, how much fun would it be to write my own version?

Next comes a set of questions that are oftentimes harder to answer:

• Who else is using it?
• Will I be using it the same way as other people who are successfully using it?
• What am I going to find out when I put more stress on it than anyone else?

One library that passed my gauntlet of questions is libredblack. It ended up on a bunch of production servers at FolderShare, and it worked out great. But there was a catch: I wanted to use it to store large numbers of items, but for every item I put in the tree, the library would allocate an object that held 4 pointers and an enum. That took 40 bytes on my dev box. Throw in malloc’s overhead, and I was up to 48 bytes. The objects I was storing pointers to would also have some heap overhead, which may be as much as 24 bytes. So to store 10M items in memory, I’d need an extra half gigabyte of memory just for overhead.

A second example from personal experience is librsync. Again, the library works exactly as advertised. But if you want to transfer deltas for large (gigabyte+ files) on machines that have hard memory limits (like embedded devices), you need to know that the memory usage is proportional to the file size. For my situation, I ended up having to adjust the window size as file sizes grew just to keep the memory usage reasonable for large files.

I don’t want anyone to think I’m complaining about this stuff – I’m a fan of both libraries. But both of these examples illustrate a class of problem that is particularly frustrating: the one you might not find until you are heavily invested in a solution. These gotchas won’t affect most people, and thus aren’t likely to show up when you are researching possible solutions. They aren’t bugs, either, but they might be something you have to deal with. So the sooner you can find out about them, the better.

Fortunately, the internet has plenty of software built for solving problems like this. Dev Diligence^[1] is a new wiki I’ve started to collect details like these. My goal is to have a reference page for any library or service developers might consider using in their solution. For sufficiently large libraries, pages for classes or functions might be necessary, but let’s not get ahead of ourselves. Ultimately, I’d like to have 5 headings for everything in the wiki:

• Overview: Brief description of the software and a link to the homepage
• Short case studies or war stories: These would include a brief description of how you are using the software, the version you used, and ideally some metrics. If you used it for a while and then switched to something else, an explanation of that decision is very valuable information. For libredblack, the relevant metrics would be things like average number of elements in your trees or insertions/deletions per second.
• “Gotchas” (like the ones I’ve mentioned above): Subtle problems (hello, heap fragmentation) and things that aren’t necessarily bugs, but issues that may affect your design or help you choose one solution over another.
• Alternatives: The name pretty much says it all. With links, please.
• Other Resources: Links to blog posts, email threads, or reference pages would be great.

I’ve gone ahead and created entries for libredblack, librsync, and zlib based on my experiences. I’d love to see some entries for the following and things like it:

• libev, libevent, boost.asio, and Twisted
• openssl
• sqlite and berkeleydb
• memcached, spread, the reliable queue solutions (Starling, TheSchwartz, etc), and anything that uses “pubsub” in its description
• libcurl and wininet (stuff like Nick Bradbury’s description of a CPU spike in WinInet that can be triggered by chunked-encoding is gold)

All of these and more are linked to from the WishList page.

Can you guys help me out? I’ve got enough people subscribed to this feed that I’m certain at least one of you has used everything on my list. If you take 10 minutes to write down your experiences, you can make the software world a better place. To justify doing it on your company’s time, keep this in mind: if you document the fact that you are successfully using a solution, you increase the chance that other people will use it as well. The more users a solution has, the the better it will become.

[1] “Dev Diligence” is of course a play on the term “Due Diligence”.

I originally fell in love with assertions after reading Steve Maguire’s Writing Solid Code many years ago. After I saw how helpful they could be, I started to structure my code to make it more “assertable.” For example, I like state machines that use a table of valid transitions combined with assertions. This prevents anything I don’t explicitly anticipate from happening and is really helpful in a networked, asynchronous world.

But after developing a few long running services, I’ve started to perceive the need for a new type of assertion that is a bit higher level than a single conditional. I want something that will let me crash a service (and save a memory dump) when something “feels” wrong. For the moment, I call these “probabilistic assertions,” and I would have slept better while running FolderShare if I’d implemented them then.

Like all synchronization software, FolderShare had a few nightmare scenarios that I worried about all the time. If, due to a bug, the service told every client that its files were all deleted, I’d probably have to read a lot of nasty blog posts, and I would feel like crap for a few years. I would much rather crash all my servers and debug the problem with the system offline than take a chance on pissing off so many people.

Anyway, a normal assertion that looks at a single conditional wouldn’t help in this case. Asserting that no files have been deleted when a client connects doesn’t make any sense. But for a large enough sample size, I could assert that 80% of clients that connect shouldn’t have any files deleted. And I could probably assert that 95% of clients that connect shouldn’t have all their files deleted.

Functions like these cover most of what I’m thinking about:

Depending on your application, these functions could be implemented either as assertions or as triggers to alert an administrator. For alerts, you could really take this to another level by checking that incoming events are following a certain distribution for example. Granted that it’s a bit over the top, but I can’t help but imagine checking if a stream of incoming events obeys a Poisson distribution or if the sizes of certain data are following a Gaussian distribution.

Has anyone seen anything like this in the wild? I’d love to hear about it.

Audiogalaxy had a small client that simply handled the P2P transfers and a complicated website for everything else, including account settings. One of the adjustable account settings on the website was the “max number of transfers.” To encourage users to send as much as they received, we only gave them a single number for this setting. With a value of 1, a Satellite would only send a single file at a time, but it could only download one file a time as well.

Things were not so simple on the back-end. For better or for worse, I had designed some flexibility into the system. The max transfers value was actually stored in two columns in the Users table – MaxSend and MaxRecv. The back-end – the part that actually looked at these values when it was setting up transfers–had no idea these columns were linked. The front-end enforced what went into the database, and the back-end obeyed it. Whenever the Satellite reconnected to the cloud, our server would read the value out of the database and store it in memory for the duration of that connection.

Of course, somewhere between the frontend and the backend is mysqlclient, but I’ll get to that in a moment.

Quake III Arena was my game of choice at the time I worked for AG. We had a few developers that also enjoyed the game, and it was common to find people staying late on the weekend to take advantage of our nice internet connection. Unfortunately, our nice internet connection had a dozen people running our p2p music sharing client on one side, so it would periodically slow down when someone’s computer started blasting a file out at high speed. These slowdowns drove us crazy, particularly when they prevented us from using the game’s rail gun effectively.

Good developers like to fix problems, and developers at startups also tend to have access to the database. So, you can probably imagine what a developer might do. And if you know a little bit about SQL, you can also imagine what might go horribly wrong. I never found out who issued the bad query, but I can just imagine how it played out:

Hey, I’ve got an idea about how we can keep the games from lagging tonight. I’ll just block everyone in the office from sending files. One simple ‘Update Users set MaxSend = 0′ and we should be good to go for the evening… Why is that query taking so long? Uh oh…

SQL is good for a lot of things, but I’ve always marveled at how easy it is to destroy an entire table simply by forgetting a where clause. And thus, in a few short minutes, every one of our 30 million users had a subtle change applied to their accounts. Did I mention that the single value we displayed on the website for this setting came from the MaxRecv column? Whoops…

Monitoring the health of the system was one of my jobs, so I kept a close eye on my graph of the “current transfer rate.” Ultimately, most problems in the system resulted in less files getting transferred, so the global transfer rate was a good proxy for the health of the system.

Every day of the week plotted a unique and predictable curve that I knew by heart, and so it didn’t take me long to realize that something was wrong. Transfer rates were dropping. But why? I called our ISP and asked if they knew of any problems with the Internet. Nope. We had exactly the right number of clients connected. No one had trenched over a fiber optic cable in the middle of nowhere. Requests were coming into the system at the normal rate; they just weren’t getting fulfilled. Microsoft hadn’t pushed any patches out that might have firewalled off half the world.

Clients generally stayed connected for days or weeks at a time. As they gradually reconnected, more and more of the network got their new MaxSend setting and dutifully started not sending anything. Users weren’t complaining – it was perfectly normal for rare songs to be inaccessible, and nobody noticed if his client just wasn’t sending anything.

After tearing my hair out for a day or so about this, I finally realized I was seeing a lot more “client busy – no free slots” type messages than I usually did while tail –f’ing the log files. Digging into that, I noticed some other funny messages, and eventually I was staring in shock at the results of a “select MaxSend, MaxRecv from Users limit 1000.”

Fixing the problem was easy enough: “Update Users set MaxSend = MaxRecv,” but you can imagine I spent quite some time staring at that query before issuing it.

So what’s the moral of the story? Don’t let your developers have access to the production database? Maybe, but that isn’t practical for a small startup. Better logging? That certainly could help. Force everyone to access the database using the –i-am-a-dummy flag for MySQL? That is not a bad idea and will get you some of the way there, but a shoddily written script can do exactly the same kind of damage. Backups? Sure, we had backups, but we were adding customers so quickly that restoring data more than a few hours old would have pissed off many thousands of people. An Admin class of users, with configurable policy that prevented them from sending files between 7pm and 3am on weekends? Yeah, right.

If you run a big and complicated system, problems you will never predict are going to happen and cause your system to do impossibly weird things you don’t expect. You must invest in tools to give you visibility into your system. My transfer rate graph was the only reason I was even able to go looking for a problem. I knew something was wrong, and it was just a matter of digging until I found it. Let your admins see into the system (specifically – how the system is behaving right now) so that they can develop intuition about what it should look like. Finding a bug in production is never fun. But it is going to happen, and it is always better if you find it before your users do.