I know the app is still there. I know the name is “Audiogalaxy Music”. Please help my users find it when they search for it by name.

Are you caching search results from a device that we don’t support and then showing them to me? This feels like that sort of bug.

Update — and now we are back (though that text about no matches still cracks me up):

After a year of working on it, we are happy with our backend and our functionality, so we’ve exposed an API. Check out my post on the Audiogalaxy Blog for some more information, and follow @audiogalaxy for all the latest news.

Is this because:

• Nobody needs anything other than Google Website Optimizer?
• Startups don’t actually do AB testing, possibly because they don’t get enough traffic to get meaningful results, or maybe because they don’t have time?
• AB testing (including the statistical analysis to determine if results are valid) is so simple that everyone just bangs out their own?
• As a largely theoretical issue for most startups, scalability is more fun to talk about on the Internet?
• Everyone that is using AB testing is so happy that they are trying to suppress information about it so their competitors don’t start doing it too?

If everyone is secretly using some great framework please shoot me an email and let me know.

If you haven’t thought much about it before, here is a short paper on AB testing from some folks that made Amazon a ton of money.

• A clever proposal from Google: Shared Dictionary Compression over HTTP
• Cacheset – a tool for clearing the windows disk cache (useful for testing cold starts).
• Fun fact: the Tesla Roadster carries 3 milligrams of electrons when fully charged.
• The ultimate Airplane on a Treadmill debate resource: www.airplaneonatreadmill.com
• A 728-ton tuned mass damper in a skyscraper

But, I don’t think I’ve reached the critical mass of followers necessary to really unlock the Q&A potential of the site. Having a few hundred technical folks all following each other would be a tremendously useful resource for everyone involved. For example, I’m considering upgrading my desktop to 8 or 16GB of RAM. I’m going to need a new motherboard, processor, and RAM. My normal approach for this would be to spend a few hours on Newegg and the hardware review sites trying to figure out where the price/performance curve is and making sure I’m not getting ripped off. If someone else has done this same research it would be nice to use their information as a starting point, and twitter provides the kind of free-form conversation necessary for that kind of sharing.

To really make this work, you need to run one of the desktop apps so you don’t have to constantly reload the website (I use Twhirl).

Positive Feedback
Being told that I only spent 10% of my day doing work is good to know, but getting a low number might depress me rather than motivate me. I suggest a system that actually rewards me when I have a killer day or a great week. For example, I give the service $25 or $50 up front, and after I meet some sort of goal it buys me something off my Amazon wish list.

Wouldn’t that be neat? You’re having a good week, and suddenly a book you want shows up at your door. The key to this is making the rewards somewhat random:

Several studies have been conducted which targeted neural response to rewards. The results were unanimous in the fact that when one performed an action over and over again, and was given a reward randomly, dopamine levels rose. If the reward was given consistently, i.e. every four times the action was performed, the dopamine levels remained constant.

A slight variation that might work better would be for each contiguous block of productivity over a certain length, you have a chance of earning a credit towards a purchase. After N credits, the service automatically buys and sends you the item. Structuring it like this would make the feedback more rapid and allow for a little burst of dopamine each time you get an email saying you earned a credit. Isn’t this why MMORPGs are so much fun?.

A program to help you get addicted to work is either terrifying or a big win. Either way, it would be really neat to try.

Targeted Recommendations
Some software is just better than the default stuff that ships with Windows. For example, I like Textpad and Paint.net a lot more than Notepad and MS Paint. I’ve also been pleased with my switch from Bloglines to Google Reader, and from web based twitter to Twhirl. If a program spends all day monitoring my activity, it would be a cinch for it to recommend the tools and websites that are considered “best in class.”

There is obvious potential here in terms of sponsored recommendations, but it would be nice to see those separated out from community or editor controlled listings. Recommendations could be driven by some sort of wiki, which would make for all kinds of interesting fights over things like whether Google Reader is better than Bloglines. Any recommendation could also come with an estimate of the number of people that are currently using it, which would help the cream rise to the top.

Ultimately, it’s not just about how much time you spend slogging away – making good use of computer time is an important dimension of productivity as well.

A slight variation on this idea is to recommend something like LeechBlock if the user is spending too much time on the web.

Attention API
I know I’ve seen this idea somewhere before, but because things like RescueTime are actually in a position to make it happen, I’m going to mention it here. Interruptions are usually bad, but there are some times that they are worse than others. If I’ve been focused on Visual Studio and Windbg for 30 minutes with no breaks, I’m almost certainly in that fascinating “flow” state, and I’m going to be angry if I get an IM or (even worse) if some random app asks me to download a new version.

To deal with this kind of thing, it would be great to have a standard for publishing my current tolerance for interruptions, just like IM apps publish my presence. Both desktop apps and remote users could use this to determine if what they want to tell me is important enough to interrupt me. Of course, this only works if apps pay attention to it, but first we would need some apps that can accurately measure it. I’m not terribly good with naming things, but unless someone has something better, I’m going to suggest calling this value your “inTolerance”.

So there you go. One idea to help you spend more time being productive, another to help you make better use of the time you are actually working, and a third to keep you from getting interrupted. Anyone want to go and implement this stuff? I’ll be happy to beta test it for you.

But ultimately, better movie recommendations don’t matter a whole lot to me. I’m more interested in the fact that by providing a unique set of data and a prize, they’ve been able stimulate so much interest. The other day I was thinking about which companies are in a position to sponsor contests in other fields that might have a bigger impact on my life, and one thought jumped into my head – Google’s 411 phoneme collection service. Marissa Meyers says:

You may have heard about our [directory assistance] 1-800-GOOG-411 service. Whether or not free-411 is a profitable business unto itself is yet to be seen. I myself am somewhat skeptical. The reason we really did it is because we need to build a great speech-to-text model … that we can use for all kinds of different things, including video search.

The speech recognition experts that we have say: If you want us to build a really robust speech model, we need a lot of phonemes, which is a syllable as spoken by a particular voice with a particular intonation. So we need a lot of people talking, saying things so that we can ultimately train off of that

Presumably, Google has already done the heavy lifting to manually transcribe a large number of these samples so that they can train their own algorithms. Why not create a contest that lets teams submit an algorithm that gets trained on a subset of the data and then tested against the rest? Speech recognition is more complicated than movie recommendations, but making it easy to train and test an algorithm against an interesting number of samples would certainly lower the barrier to entry.

Google would benefit from this in hiring, if nothing else. It would give them a chance to realistically evaluate the work of all kinds of grad students and researchers, and demonstrate to the candidates the advantages of working for the company with the biggest databases.

Design the system to never need human interaction, but understand that rare events will occur where combined failures or unanticipated failures require human interaction.

Yes, designing the system to never need human interaction is a great ideal to shoot for, but when you are working for a startup with three guys and a dozen servers, you don’t have the resources or the justification to do it from Day 1. It is entirely likely that your business model will fail before you lose a single disk. And since backend refinements don’t pay the bills at a small scale, something with a pair of hands is going to be interacting with your system until you get enough people and servers to justify more automation.

These events will happen and operator error under these circumstances is a common source of catastrophic data loss.

That is a wonderfully simple and accurate summary of How Bad Things Happen in your datacenter. It starts when you lose a hard drive or MySQL crashes, and you have to promote your slave until you can check the master tables, or anything painful but routine happens. But then, as you are trying to fix things, you notice, for example, that you are almost out of disk space. When you start trying to fix more than one problem under pressure, you are entering a world of pain.

The big issue here as James points out is that you are going to do something wrong. You’ll probably use a much stronger word than “wrong” once it is all over, but let’s settle on “stupid” for right now.

It won’t feel stupid until after you hit “enter,” but when you are making unfamiliar decisions quickly under pressure, you are extremely likely to overlook something. Maybe you won’t shut down mysql before you start a myisamchk from the shell, or maybe you’ll reverse the arguments to “tar -cvzf” and wipe out something important. Or perhaps you’ll screw up a firewall rule and block ssh access to the machine you are frantically trying to fix. Accidently killing the ssh daemon is another favorite. The point is that during a stressful situation in the datacenter, the human operator is the biggest potential source of more downtime or “catastrophic data loss.”

Assuming you can’t automate everything, what can you do? Well, the absolute best thing you can do is practice. Corrupt some data on your dev master db, and see how long it takes you to get it restored from backups or a slave copy. Practice what would happen if you lost a database slave and had to activate a spare machine to take its place (I hope you have at least one spare machine). But of course, no one at a small startup has time to practice. Maybe once you hire a full time ops guy, it would be good to make sure he is practicing this sort of thing occasionally. But when practicing is going to take away from writing code, you aren’t going to practice.

Since you aren’t going to practice, what else can you do? The next best thing is to cultivate the attitude that you are the most likely source of problems. Don’t worry about hard drives, worry about bad decisions. Develop some humility about how you expect to behave when you get woken up at 4am to fix a database the morning of your launch or when a switch fails an hour before your big demo. From that mindset, here are a few things to do:

Script what you can
Off the top of my head, a good place to start would be writing scripts for some of the steps in setting up master/slave replication and manipulating firewall traffic (allowing or blocking external traffic, for instance).

Use the buddy system
It is not a bad idea to have somebody else there looking at what you are typing, or at least on the phone confirming things verbally.

Take your fingers off the keyboard before you hit enter
Are you in the right directory? Are you on the right machine? Are those arguments in the right order? Can you just rename this old stuff instead of deleting it? All of these are excellent questions to ask yourself or your coworker while you have your hands in your lap. This is also a good idea when you are doing something scary with SQL, like running any query that doesn’t have a where clause.

Slow things down
As soon as you make one mistake, no matter how minor, it is time to slow things down. Beyond the fact that making a mistake will fluster you, making one mistake demonstrates that right now, you are likely to make mistakes. That is a huge red flag. At this point, the safest thing may be to accept a slightly longer downtime just so you can slow things down, get some water, and relax. Trying to compensate for a little mistake by doing things faster can result in a much, much worse mistake. Unless you’ve just rolled a server cage down the stairs, there is always a worse mistake you can make.

Make it hard for people you work with to make mistakes
A quality server naming scheme is the easiest thing you can do here. No colors, deities, countries, snack foods, snakes, etc. I like $machineType-$number myself, but with distinct number ranges, even between different machine types. So, don’t have SQL-001 and Web-001. One day, some very sleepy datacenter employee may get things mixed up when you call and ask him to reboot Web-001. I’m sure you’ll get an apology, but you won’t get your uptime back. So make it harder for him to screw up: if your web machines start at Web-201, he’ll have to make 2 mistakes before he accidently reboots your primary database.

Talk about this stuff ahead of time
You probably have plenty of stuff to talk about at lunch with your coworkers, but here are a few convers ation starters if you want to sharpen your disaster recovery skills:

• “What happens if we lose power to one of our racks?”
• “How many of our switches could we lose and get the site back up?”
• “What is the smallest amount of hardware we could lose that would knock us 100% offline?”

This stuff isn’t theoretical. I woke up at 2am one weekend during FolderShare with a ton of text messages from our cluster. The kindly folks at the datacenter had been doing power supply maintenance. At some point, they powered down 2 of our racks. Then, they powered them right back up. It wasn’t tough to fix, but it was so unexpected that it took me a few minutes to even realize what had happened.

Use tricks to deal with the general class of “running a command on the wrong machine” problem.
Typing the right command on the wrong machine is obviously something to avoid. But when you have a sea of ssh windows open, what can you do?

• Use a different color background for your terminal to machines hosting master databases versus slaves
• Make sure the machine name shows up in the command prompt

Does anyone else have any good ideas or horror stories to tell? Post a comment and share your wisdom and/or pain.

• Am I getting more than I need? It pains me to add a multi megabyte DLL to a client download for a small amount of functionality.
• Will I spend more time learning the interface than I would writing the functionality I need myself?
• Is this an active project, and is there any documentation?
• If scheduling isn’t an issue, how much fun would it be to write my own version?

Next comes a set of questions that are oftentimes harder to answer:

• Who else is using it?
• Will I be using it the same way as other people who are successfully using it?
• What am I going to find out when I put more stress on it than anyone else?

One library that passed my gauntlet of questions is libredblack. It ended up on a bunch of production servers at FolderShare, and it worked out great. But there was a catch: I wanted to use it to store large numbers of items, but for every item I put in the tree, the library would allocate an object that held 4 pointers and an enum. That took 40 bytes on my dev box. Throw in malloc’s overhead, and I was up to 48 bytes. The objects I was storing pointers to would also have some heap overhead, which may be as much as 24 bytes. So to store 10M items in memory, I’d need an extra half gigabyte of memory just for overhead.

A second example from personal experience is librsync. Again, the library works exactly as advertised. But if you want to transfer deltas for large (gigabyte+ files) on machines that have hard memory limits (like embedded devices), you need to know that the memory usage is proportional to the file size. For my situation, I ended up having to adjust the window size as file sizes grew just to keep the memory usage reasonable for large files.

I don’t want anyone to think I’m complaining about this stuff – I’m a fan of both libraries. But both of these examples illustrate a class of problem that is particularly frustrating: the one you might not find until you are heavily invested in a solution. These gotchas won’t affect most people, and thus aren’t likely to show up when you are researching possible solutions. They aren’t bugs, either, but they might be something you have to deal with. So the sooner you can find out about them, the better.

Fortunately, the internet has plenty of software built for solving problems like this. Dev Diligence^[1] is a new wiki I’ve started to collect details like these. My goal is to have a reference page for any library or service developers might consider using in their solution. For sufficiently large libraries, pages for classes or functions might be necessary, but let’s not get ahead of ourselves. Ultimately, I’d like to have 5 headings for everything in the wiki:

• Overview: Brief description of the software and a link to the homepage
• Short case studies or war stories: These would include a brief description of how you are using the software, the version you used, and ideally some metrics. If you used it for a while and then switched to something else, an explanation of that decision is very valuable information. For libredblack, the relevant metrics would be things like average number of elements in your trees or insertions/deletions per second.
• “Gotchas” (like the ones I’ve mentioned above): Subtle problems (hello, heap fragmentation) and things that aren’t necessarily bugs, but issues that may affect your design or help you choose one solution over another.
• Alternatives: The name pretty much says it all. With links, please.
• Other Resources: Links to blog posts, email threads, or reference pages would be great.

I’ve gone ahead and created entries for libredblack, librsync, and zlib based on my experiences. I’d love to see some entries for the following and things like it:

• libev, libevent, boost.asio, and Twisted
• openssl
• sqlite and berkeleydb
• memcached, spread, the reliable queue solutions (Starling, TheSchwartz, etc), and anything that uses “pubsub” in its description
• libcurl and wininet (stuff like Nick Bradbury’s description of a CPU spike in WinInet that can be triggered by chunked-encoding is gold)

All of these and more are linked to from the WishList page.

Can you guys help me out? I’ve got enough people subscribed to this feed that I’m certain at least one of you has used everything on my list. If you take 10 minutes to write down your experiences, you can make the software world a better place. To justify doing it on your company’s time, keep this in mind: if you document the fact that you are successfully using a solution, you increase the chance that other people will use it as well. The more users a solution has, the the better it will become.

[1] “Dev Diligence” is of course a play on the term “Due Diligence”.