Barry Ferg's shared items in Google Reader

The Rands Test

(author unknown) — 2011-10-11T10:17:24-07:00

It's hard to pick a single best work by Joel Spolsky, but if I was forced to, I'd pick The Joel Test. It's his own, highly irresponsible, sloppy test to rate the quality of software, and when anyone asks me what is wrong with their team I usually start by pointing the questioner at the test. Start here.

It's a test with 12 points and as Joel says, "A score of 12 is perfect, 11 is tolerable, but a 10 or lower and you've got serious problems". More important than the points, his test clearly documents what I consider to be healthy aspects of an engineering team, but there are other points to be made. So it is completely an homage to Joel that I offer The Rands Test.

I was employee #20 at the first start-up and the first engineering lead. Over the course of two years, the team and the company exploded to close to 200 employees. This is when I discovered that growing rapidly teaches you one thing well: how communication continually finds new and interesting ways to break down. The core issue being the folks who've been around longer who also tend to have more responsibility. As far as they're concerned, the ways they organically communicated before will remain as efficient and simple each time the group doubles in size.

They don't. A growing group needs to continually invest in new ways to figure out what it is collectively thinking so anyone anywhere can answer the question: "What the hell is going on?" This is the first question The Rands Test answers. As I'll explain shortly, the second question The Rands Test helps you answer is selfish. The second asks: "Where am I?"

12 Points

Let's start with bare bones versions of the questions and then I'll explain each one.

Do you have a 1:1?
Do you have a team meeting?
Do you have status reports?
Can you say No to your boss?
Can you explain the strategy of the company to a stranger?
Can you explain the current state of business?
Does the guy/gal in charge regularly stand up in front of everyone and tell you what he/she is thinking? Are you buying it?
Do you know what you want to do next? Does your boss?
Do you have time to be strategic?
Are you actively killing the Grapevine?

(Note: While I'll explain each point from the perspective of a leader or manager, these questions and their explanations apply equally to individuals.)

Do you have a consistent 1:1 where you talk about topics other than status? (+1)

I think you'd be hard pressed to find anyone who would suggest 1:1s are a bad idea, but the 1:1 is usually the first meeting that gets rescheduled when it hits the fan. I'm of the opinion that when it hits the fan, the last thing you want to do is reschedule 1:1 time with the folks who are likely either responsible for it hitting the fan and/or are the most qualified to figure out how to prevent future fan hittage.

Furthermore, as I wrote about in The Update, The Vent, and The Disaster, conveyance of status is not the point of a 1:1; the point is to have a conversation about something of substance. Status can be an introduction, status can frame the conversation, but status is not the point. A healthy 1:1 needs to be strategic, not a rehashing of tactics and status that can easily be found elsewhere.

A 1:1 is a weekly investment in the individuals that make up your team. If you're irregularly doing 1:1s or not making them valuable conversations, all you're doing is reinforcing the myth that managers are out of touch.

Do you have a consistent team meeting? (+1)

The team meeting has all the requirements of the 1:1 -- consistency and a focus on topics of substance -- but don't give yourself a point just yet.

Status does have a bigger role in a team meeting. As we'll talk about shortly, the Grapevine is a powerful beast and a team meeting is a chance to kill it. I have a standing agenda item for all team meetings that reads "gossip, rumors, and lies" and when we hit that agenda item, it's a chance for everyone on the team to figure out what is the truth and what is a lie.

After that's done, my next measure of a team meeting is: did we make tangible progress on something? I don't know what you build, so I don't know what's broken on your team, but I do know that something is broken and a team meeting is a great place to not only identify the brokenness, but also to start to discuss how to fix it.

If you're killing lies and fixing what's broken in a team meeting, give yourself a point.

Are handwritten status reports delivered weekly via email? (-1)

If so, you lose a point. This checklist is partly about evaluating how information moves around the company and this item is the second one that can actually remove points from your score. Why do I hate status so much? I don't hate status; I hate status reports.

My belief is that email-based status reports are one of the clearest and best signs of managerial incompetence and laziness. There are always compelling reasons why you need to generate these weekly emails. We're big enough that we need to cross-pollinate. It's just 15 minutes of your time.

Bullshit. The presence of rigid, email-based status reports comes down to control, a lack of imagination, and a lack of trust in the organization.

I want you to count the number of collaboration tools you use on a daily basis to do your job -- not including email. If you're a software engineer, I'm guessing it's a combination of version control, bug tracking, wikis, CRM, and/or project management software. All of these tools already automatically generate a significant amount of status regarding what has tactically gone down each week.

When someone asks for a status report, my first thought is: "I'm already generating piles of status on these various tools, why not just ask those?"

Well, there's a lot of noise in those tools. Well, write a report that takes out the noise -- collaboration tools are built around reporting. The status information is out there. In what managerial textbook does it say it's a good idea to "Distribute the task of figuring out what is going on to the people who are performing the work?" That's, like, your job.

Well, what I really want is your high level assessment of the week. Three things that are working, three things that aren't, and what we're going to do about it. Ok, now we're talking. I can do a strategic assessment of the week, but why don't we just put that at the beginning of the 1:1? That way when you have questions (and you will), we can have a big fat debate.

But I'd like to have a record I can review later. Super, feel free to write down anything we talk about.

Yes, status reports are a hot button for me. I've written hundreds of them and each time I've begun one, I start by thinking, "Why in the world do I feel like I'm performing an unnecessary act?" Status reports usually show up because a distant executive feels out of touch with part of his or her organization and they believe getting everyone to efficiently document their week is going to help. It doesn't. Emailed status reports say one thing to 90% of the people who wrote them: "You don't value my time". This leads us to our next point...

Are you comfortable saying NO to your boss? (+1)

Perhaps a better way to phrase this point is: do you feel your 1:1 with your boss is somehow different than every other meeting you have during the week? Part of healthy communication structure is when information moves easily around the team, organization, and company, and if you walk into a meeting with your boss always on your best behavior and unwilling to speak your mind I say something is broken.

Yes, he or she is your boss and that means they write your annual review and can affect the trajectory of your career, but when they open their mouth and say something truly and legitimately stupid, your contractual obligation as a shareholder of the company is to raise your hand and say, "That's stupid. Here's why..."

Easier said than done, Rands.

Ok, don't say it's stupid.

Here's the deal. I believe that leaders who think they're infallible slowly go insane with power created by the lie that being wrong is a sign of weakness. I screw up -- likely regularly -- and I've been doing various forms of this gig for twenty years. While it still stings when I stumble upon or others point out my screw-ups, I'd sooner I admit I fucked up, because then I can figure out what I really did wrong faster, and that starts with someone saying "No".

Can you explain the strategy of your company to a stranger? (+1)

Moving away from communications, this point is about strategy and context. If I was to walk up to you in a bar and ask what your company did, could you easily and clearly explain the strategy?

This is the first point that demonstrates whether you have a clear map of the company in your head, and you might be underestimating the value of this map. If you're a leadership type, chances are you can draw this map easily. If you're an individual, you might think this map is someone else's responsibility and you'd be partially correct: it is someone else's job to define the map, but it's entirely your responsibility to understand it so you can measure it.

As we'll see with the following questions, The Rands Test isn't just about understanding communications, it's about understanding context and strategy. How do you think the employees of HP and Netflix feel given the strategy flip-flops over the past few months? Safe or suspicious? Let's keep going...

Can you tell me with some accuracy the state of the business? (Or could you go to someone / somewhere and figure it out right now?) (+1)

It's a brutal exaggeration, but I think you should independently judge your company the same way that Wall Street does: your company is either growing or dying. Have you ever watched the stock price of a publicly traded company the day after they announce that they are going to miss their earnings numbers? More often than not, no matter what spin the executives have, the stock is hammered. It's irrational, but what I infer when I see this happen is that Wall Street believes the company has begun a death cycle. If the executives can't successfully predict the state of their business, something is wrong.

I realize this isn't fair and there are myriad factors that contribute to the health of the business every single day, and I encourage you to research and understand as many of those as possible. But when you're done, I'd also like you to have a defensible opinion regarding the state of the business, or at least a set of others whose opinion you trust.

This is a picture that you are constantly building, and this is an easier task if you've given yourself a point on the prior question regarding company strategy. If you have a map of what the company intends to do, it's easier to understand whether or not it's doing it. This leads us to...

Is there a regular meeting where the guy/gal in charge gets up in front of everyone and tells you what he/she is thinking? (+1) And are you buying it? (+1)

Our last point regarding context involves the person in charge. In rapidly growing teams and companies there's a lot going on -- every single day. When the team was small, the distribution of information was easy and low cost because everyone was within shouting distance. At size, this communication becomes more costly at the edges. Directors, leads, and managers -- these folks tend to stay close to current events because it's increasingly their job, but it's also their job to take steps to keep the information flowing, and it starts with the CEO.

On a regular basis, does your CEO stand up and give you his impressions of what the hell is going on? Whether it's 10 or 10,000 of you, this is an essential meeting that:

Gives everyone access to the CEO.
Allows him/her to explain their vision for the company.
Hopefully allows anyone to stand up and ask a question.

If the value of this meeting isn't immediately obvious to you, I'd suggest that you are one of those lucky people who already has a good map of the company as well as a sense of the state of the business. That's awesome -- here's a bonus point for you: does the CEO's version of the truth match yours or is he/she in a high Earth orbit with little clue what is actually going on? Give yourself a point if it's the former and if it's the latter, what does that say about the state of the business? Growing or dying?

Can you explain your career trajectory? (+1) [Bonus: Can your boss? (+1)]

Next, switching gears a bit, give yourself a point if you -- right this very moment -- can tell me your next move. You're already doing something, so explain what you're going to do next. It's a simple statement, not a grand plan. One day, I'd like to lead a team.

Part of a healthy organization isn't just that information is freely moving around; it's what the folks receiving and retransmitting it are doing with it. You're going to mentally file and ignore a majority of this information, but every so often a piece of information will come up in a 1:1, a meeting, or a random hallway conversation, and it will be strategically immediately useful for you to know what you want to do next.

Angela got a promotion and her team is great and I've always wanted to be a manager.
Jan just opened a requisition and his group is working on technology I need to learn.
They fired Frank. That creates a very interesting power vacuum...

You can argue that even without a plan you'd make the same opportunistic leap, but I've found that having a map is usually a better way of getting to a destination.

There's a bonus point here as well. Does your boss know what you want to do next? He or she likely has even more access to the information moving around the company, and whether they like it or not, have equal responsibility to figure out how to get you from here to there.

Do you have well-defined and protected time to be strategic? (+1)

If you gave yourself two points on the prior question, congratulations, I think you're in better shape than most, but there's one more point. Are you making progress towards this goal? Can you point to time on your calendar or even just in your head where you are growing towards your goal?

I like being busy. Like really busy. Like getting in, grabbing a cup of coffee, and suddenly finding the coffee is cold, it's 6pm, and I forgot to eat busy. Busy feels great, but busy is usually tactical and not strategic. This is why I'm constantly maintaining my Trickle List -- it's my daily reminder of doing work that is larger than right now.

If you have time where you're investing in yourself while you're at work and your boss is cool with it -- give yourself a point.

Are you actively killing the Grapevine? (+1)

When Grace walks in your office, you know she knows something by the look on her face. She moves to the corner of the office and starts with, "Did you hear...?" and the story continues. It's a doozy, full of corporate and political intrigue, resulting in your inevitable response: "No. Way."

Being part of a secret feels powerful. In a moment the organization reveals a previously hidden part of itself, and in that moment you feel you can see more of the game board. So, that's why they fired him. I was wondering. Grace finishes with the familiar, "Don't tell anyone," which is ironic since that's precisely what was asked of her 15 minutes ago.

There is absolutely no way you're going to prevent folks from randomly talking to each other about every bright and shiny thing that's going on in your company. In fact, you want to encourage it. 1:1s and meetings are only going to get you so far. The thing you can change is the quality of the information that's wandering the company.

In the absence of information, people make shit up. Worse, if they at all feel threatened they make shit up that amplifies their worst fears. This is where those absolutely crazy rumors come from. See, Kristof is worried about losing his job so he's making up crazy conspiracy theories that explain why THE MAN IS OUT TO GET HIM.

Without active prevention, the Grapevine can be stronger than any individual. While you can't kill the Grapevine, you can dubiously stare at it when it shows up on your doorstep and simply ask the person delivering it, "Do you actually believe this nonsense? Do you believe the person who fed you this trash?" Rumors hate to justify themselves, so give yourself a point if you make it a point to kill gossip.

Magnitude and Direction

There is a higher order goal at the intersection of the two questions The Rands Test intends to answer: Where am I? and What the hell is going on? While understanding the answers to these questions will give you a good idea about the communication health of your company, the higher order goal is selfish. I'll explain.

I think of the two lines of questions as a vector. A simple vector can be drawn as two points connected by an arrow, but a vector is far more interesting. It's a geometric object that describes both direction and magnitude. Understanding how information moves, how you communicate with your boss, and being able to describe both your career strategy and that of your company sketches a vector in your head. The first point is you at this very moment and the other point is where you want to be. The distance and direction between the two start to explain how you're going to get there. I love vectors because they draw a picture about a complex problem and I hope as you were answering the questions above this mental picture began to appear in your head.

Like the Joel Test, the point of the Rands Test is not the absolute score, but the score is good directional information. If you got a 12, I'd say you're in a rare group of people who have a clear picture of their company and where they fit in. Between 8 and 10, you are likely troublingly deficient in either communications, strategy, or your development - it depends where the points are missing. Less than 8 and I think you've got a couple of problems.

There are a lot of different scenarios I expect folks to find themselves in as they explore these questions, which is why it's tricky to proscribe specific action. Your company may be doing well, but you may be unhappy and have no clue what you want to do next. You might love your job, but have no idea whether the company is actually growing. Your course is dependent on what you care about and the Rands Test points out good places to start.

sunday fantasy #350: Enemy Ace. Art by Joe Kubert

(author unknown) — 2011-10-09T11:24:00-07:00

sunday fantasy #350: Enemy Ace. Art by Joe Kubert

The great bank robbery – Global Public Square - CNN.com Blogs

(author unknown) — 2011-09-05T11:46:13-07:00

Shared by Barry Ferg
Banks are part of the problem.

Presto (Pixar, 2008)

My favorite cartoon. The most economical...

(author unknown) — 2011-07-23T22:16:14-07:00

Shared by Barry Ferg
A big favourite in our house too...

Presto (Pixar, 2008) My favorite cartoon. The most economical five minutes of animation I’ve ever seen and, not inconsequentially, a rare everybody-loves-it hit around our house. Love that bunny. “Tink…tink.” [More and More]

Sticker Pinpoints The Sweet Spot For Destroying Your Hard Drive

Jonathan Fallon — 2011-07-15T06:33:42-07:00

From CubicleBot:If you’re a superhero who sucks with computers, you may want to take some precautions to ensure you can destroy your hard drive just in case it happens to fall in the hands of your evil archenemy who’s trying to take over the world. God knows you don’t want him to know how much porn you watch.

One way to do it is to use the appropriate iFixt teardown to locate the exact position of your laptop’s hard drive, and then apply a sticker over the spot made from the pattern found here. Then you can simply drill through it when the time is right. This may also give people peace of mind when they want to dispose of an old computer, but are worried about any personal information remaining on the drive.

(Free Art & Technology via Technabob)

Leonard Nimoy Is Lazy As Hell [Video]

Sean Fallon — 2011-05-28T06:23:47-07:00

Ugh, Bruno Mars. I’ll say this though, Leonard Nimoy is fantastic in this video for the alternate version of The Lazy Song. Too bad this video doesn’t pop up every time the song plays—it would make Bruno Mars tolerable.

Check out the video after the break.

(via GAS)

js/linux

Jim Mathies — 2011-08-16T17:44:26-07:00

This PC emulator is written in Javascript. The emulated hardware consists of:

a 32 bit x86 compatible CPU
a 8259 Programmble Interrupt Controller
a 8254 Programmble Interrupt Timer
a 16450 UART

The code is written in pure Javascript using Typed Arrays which are available in recent browsers. It was tested with Firefox 4 and Google Chrome 11 on Linux, Window and Mac (it does not work with Chrome 12 beta. As far as I know, it is a bug in the browser). In any case, a fast Javascript engine is needed to have good performance.

I compiled a 2.6.20 Linux kernel (I guess any other version would work provided there is still an FPU emulator). The configuration is here and there is a small optional patch to avoid outputting warnings due to the slow serial port. An uncompressed kernel image is used instead of a compressed one to have a faster boot. It is generated with “objcopy -O binary vmlinux vmlinux.bin”.

The disk image is just a ram disk image loaded at boot time. It contains a filesystem generated with Buildroot containing BusyBox. I added my toy C compiler TinyCC and my unfinished but usable emacs clone QEmacs.

Link

One perfectly optimized gin n' tonic, three graphs

Maggie Koerth-Baker — 2011-05-17T08:35:10-07:00

Here in the Northern Hemisphere, we are rapidly approaching gin n' tonic season. But don't you just hate it when your gin dilutes or the drink becomes unacceptably warm before you have finished imbibing? Haven't you always wished for a gin n' tonic recipe that was optimized to maintain a temperature of 20°F and 0% dilution throughout the 20 minutes it takes to properly savor a drink? If you prefer your drink recipes include multiple graphs, there's a post at Kaiser Penguin that will make you very, very happy. (Via Andrew Thaler)

Techno Life Skills

(author unknown) — 2011-04-28T16:39:46-07:00

Shared by Barry Ferg
Sage advice from KK. Best ones: "You will be newbie forever", "... don't think, try."

If you are in school today the technologies you will use as an adult tomorrow have not been invented yet. Therefore, the life skill you need most is not the mastery of specific technologies, but mastery of the technium as a whole -- how technology in general works. I like to think of this ability to deal with any type of new technology as techno-literacy. To be at ease with the flux of technology in modern-day life you'll need to speak the language of the technium, and to master the the following principles:

• Anything you buy, you must maintain. Each tool you use requires time to learn how to use, to install, to upgrade, or to fix. A purchase is just the beginning. You can expect to devote as much energy/money/time in maintaining a technology as you did in acquiring it.

• Technologies improve so fast you should postpone getting anything until 5 minutes before you need it. Get comfortable with the fact that anything you buy is already obsolete. Therefore acquire at the last possible moment.

• You will be newbie forever. Get good at the beginner mode, learning new programs, asking dumb questions, making stupid mistakes, soliticting help, and helping others with what you learn (the best way to learn yourself).

• Often learning a new tool requires unlearning the old one. The habits of using a land line phone don't work in email or cell phone. The habits of email don't work in twitter. The habits of twitter won't work in what is next.

• Take sabbaticals. Once a week let go of your tools. Once a year leave it behind. Once in your life step back completely. You'll return with renewed enthusiasm and perspective.

• How easy to switch? You will leave the tool you are using today at some time in the near future. How easy will it be to leave? If leaving forces you to leave all your data behind, or to learn a new way of typing, or to surrender four other technologies you were still using, then maybe this is not the best one to start.

• Quality is not always related to price. Sometimes expensive gear is better, sometimes the least expensive is best for you. Evaluating specs and reviews should be the norm.

• For every expert opinion you find online seek an equal but opposite expert opinion somewhere else. Your decisions must be made with the full set of opinions.

• Understanding how a technology works is not necessary to use it well. We don't understand how biology works, but we still use wood well.

• Tools are metaphors that shape how you think. What embedded assumptions does the new tool make? Does it assume right-handedness, or literacy, or a password, or a place to throw it away? Where the defaults are set can reflect a tool's bias.

• What do you give up? This one has taken me a long time to learn. The only way to take up a new technology is to reduce an old one in my life already. Twitter must come at the expense of something else I was doing -- even if it just daydreaming.

• Every new technology will bite back. The more powerful its gifts, the more powerfully it can be abused. Look for its costs.

• The risks of a new technology must be compared to the risks of the old technology, or no technology. The risks of a new dental MRI must be compared to the risks of an x-ray, and the risks of dental x-rays must be compared to the risks of no x-ray and cavities.

• Be suspicious of any technology that requires walls to prevent access. If you can fix it, modify it or hack it yourself, that is a good sign.

• The proper response to a stupid technology is to make a better one yourself, just as the proper response to a stupid idea is not to outlaw it but to replace it with a better idea.

• Nobody has any idea of what a new invention will really be good for. To evaluate don't think, try.

• The second order effects of technology usually only arrive when everyone has one, or it is present everywhere.

• The older the technology, the more likely it will continue to be useful.

• Find the minimum amount of technology that will maximize your options.

Minimalist DC Superhero Poster - by Michael B.... – HeroChan

(author unknown) — 2011-04-19T16:36:55-07:00

Minimalist DC Superhero Poster - by Michael B....

Anatomy of a Crushing

(author unknown) — 2011-03-09T21:00:26-08:00

Shared by Barry Ferg
What happens when the thundering hordes come for a visit.

A number of people asked about the technical aspects of the great Delicious exodus of 2010, and I've finally had some time to write it up. Note that times on all the graphs are UTC. On December 16th Yahoo held an all-hands meeting to rally the troops after a big round of layoffs. Around 11 AM someone at this meeting showed a slide with a couple of Yahoo properties grouped into three categories, one of which was ominously called "sunset". The most prominent logo in the group belonged to Delicious, our main competitor. Milliseconds later, the slide was on the web, and there was an ominous thundering sound as every Delicious user in North America raced for the exit. [*] I got the message just as I was starting work for the day. My Twitter client, normally a place where I might see ten or twenty daily mentions of Pinboard, had turned into a nonstop blur of updates. My inbox was making a kind of sustained pealing sound I had never heard before. It was going to be an interesting afternoon. Before this moment, our relationship to Delicious had been that of a tick to an elephant. We were a niche site and in the course of eighteen months had siphoned off about six thousand users from our massive competitor, a pace I was was very happy with and hoped to sustain through 2011. But now the Senior Vice President for Bad Decisions at Yahoo had decided to give us a little help. I've previously posted this graph of Pinboard web traffic on the days immediately before and after the Delicious announcement. That small blue bar at bottom shows normal traffic levels from the week before. The two teal mountain peaks correspond to midday traffic on December 16 and 17th. My immediate response was to try to log into the server and see if there was anything I could to do keep it from falling over. Cegłowski's first law of Internet business teaches: "Never get in the way of people trying to give you money", and the quickest way to violate it would have been to crash at this key moment. To my relief, the server was still reachable and responsive. A glance at apachetop showed that web traffic was approaching 50 hits/second, or about twenty times the usual level. This is not a lot of traffic in absolute terms, but it's more than a typical website can handle without warning. Sites like Daring Fireball or Slashdot that are notorious for crashing the objects of their attention typically only drive half this level of traffic. I was expecting to have to kill the web server, put up a static homepage, and try to ease the site back online piecemeal. But instead I benefitted from a great piece of luck. Pinboard shares a web server with the Bedbug Registry, a kind of public forum for people fighting the pests. I started the registry in 2006 (a whole other story) and it existed in quiet obscurity until the summer of 2010, when bedbugs infested some high-profile retail stores in New York City and every media outlet in the country decided to run a bedbug story at the same time. The Summer of Bug culminated in a September link from the CNN homepage that drove a comparable volume of traffic (about 45 hits/second) and quickly turned the server to molasses. At that point Peter spent a frantic hour reconfiguring Apache and installing pound in order to safely absorb the attention. Neither of us realized we had just stress-tested Pinboard for the demise of Delicious three months later. [**] Thanks to this, the Pinboard web server ran like a champ throughout the Delicious exodus even as other parts of the service came under heavy strain. We were able to keep our median page display times under a third of a second through the worst of the Yahoo traffic, while a number of other sites (and even the Delicious blog!) went down. This gave us terrific word-of-mouth later. Of course, had bedbugs been found in the Delicious offices, our server would have been doomed. Another piece of good luck was that I had overprovisioned Pinboard with hardware, basically due to my great laziness. Here's what our network setup looked like on that fateful day: As you can see, we had a big web server connected to an even bigger database server, with a more modest third machine in charge of background tasks. It has become accepted practice in web app development to design in layers of application caching from the outset. This is especially true in the world of Rails and other frameworks, where there is a tendency to treat one's app like a high-level character in a role-playing game, equipping it with epic gems, sinatras, capistranos, and other mithril armor into a mighty "application stack". I had just come out of Rails consulting when I started Pinboard and really wanted to avoid this kind of overengineering, capitalizing instead on the fact that it was 2010 and a sufficiently simple website could run ridiculously fast with no caching if you just threw hardware at it. After trying a number of hosting providers I found Digital One, a small Swiss company that rented out HP blade servers with prodigious (at least by web hosting standards) quantities of RAM. This meant that our two thousand active users were completely swallowed up within a vast, cathedral-like database server. If you offer MySQL this kind of room, your data is just going to climb in there and laugh at you no matter what kind of traffic it gets. Since Pinboard is not much more than a thin wrapper around some carefully tuned database queries, users and visitors could page through bookmarks to their hearts' content without the server even noticing they were there. That was the good news. The bad news was that it had never occurred to me to test the database under write load. Now, I can see the beardos out there shaking their heads. But in my defense, heavy write loads seemed like the last thing Pinboard would ever face. It was my experience that people approached an online purchase of six dollars with the same deliberation and thoughtfulness they might bring to bear when buying a new car. Prospective users would hand-wring for weeks on Twitter and send us closely-worded, punctilious lists of questions before creating an account. The idea that we might someday have to worry about write throughput never occurred to me. If it had, I would have thought it a symptom of nascent megalomania. But now we were seeing this: There were thousands of new users, and each arrived clutching an export file brimming with precious bookmarks. Within a half hour of the onslaught, I saw that imports were backing up badly, the database was using all available I/O for writes, and the two MySQL slaves were falling steadily behind. Try as I might, I could not get the imports to go through faster. By relational database standards, 80 bookmark writes per second should have been a quiet stroll through a fragrant meadow, but something was clearly badly broken. To buy a little time, I turned off every non-essential service that wrote anything to the database. That meant no more tag clouds, no bookmark counts, no pulling bookmarks from Instapaper or Twitter, and no popular page. It also meant disabling the search indexer, which shared the same physical disk. Much later we would learn that the problem was in the tags table. In the early days of Pinboard, I had set it up like this: create table tags tag char(255), bookmark_id int, .... unique index (tag, bookmark_id) charset=utf8 Notice the fixed-length char instead of the saner variable-length varchar. This meant MySQL had to use 765 bytes per tag just for that one field, no matter how short the actual tag was[***]. I'm sure what was going through my head was something like 'fixed-width rows will make it faster to query this table. Now how about another beer!'. Having made this brilliant design decision, I so thoroughly forgot about it that in later days I was never able to figure out why our tags table took forever to load from backup. The indexes all seemed sane, and yet it took ages to re-generate the table. But of course what had happened was the table had swollen to monstruous size, sprawling over 80 GB of disk space. Adding to this table and updating its bloated index consumed three quarters of the write time for every bookmark. Had I realized this that fateful afternoon, I might have tried making some more radical changes while my mind was still fresh. But at that moment the full magnitude of what we were dealing with hadn't become clear. We had had big spikes in attention before (thanks @gruber and @leolaporte!), and they usually faded quickly after a couple of hours. So I focused my efforts on answering support requests. Brad DeLong, the great economics blogger, was kind enough to collect and publish our tweet stream from that day for posterity. We had always prided ourself on being a minimalist website. But the experience for new users now verged on Zen-like. After paying the signup fee, a new user would upload her delicious bookmarks, see a message that the upload was pending, and... that was it. It was possible to add bookmarks by hand, but there was no tag cloud, no tag auto-completion, no suggested tags for URLs, the aggregate bookmark counts on the profile page were all wrong, and there was no way to search bookmarks less than a day old. This was a lot to ask of people who were already skittish about online bookmarking. A lot of my time was spent reassuring new users that their data was safe and that their money was not heading into a Nigerian bank account. At seven PM Diane ran out for a bottle of champagne and we gave ourselves ten minutes to celebrate. Here I am watching three hundred new emails arrive in my mailbox. To add spice to the evening, our outbound mail server had now started to crash. Each crash required opening a support ticket and waiting for someone in the datacenter to reboot the machine. Whenever this happened, activation emails would queue up and new users would be unable to log in until the machine came back online. This diverting task occupied me until midnight, at which point I had been typing nearly nonstop for eleven straight hours and had lost about fifty IQ points. And imports were still taking longer and longer; at this point over six hours. Pinboard has a three-day trial period, and I was now having nightmare visions of spending the next ten days sitting in front of the abysmally slow PayPal site, clicking the 'refund' button and sniffling into a hankie. My hope had been that we could start to catch up after California midnight, when web traffic usually dies down to a trickle. But of course now Europe was waking up to the Yahoo news and panicking in turn. There was no real let-up, just the steady drumbeat of new import files. At our worst we fell about ten hours behind with imports, and my wrists burned from typing reassuring emails to nervous new customers explaining that their bookmarks, would, in the fullness of time, actually show up on the big blank spot that was their homepage. We added something like six million bookmarks in the first 24 hours (doubling what we had collected in the first year and a half of running the site), another 2.5 million the following day, and a cumulative ten million new bookmarks in that first week. This graph shows the average expected time in minutes users had to wait after uploading their stuff: It wasn't until dawn that the import lag started to decrease. It was now Friday immediately before Christmas week, and I felt if we could steer the site safely into the weekend we would get some breathing room. Saturday night would be the perfect time to run the expensive ALTER TABLE statement that would fix the tags issue. Comforted by this thought I went out for a run (because why not?), and then dived into bed with iron instructions to be awakened ninety minutes later, no matter how much I cried. Fully refreshed, I could turn my attention to the next pair of crises: tag clouds and archiving. Tag clouds on Pinboard are a simple UI element that shows the top 200 or so tags you've used on the right side of your home page. Since I had turned off the script that made the clouds, new users were apprehensive that their tags had not imported properly. Up to this point I had generated tag clouds by running a SQL query that grouped all a user's tags together and stored the counts in a summary table. Anytime a user added or edited a bookmark they got thrown on a queue, and a script lurking in the background regenerated their tag counts from scratch. This query was fairly expensive, but under minimal load it didn't matter. Of course, we weren't under minimal load anymore. The obvious fix was to calculate the top tags in code and only update the few counts that had changed. But coding this correctly was surprisingly difficult. The experience of programming on so little sleep was like trying to cook a soufflé by dictating instructions over a phone to someone who had never been in a kitchen before. It took several rounds of rewrites to get the simple tag cloud script right, and this made me very skittish about touching any other parts of the code over the next few days, even when the fixes were easy and obvious. The part of my brain that knew what to do no longer seemed to be connected directly to my hands. The second crisis was more serious. For an an extra fee, Pinboard offers archival accounts, where the site crawls and caches a copy of every bookmark in your account. New users have the option of signing up for archiving from the start, or upgrading to it later. A large number of recent arrivals had chosen the first option, which meant that we had a backlog of about two million bookmarks to crawl and index for full-text search. It also meant we had a significant group of new users who had paid extra for a feature they couldn't evaluate.[****] Like many other parts of the service, the crawler was set up to run in one process per server. It was imperative to rewrite the crawler script so that multiple instances could run in parallel on each machine, and then set up an EC2 image so that we chew through the backlog even faster. The EC2 bill for December came to over $600, but all the bookmarks were crawled by Tuesday, and my nightmare of endless refund requests didn't materialize. On Monday our newly provisioned server came on line. Figuring that overkill had served me well so far, this one had 64 GB of memory and acres of disk space. On Tuesday morning I was invited to appear on net@night with Leo Laporte and Amber MacArthur, who were both terrifically encouraging. At this point I could barely remember my own name. And then, mercifully, it was Christmas, and everyone got offline for a while. In these writeups it's traditional to talk about LESSONS LEARNED, which is something I feel equivocal about. There's a lot I would have done differently knowing what was coming, but the whole thing about unexpected events is that you don't expect them. Most of the decisions that caused me pain (like never taking the time to parallelize background tasks) were sensible trade-offs at the time I made them, since they allowed me to spend time on something else. So here I'll focus on the things that were unequivocally wrong: Too many tasks required typing into a live database It is terrifying and you are very tired. At the outset Peter and I had to do live SQL queries to find user accounts, fix names, emails, and logins, and do other housekeeping tasks. I lived in constant fear of forgetting a WHERE clause. We had no public status page I could have avoided a very large volume of email correspondence by having a status page to point to that told people what services were running and which were temporarily disabled. I assumed slaves would be within a few minutes of the master There were multiple places in my code where I queried a slave and updated the master. This only works if you don't care about being many hours out of date. For example, it would have been fine for the popular page, but was not acceptable for bookmark counts. There were also some things that went well: We used dedicated hardware To quote a famous businessman: "It costs money. It costs money because it saves money". We charged money for a good or service I know this one is controversial, but there are enormous benefits and you can immediately reinvest a whole bunch of it in your project *sips daiquiri*. Your customers will appreciate that you have a long-term plan that doesn't involve repackaging them as a product. If Pinboard were not a paid service, we could not have stayed up on December 16, and I would have been forced to either seek outside funding or close signups. Instead, I was immediately able to hire contractors, add hardware, and put money in the bank against further development. I don't claim the paid model is right for all projects that want to stay small and independent. But given the terrible track record of free bookmarking sites in particular, the fact that a Pinboard account costs money actually increases its perceived value. People don't want their bookmarks to go away, and they hate switching services. A sustainable, credible business model is a big feature. So that's the story of our big Yahoo adventure - ten million bookmarks, eleven thousand new users, forty-odd refunds, and about a terabyte of newly-crawled data. To everyone who signed up in the thick of things, thank you for your terrific patience, and for being so understanding as we worked to get the site back on its feet. And a final, special shout-out goes to my favorite company in the world, Yahoo. I can't wait to see what you guys think of next! * The list also included the mysteriously indestructible Yahoo Bookmarks, though that didn't seem to affect anyone. How Yahoo Bookmarks has persisted into 2011 remains one of the great unsolved mysteries of computer science. ** I should point out that Yahoo claims Delicious is alive and well, and will bounce back better than ever just as soon as they can find someone — anyone — to please buy it. Since the entire staff has been fired and the project is a ghost ship, I'm going to stick with 'demise'. *** This is because utf8 strings in MySQL can be up to three bytes per character, and MySQL has to assume the worst in sizing the row. **** I ended up extending the refund window by seven days and giving everyone a free extra week of archiving.

HOWTO Debug CSS You’re welcome.

(author unknown) — 2011-03-08T19:24:20-08:00

HOWTO Debug CSS

You’re welcome.

Gaming the System

(author unknown) — 2011-02-15T20:11:24-08:00

Shared by Barry Ferg
Ahh, Quake. Back in the day, exploding fellow cubicle-dwellers into glorious gibs certainly made work life more tolerable.

Frank slammed his axe into his co-worker's skull. Ernest grunted and raised his double-barreled shotgun in reply. "Merry Christmas!" he shouted as he fired both barrels. Frank exploded into several gore-colored polygons.

"Jerk," Frank grumbled as he waited for his respawn. It was late Decemeber, 1997, an era of before thumb-drives and when Quake was the best deathmatch money could buy. Normally, such lunch-time and break-time violence was frowned upon, but it was the holidays. When most of the office is on vacation, and the people that aren't just need to keep the lights on and not make trouble, you can get away with those sorts of things, so long as you uninstall it after the New Year. They Quaked away through the holidays.

Months later, the Auditor swooped down onto their floor. He arrived cloaked in darkness, and paperwork clouded his footsteps. His sign was a red floppy disk and the odor of rotting roses. "I'm here," he announced, "to check for unauthorized software."

Guided by the dark god of Audit Compliance, the Auditor started with Ernest's desk. He ordered Ernest to stand aside, inserted the floppy disk, and rebooted the machine. The machine booted from the disk, churned through the hard drive, and popped up a series of messages. "MALWARE DETECTED - INFRACTION LOGGED TO a:\audit001.log."

Ernest's stomach churned when he realized that the infraction was Quake. "Unauthorized Software" was a serious offense, but having games installed had sent people to the unemployment line. Ernest watched everything the Auditor did through a veil of terror, and started thinking about how to brush up his resume.

"Thank you for your time," the Auditor said.

Ernest wasn't the only one that had forgotten. The Auditor worked up one side of the aisle of cubes, and then started back down the other. Several groans and, "That isn't mine!" protests announced his victims. Ernest cast about, and saw Frank's cube right behind him. The Auditor was a few desks away.

Ernest rang Frank's phone. In a harsh whisper, he said, "Don't turn around. Share your floppy drive."

Frank turned around and stared at Ernest, even as he whispered into the phone. "What? Why?"

"I'm going to fix the audit."

"What?" Frank raised his voice and Ernest winced. "What?" he repeated, more quietly. "Use somebody else's drive. I actually cleaned out my disk, just like I was supposed to. I'm not getting fired for you."

"You aren't going to get fired," Ernest promised. "Don't do it for me. Do it for Dave, and Chris. Do it for John Carmack."

"Fine." Frank rolled his eyes and hung up.

Ernest quickly mapped the drive from a DOS prompt. As the Auditor stalked into Frank's cube, Ernest typed format x: /Q. He stared at the screen, carefully watching the faint reflection of Frank's cube. His ears strained to hear the "chunk" of the computer accepting the disk.

As soon as he heard it, Ernest slammed the enter-key. Even as the Auditor grabbed the mouse to shut-down the machine, Ernest saw Frank's floppy drive activity light shine. As the Auditor started the reboot, Ernest saw the best words he had ever seen appear on his own screen: Format Complete.

Nothing was actually deleted by the quick format. A quick run through with a recovery tool could easily fix it. Frank played dumb, and for the Auditor it came naturally. He sat with Frank for twenty minutes, absolutely convinced that Frank's floppy drive was non-functional. Eventually, he agreed that his disk was no good and tossed it out. "I'll be back," he warned. "And we'll have to start the audit all over again."

Ernest cleaned up his computer, and smiled, confident that his job had been saved. Unfortunately, a week later, it was revealed that the audit had been little more than an attempt to shrink the department down before everyone was laid off. Their twenty-person IT staff could be replaced by twenty people in Bangalore for a fraction of the cost. But a few lucky souls, like Ernest and Frank, were retained for an extra two months to oversee the transition. It provided them plenty of time investigate the newly released Quake II.

ongoing by Tim Bray · Broken Links

(author unknown) — 2011-02-13T20:49:03-08:00

Shared by Barry Ferg
Hear, hear.

Micromorts

(author unknown) — 2011-02-08T18:49:56-08:00

Shared by Barry Ferg
Good word. Now, how to work it into a conversation?

I'd never heard the term "micromort" before. It's a probability: a one-in-a-million probability of death. For example, one-micromort activities are "travelling 230 miles (370 km) by car (accident)," and "living 2 days in New York or Boston (air pollution)." I don't know if that data is accurate; it's from the Wikipedia entry. In any case, I think it's a useful term.

37signals Product Blog: We’ll be retiring our support of OpenID on May 1

(author unknown) — 2011-01-25T19:09:05-08:00

What Your 16-Hour Workday Says About You!

(author unknown) — 2011-01-25T11:18:33-08:00

jessiechar:

You’re a really hard worker

Your time is poorly managed

You don’t know what to do with your life

Your boss knows you’re gullible

At least 40% of your diet consists of pre-packaged food

You send out work emails at inappropriate hours

You have no perspective on life

You don’t sleep enough for proper brain function

You have very little self-respect

Your salary should be higher, but isn’t

You drink either too much or not enough

Farhad Manjoo is Right and I Will Go to This Barricade With Him

John Scalzi — 2011-01-14T11:21:24-08:00

The vile perniciousness that is the second space after a period. If you do this, you are everything that is wrong and bad in this world. That is all.

INSANELY awesome solar eclipse picture

Phil Plait — 2011-01-04T09:49:03-08:00

Earlier today Europe, Asia, and Africa got to see a nice partial solar eclipse as the Moon passed in front of the Sun, blocking as much as 85% of the solar surface. The extraordinarily talented astrophotographer Thierry Legault traveled from his native France to the Sultanate of Oman to take pictures of the eclipse. Why there, of all places? Heh heh heh. It’ll be more clear when you see this ridiculously awesome picture he took:

Holy solar transits! Click to embiggen, which you really really should do.

Can you see why he traveled so far to get this shot? The silhouette of the Moon taking a dark bite out of the Sun is obvious enough, as are some interesting sunspots on the Sun’s face… but wait a sec… that one spot isn’t a spot at all, it’s the International Space Station! This was a double eclipse!

That’s why Thierry sojourned to Oman; due to the geometry of the ISS orbit, it was from there that he had the best chance of getting a picture of the station as it passed in front of the Sun during the relatively brief duration of the actual solar eclipse. But talk about brief; the ISS was in front of the Sun for less than second, so not only did he have one chance at getting this spectacular once-in-a-lifetime shot, but he had only a fraction of a second to snap it!

To give you an overall idea of what you’re seeing here: the Sun is 147 million kilometers away (less than usual because this eclipse happened, coincidentally, very close to perihelion, when Earth was closest to the Sun). The Moon is 390,000 kilometers away. The Sun is about 400 times bigger than the Moon, but also about 400 times farther away, making them look about the same size in the sky. If you’re still having a hard time picturing the scale, take a look at the dark sunspot in the lower right of the big picture: it’s about twice the size of the Earth!

The space station, on the other hand, is 100 meters across (the size of a football field) and orbits about 350 km (210 miles) above the Earth’s surface. So the Moon was very roughly 1000 times farther away than the ISS when this picture was taken, and the Sun 400,000 times more distant. Yet all three lined up just right to make this extraordinary photograph possible.

Thierry has taken some of the most amazing pictures of the station passing in front of the Sun and Moon I’ve ever seen — his shot of the ISS and the Moon shortly before last week’s lunar eclipse was beautiful — but this one really stands out. It took an extraordinary amount of planning, scheduling, travel, and plain old good thinking to make this picture happen. Congratulations to him for getting it, and I thank him for sending it to me!

[More eclipse pictures can be found linked from Stuart's Astronomy Blog as well as on the BBC's Sky at Night Flickr pool of pictures.]

Related posts:

- When natural and artificial moons align
- ISS, Shuttle transit the Sun
- Check. This. Out. Amazing photo of the Sun!
- Shuttle and ISS transit the Sun

2010-12-26 11:38:39

nobody@domain.com (Cal Henderson) — 2010-12-26T03:38:39-08:00

americans are horribly misinformed about who has money (hint: the rich have nearly all of it)

Barry Ferg's shared items in Google Reader

The Rands Test

sunday fantasy #350: Enemy Ace. Art by Joe Kubert

The great bank robbery – Global Public Square - CNN.com Blogs

Presto (Pixar, 2008)My favorite cartoon. The most economical...

Sticker Pinpoints The Sweet Spot For Destroying Your Hard Drive

Leonard Nimoy Is Lazy As Hell [Video]

js/linux

One perfectly optimized gin n' tonic, three graphs

Techno Life Skills

Minimalist DC Superhero Poster - by Michael B.... – HeroChan

Anatomy of a Crushing

HOWTO Debug CSS You’re welcome.

HOWTO Debug CSS

Gaming the System

ongoing by Tim Bray · Broken Links

Micromorts

37signals Product Blog: We’ll be retiring our support of OpenID on May 1

What Your 16-Hour Workday Says About You!

Farhad Manjoo is Right and I Will Go to This Barricade With Him

INSANELY awesome solar eclipse picture

2010-12-26 11:38:39

Presto (Pixar, 2008)

My favorite cartoon. The most economical...