random($foo)

Keyboard Cleaner - plok [del.icio.us]

lhl — Tue, 13 May 2008 02:10:57 -0500

"I created Keyboard Cleaner. It comes in handy when you want to clean your keyboard, but you are afraid you might trigger a command or change or delete some of your current work beyond repair and undo."

Sad Trombone [del.icio.us]

lhl — Tue, 13 May 2008 01:49:40 -0500

best. website. ever.

Khoi Vinh -- Talk to the Newsroom -- The New York Times -- Reader Questions and Answers - New York Times [del.icio.us]

lhl — Tue, 13 May 2008 01:44:55 -0500

Where is the path (or street) - demo of OS OpenSpace API [del.icio.us]

lhl — Mon, 12 May 2008 23:10:24 -0500

The Ordnance Survey (official British mapping people) have opened up a lot of good maps in the UK. Here's a comparison of their maps vs GMaps. Impressive especially as you zoom in

Focation@Saigon [del.icio.us]

lhl — Mon, 12 May 2008 23:03:34 -0500

User-contributed maps for Vietnam w/ some interesting features (heatmappy POI grouping for example, very, very liberal contribution model - cultural differences?) -- definitely lots of opportunities for mapping in Asia...

Concharto - Home [del.icio.us]

lhl — Mon, 12 May 2008 22:58:21 -0500

a geowiki (w/ diffs!). pretty nice. "Concharto is an encyclopedic atlas of history and happenings that anyone can edit. It is a geographic wiki."

Remix - remix.nin.com [del.icio.us]

lhl — Sun, 11 May 2008 23:01:56 -0500

This and the new release of Slip (CC BY-NC-SA) in V0, FLAC, and 24/96 WAV (!) are ... well, awesome.

Catching Up On Reading

Sat, 10 May 2008 00:12:45 -0700

In between the projects and mini-projects, I've also begun to do some catching up on reading. Carrying around a book everywhere and taking lots of public transportation has helped tremendously in that regard. I started with Shirky's Here Comes Everybody a few weeks ago, just finished Weinberger's Everything Is Miscellaneous (that's been sitting around since his original book tour when he came to Yahoo!) and polished off Cory's new YA novel Little Brother in a single sitting (what's so compelling and chilling is the realization that we're about half-a-step away from a sort of grim-meathookiness. It gave me more of an appreciation of the fiction coming out of the 80s). And I'm now starting Glut.

One thing that's interesting is that all these books have related/complementary media (podcasts, talks, etc.) attached to them (and all worth the time spent, I'd say, particularly Alex Wright's Web That Wasn't presentation). Now, obviously anyone who's done much spelunking on Wikipedia (lately, I've begun doing a bit of random clicking around on Slideshare as well), or you know, random Internet browsing can tell you there's a lot of "stuff" out there. However, I thought I'd share a list of some of the more useful/structured resources I've found (online video, lectures, etc):

This post sort of got kicked off while I was watching this very engaging recent talk (via) that I can't favorite because YouTube still has a 500-favorite limit (now below 0.6 favs/day):

prspammers wiki [del.icio.us]

lhl — Fri, 09 May 2008 21:55:10 -0500

Yay Gina! "In the spirit of sharing, here's a current list of PR companies who have sent me unsolicited (and almost always irrelevant) product pitches. Please add your own."

May 2008 Mix

Thu, 08 May 2008 20:39:32 -0700

It's been a good couple months for music. Sometimes mixes don't really have a name or a theme. Here's some stuff I've been digging lately. 43 minutes.

Oh yeah, trying Schiller's latest SoundManager 2 release (mostly based off of the inline player) and some custom posting code. (Mixwit doesn't seem to have an API?)

01 - Martina Topley Bird - Phoenix
02 - Zeigeist - Bunny
03 - Santogold - You'll Find A Way (Switch & Sinden Remix)
04 - Hadouken! - Declaration Of War
05 - Styrofoam - A Thousand Words
06 - The Notwist - Gloomy Planets
07 - The Bird And The Bee - Birthday
08 - The Paper Cranes - 100 Years War
09 - Wolf Parade - The Grey Estates
10 - Murder By Death - A Second Opinion
11 - Cloud Cult - Love You All
12 - Chris Walla - A Bird Is A Song

Tech note: for those that are looking to use SoundManager, you might want to check out to see if you're build has the onpause handler. If it doesn't you can add it to the SM the defaultOptions at the top and then call it in the pause() function around line 727 (just apply the onpause the same way the onstop above it is applied).

The Guardian wants you to play GTA IV. Destructoid offers reviews, previews, trailers, cheats, and more. [del.icio.us]

lhl — Thu, 08 May 2008 21:57:52 -0500

The Guardian is a British newspaper which has now run its third pro-gaming piece of recent weeks.

A Universal Messaging Hub « The Abstract Truth [del.icio.us]

lhl — Thu, 08 May 2008 21:56:12 -0500

In conjunction with complex-event processing systems, message switching will have a revolutionary impact on the efficiency of distribution and delivery of real-time data around the web.

one small voice » Microblogging Over XMPP [del.icio.us]

lhl — Thu, 08 May 2008 21:54:28 -0500

There’s been plenty of talk recently about using XMPP to build a decentralized microblogging platform (think Twitter busted apart to run as a distributed network of microblogging providers). Indeed, as Bob Wyman points out, we have all the pieces

XEP-0060: Publish-Subscribe [del.icio.us]

lhl — Thu, 08 May 2008 21:53:25 -0500

This specification defines an XMPP protocol extension for generic publish-subscribe functionality. The protocol enables XMPP entities to create nodes (topics) at a pubsub service and publish information at those nodes; an event notification (with or witho

Transporting Atom Notifications over the Extensible Messaging and Presence Protocol (XMPP) [del.icio.us]

lhl — Thu, 08 May 2008 21:52:54 -0500

Transporting Atom Notifications over the Extensible Messaging and Presence Protocol (XMPP)

Twitter Can Be Liberated - Here’s How [del.icio.us]

lhl — Thu, 08 May 2008 21:52:13 -0500

decentralizing statuscasting...

Consola - The Open Source Bluetooth Proximity Media Server [del.icio.us]

lhl — Thu, 08 May 2008 21:11:40 -0500

Jabberwocky [del.icio.us]

lhl — Thu, 08 May 2008 21:11:03 -0500

Jabberwocky is a freely available mobile phone application designed to promote urban community connections and a sense of familiarity, anxiety, and play in public urban places.

Hackety Hack: the Coder's Starter Kit [del.icio.us]

lhl — Thu, 08 May 2008 21:10:18 -0500

One of Hackety Hack's sincere pledges is to make the most common code very easy and short. Downloading an MP3 should be one line of code. A blog should be very few.

hackety org » The Tiny And Accomplished Lew [del.icio.us]

lhl — Thu, 08 May 2008 21:09:28 -0500

LEW is an art tool which embeds the Lua language and it’s inspired by (you guessed it) NodeBox and Processing. And BASIC. The gallery is incredibly impressive. And equally impressive is the brevity and usefulness of its manual.

Publisher Tested the Waters Online, Then Dove In - New York Times [del.icio.us]

lhl — Thu, 08 May 2008 21:08:23 -0500

Today, I.D.G. says, the InfoWorld Web site is generating ad revenue of $1.6 million a month with operating profit margins of 37 percent. A year earlier, when it had both print and online versions, InfoWorld had a slight operating loss on monthly revenue o

Brightkite: A bright future for mobile social networking? [del.icio.us]

lhl — Thu, 08 May 2008 21:06:32 -0500

(eh)

Adium Productivity Tip

Mon, 05 May 2008 17:29:11 -0700

Recently, in my quest to minimize distractions, I've been going through and ...fixing things. First was updating my procmail and my spam filtering so that the only mail that ends up in my Inbox is real mail. (Inbox Zero for the past month!) And here's what I did for Adium so I could leave IM on while I was working:

Turning off dock animations (no more bounces) and hiding Adium while in the background (no more window popping) were really huge. I also have my dock hidden, which helps as well. (The menu bar gives me enough cues so that I can see what's going on, but that isn't too bothersome while I'm trying to work.)

And of course, I have my work.py script, which lays out my Terminals appropriately. At some point I might start scripting Space switching for that, but at least right now, this seems to be working out OK.

Flickr Favs: Thu May 01, 2008

Thu, 01 May 2008 09:00:13 -0700

Web 2.0 Expo Presentation Rundown

Wed, 30 Apr 2008 18:17:58 -0700

There were actually a surprising amount (to me, at least - most of the people I talked to had low expectations) of very good presentations at Web 2.0 Expo last week. Most of them are now posted on either SlideShare (more presentations here) or, for the keynotes, on Blip.tv. This is I think, a very exciting and positive development for industry conferences (which I think will only have net-positive effects on attendance; conference proceedings are de rigueur at academic conferences). Here're the ones I thought were most interesting.

Keynotes (overall, I liked the 10min What X Knows format that asks companies to boil down numbers and insights):

Clay Shirky (NYU) - Hands down my favorite keynote of the conf
What Tellme Knows - loved these numbers
Tim O'Reilly - standard what Tim is interested in talk, but great if you haven't heard it
Max Levchin (Slide) - the affinity anecdote and the discussion of business goals talk is pretty good
What AIM Knows - the charts are nice as is the architecture slide
Marc Andreessen (Ning) - wide-ranging conversation that covers a number of historical and current topics
What Current TV Knows - people like watching themselves on TV! The first half is better than the 2nd.
What Dash Knows - good to see driving collective intelligence actually working
What WP Knows - some good WP numbers
Jonathan Schwartz (Sun) - wide-ranging conversation on hardware/infrastructure
Scott Berkun on Innovation - if you haven't heard his spiel, there's a nugget or two in there
Fake Steve Jobs - overly long, but the core story, about 15min in is good
What MySpace Knows - only interesting part is about 50s to 2min to see the #s slides

Lots of awesome sessions, the quality of the presentations (primarily in terms of prep/interestingness) was higher than usual:

A Flickr Approach to Making Sense of the World - my favorite session of the conference. If you're doing "geo stuff," you owe it to yourself to take a look at this. The divisive hierarchical agglomerative clustering bit is great (using morton curves for better pathing, clever). Now there's not a lot on reverse-geocoding, which I believe I am now doing unique and interesting work on -- once I prove it works, I'll have to publish/present about that. :)
Capacity Planning for Web Operations - sure you can't clone Allspaw, but reading what he has to say is probably the next best thing.
Website Psychology - linking to an earlier version of Gavin's talk (with notes, yay) - he does a really great job mapping cognitive psychology concepts onto site usage and development. Well worth reading and thinking about
Grasping Social Patterns - by far my favorite Ignite talk this year, all kinds of hooks for thinking about how far social apps and the "social graph" needs to go
Making Email a Useful Web App - Bots are awesome and underrated. I've been working a lot more w/ them recently and this was a good overview (would love an even more comprehensive history of cool bots...)
Even Faster Website (PPT) - Steve Souders (now at GOOG, doing the same sorta thing he was doing at YHOO) talks about the current stuff he's working on, which is optimizing JS (the logical progression). Great new stuff, just as useful as the older stuff
Adding "Where" to Mobile and Web Applications - a bit basic, but a good overview of how location stands today. Come to Where 2.0 and Wherecamp to learn more...
Polite, Pertinent, and... Pretty: Designing for the New-wave of Personal Informatics - slides not online. Boo-urns
Casual Privacy - slides not online. Boo-urns
Next Generation Mobile UIs - slides not online. Boo-urns

Talks I didn't make but that have interesting decks:

Tagging: Opportunities & Challenges of Scale - Kakul gives an interesting overview of tags, growth patterns and future directions (super-interesting stuff) at Flickr. Unfortunately, there was a conflict so I couldn't make it, but it looks pretty darn interesting; would be interesting for an open source library to support typing via machine tags, and clustering (I assume Flickr just k-means on photos w/ "tag")
Why Startups Need Automated Infrastructures - this conflicted w/ Kellan's talk otherwise I woulda been there; building my own one of these right now...
Startup Metrics 101: a Product & Marketing Workshop - this densely-packed deck that is chock-full of info. great overview from a business-metrics perspective
Failure Happens - fun talk on what HA means lots of infrastructure details (colo to deploy stuff mostly)
Copy as Interface - Erika Hall gives lots of specifics on writing gooder for the web
Customer Service Is The New Marketing - great slides and clear/focused message
The Dark Side of Ajax - great overview/analysis of XSS/CSRF problems with AJAX and a looking at which JS frameworks are addressing issues
Cross-Cultural User-Experience Design - great list of references, very interesting comparisons
Open Platforms in Web 2.0 - highlights the big story of 2008-2009: really being able to write your brilliant additional feature w/o having to rebuild everything; tying together services and the social context...
Do Try This At Home - interesting FE hacking; looks like this would definitely benefit from audio/preso notes
Announcing Project Z - open dialogue and deliberation framework? looks potentially interesting. This was the type of thing I was very much into a few years ago.
Mobile Ajax and the Future of the Web - actually has useful dev tips
Blogging for Personal Branding - interesting seeing insights from an actual pro-blogger (not like pro-bloggers I know, but the people actually writing about specific topics)
Designing Your API - a couple interesting tidbits from both sides.
Triggit - the most interesting LaunchPad demo
RIA Offline Desktop - decent overview of AIR and Gears
Mobile 2.0: Design and Develop for the iPhone and Beyond - megalong overview on mobile web; this was a tutorial session, which I was sort of at, but mostly busy working through

Some stuff that sounded interesting but don't have slides online: include Marc Davis' Mobile talk, Opportunity Computing in the Cloud, Social Networks and Avatars (caught a few min of this, looks like they haven't done a lot of work on the numbers (even organizing across cohorts), but still would like to see the deck), Global Design Trends (there are slides, but only enough to wish you had a recording of the talk)

Some bonus talks if you've made it through all those:

Alternate Realities - Jane McGonigal gave one of my favorite keynotes this year @ SXSW
Evaluating visual cues for window switching on large screens - a pure HCI talk on UIs once you have wraparaound screens (ie 3 widescreen LCDs). pretty awesome.
Real World Web: Performance & Scalability, MySQL Edition - Ask gave a great talk at MySQL 2008 - a little bit of everything in this :)
Collective Intelligence Indeterminacy and the Illusion of Control - on "sociomimetic" technology, w/ a 3pt rubric for evaluation

Did I forget something (quite probable), miss one of your favs? Post links in the comments.

Digging out Old Crap

Tue, 29 Apr 2008 12:45:53 -0700

I was doing a search and came across some old notes/presentations that I wrote back in 2003 (references)

CTIN 511: Social Software, Theory and Practice
- Social Software: Theory and Practice - A brief overview of "Social Software"
- Next-Generation Distributed Social Software Networks: Designs and Applications - A presentation on designing (and reasoning behind) a distributed (decentralized) social network concentrating on infrastructure and development
CTCS 505: New Media and the Consumption Cycle
The arrival of widespread digital and inter-networked media has dramatically reshaped the nature of spectatorship and cultural and media consumption, alternatively amplifying, accelerating, shifting, disrupting, and in some cases, inverting the consumption/production/distribution cycle. This cycle has become increasingly interlinked, and the distinction between these activities increasingly blurred.

Honestly, 5 years isn't a long time ago (time flies), but it's interesting to look back at my thinking at the time, and also to see where we are. There are some things that have happened that I wasn't expecting, and some that I would have expected to have happened by now (honestly, I expected Digital ID consolidation by 2008, and that's not gonna happen anytime soon)... Good times.

One thing led to the next, and soon I was looking through an old Recommended Reading List I had started making. That actually seems like a good idea, and I'll be working on one as I have time (I'll post about it once I've hashed out the basics).

Along the way of all this searching I found a dead link to an old UNIX tutorial I had written. I have a tarball of it somewhere, but honestly, it was much easier to just grab it from the Wayback Machine. So, here it is, for later reference: Stupid Unix Tricks.

It looks like it's aged pretty well (better than these old webdev workshops I also taught), although it does remind me that I need to publish my latest bash setup sometime (and maybe start writing down the new things I pick up before I forget them). The only big change (vs addition) is that when I'm not using cmd-line file transfers on OS X these days, I use Cyberduck (an open source SFTP, FTP, WebDAV, S3 client).

And lastly, where would a nostalgia post be without some old music? I've recently finally started digging through old boxes and am encoding all the CDs I lay my hand on. Here's a track, part of my "catching up on rock" period in the early 2000s:

Vitreous Humor - Sharin' Stone

Clay Shirky on Gin, Television, and Social Surplus

Mon, 28 Apr 2008 00:11:58 -0700

Clay Shirky gave the best keynote talk that I caught at Web 2.0 Expo last week. He's posted a transcript, entitled Gin, Television, and Social Surplus on his new book's site (also quite recommended; it makes it onto my "understand the internet" bookshelf).

So how big is that surplus? So if you take Wikipedia as a kind of unit, all of Wikipedia, the whole project--every page, every edit, every talk page, every line of code, in every language that Wikipedia exists in--that represents something like the cumulation of 100 million hours of human thought. I worked this out with Martin Wattenberg at IBM; it's a back-of-the-envelope calculation, but it's the right order of magnitude, about 100 million hours of thought.

And television watching? Two hundred billion hours, in the U.S. alone, every year. Put another way, now that we have a unit, that's 2,000 Wikipedia projects a year spent watching television. Or put still another way, in the U.S., we spend 100 million hours every weekend, just watching the ads. This is a pretty big surplus. People asking, "Where do they find the time?" when they're looking at things like Wikipedia don't understand how tiny that entire project is, as a carve-out of this asset that's finally being dragged into what Tim calls an architecture of participation.

If you didn't catch it, this is well worth reading.

Donations Done!

Sat, 26 Apr 2008 22:43:33 -0700

After Super Tuesday, I started a little fundraising campaign where I promised to match contributions dollar for dollar up to $4K (since $2K was about what I had left on my campaign contribution limit). I'm happy to announce that the donations passed the goal tonight (15m after a last call tweet).

The final stats: $2070 donated by 19 people (all friends or friends of friends; an average of $109). With $2070.19 matched by me.

Hooray! Gobama!

Also, since it's hilarious (although actually, Obama won Delaware as well):

Internet Asshattery, Armchair Scaling Experts Edition

Fri, 25 Apr 2008 07:00:00 -0700

I know it's never good to pay attention to the nattering classes, but there was a pretty high profile fusillade that Mike Arrington launched on Blaine Cook which seemed to bring out the arm-chair experts in full force in the comments. Now, while I think that Arrington's post is way out of line (I'll explain that in a bit), I'm almost not as bothered by it (as long as he's not to bothered for being called out on it)... What really bugs me is the number of clueless "developers" throwing in their two cents. That includes Arrington's two Rails developers with "finger on the pulse of the rails community" (ha!). My discontent was further exacerbated by this (unrelated) completely clueless piece on The Register. Is this the best that tech journalism has to offer?

First, a disclaimer: I don't know Blaine very well, and I don't have any privileged info on Twitter or Obvious Corp.

There's no question that Twitter has and continues to suffer from capacity, load, and other stability issues, and pointing that out is fair game, however pointing at Blaine's scaling talk as a personal dig is a disservice to the everyone, especially since:

The advice in the slides are generally good (and the "It's Easy" is obviously snark - just look at the failcat in the next slide; it'd be easy to confirm that by asking anyone who was in the talk (like me or several hundred other people) instead of projecting prideful boasting to justify his attack -- I'll avoid ascribing motivations to why Arrington chose to do this).
More crucially, the slides themselves point to the issues that a proper tech journalist would be able to spot and follow up on to try to find out what was really going on (assuming he cared about that).

For example, 600 QPS on 8 machines is pretty decent - but this raises the question of utilization and capacity planning. You can see from the 1x1 MySQL structure and the note on DRb that there were many single points of failure - again, this raises questions of BCP and redundancy. With the constant bumping of limits, you could guess that they were running really hot (and from a single data center, even after the move (probably w/o backup routers, etc.)) -- all these issues are as much (if not moreso, since these are technical no-brainers) business/financial decisions than architectural/technological ones.

Now, I don't know what happened between ops, management, and engineering, but guess what? Arrington doesn't either, and he never bothered to follow up and kicks Blaine in the head instead, even when such clues obviously raise significant doubts about whether it's appropriate. I agree with Arrington's point about accountability, which is why I say now that Arrington wasn't posting journalistically (the minimal followup with someone w/ half a clue would have pointed out exactly what I did), and Blaine deserves an apology. If you're gonna shit on someone and start pointing fingers, you better have the goods to back it up. Whiny, uninformed personal attacks belong on Arrington's Live Journal or (wait for it...) Twitter stream.

Now, onto the retarded comments from wanna be developers. Well first, of the entire thread, I only saw one half-decent attempt at a technical critique, and even that falls down when you look at it. I don't want to belabor the point, but the poster, Jordan, actually raises technical points worth addressing (and refuting):

On indexing: while it's true you don't want to index willy-nilly and it's incomplete to say "index everything", if your ORM isn't automatically indexing frequently used keys, you can be sure as heck that you'll want to make a point of indexing them, especially if you're doing joins. Yeah, you don't index what you don't need, but even if you have frequent writes, you need to eat it if you're ever going to ever query. Because people suffer from lack of indexes, unless you're not adding an index and examining, you're not gonna have a problem "over indexing." I don't know the exact fan-out/pub-sub architecture, but you can be sure you'll be doing a lot more reads even if you cache the hell out of it. If you're thrashing, you're looking at having mis-configured index caches more than anything else.
DRb: This is a case where it looks like he just misread. It's easy if you don't have the context of the talk. DRb was good enough... until it wasn't - which is why Starling was written to replace it. Now, we still don't know if it's a single point of failure, but it obviates that whole rant (as to why DRb was chosen in the first place, more on that later)
Caching: again, the same thing with indexes. Of course over-caching is bad, but that's never going to be your problem because you start with no caching, and you add caching until you start losing performance. Also, the "no substitute for fixing the underlying problem" is naive - most of the time, your problems are that there's no need to do complex queries or processing since the data doesn't change and should be cached. durrf.
Profiling: ok, this I'd sorta agree with. Mentioning ruby-prof would probably be good, but honestly, 90%+ of optimizations can be done on simple timers, explains, and logs alone. (And also, performance tuning doesn't have all that much to do with scaling anyway.)

As to the rest of the wannabees, it really is true that if you haven't done it, that is: been intimately involved growing a social web app from prototype to Internet-scale on a UNIX stack, then you really don't know shit. (I know more than my fair share of people that have, and I didn't see any of them posting armchair bs on the comments). I'm not trying to say this just to be dismissive, but only to say, you really really, don't understand the technical challenges involved. Generating target sets on social objects is extremely expensive and ill-suited to traditional 4NF data models in RDBMSs. So is social activity fan-out and any number of activities core to Twitter's message routing/storage and to social web apps in general. These are not traditional problems and standard, HA solutions just aren't available.

Even if you're architecturally sound, you're dealing with development with extremely tight timelines/pressures, so you have to make decisions to pick things that will work but will probably need to eventually be replaced (e.g. DRb for Twitter) -- usually you won't know when and what component will be the limiting factor since you don't know what the uses cases will be to begin with. Development from prototype on is a series of compromises against the limited resources of man-hours and equipment. In a perfect world, you'd have perfect capacity planning and infinite resources, but if you've ever experienced real-world hockey-stick growth on a startup shoestring, you know that's not the case. If you have, you understand that scaling is the brick that hits you when you've gone far beyond your capacity limits and when your machines hit double or triple digit loads. Architecture doesn't help you one bit there.

And the people that have experienced this and lived to tell the tale also know that it's impossible to critique the technical/operational aspects made w/o seeing and understanding the QPS targets, load graphs, profiling data/sar info and all manner of other architectural/technical data and details (that none of us are privy to) before commenting with any sort of authority.

Anyway, if you were given the choice of working with/hiring someone like Blaine who has had the firsthand full life-cycle scaling experience and any random developer (and definitely anyone from the TechCrunch comments), I think it's fairly obvious what the right decision would be. I guess I'll leave it at that.

This leads to Part Deux of my rant... this lead-paint baby of an article entitled Backlash starts against 'sexy' databases which has the following quote, I shit you not:

"The bottom line is don't tell me RDMBS [sic] can't scale if you can't write a decent query or design a normalized database schema."

This is by one John Holland. Now, no doubt the WordPress code can be pretty shitty (although sometimes there are good reasons for the multiple queries to support various hooks/plugins), but you will never hit the type of performance problems in a WP (non-mu) installation that have people looking for MySQL alternatives because WP just doesn't have the types of queries that destroy RDBMs.

I can understand that it's not the article author's (Phil Manchester) fault for conflating the "cons" with arguments that WP is badly coded with the "pros" (correct!) that you can't write the kinds of queries you need for social apps because if he's like the reporters I know, he probably doesn't actually understand it at all and is doing his beat writeup, but dammit, can't the author get some decent frickin' technical advisors to explain this if he's doing tech journalism? The entire article is based on characterizing a misinformed blog post as a "brewing controversy."

I mean, I don't want to be more mean than I have to about this, but John Holland just has no idea what he's talking about. He picks up on Atwood's post on WP inefficiency, and then uses that to (completely incorrectly, and not without a tinge of reverse elitism) generalize on why the "cool kids" are hyping non-relational data stores. He goes on to boldly state "Relational databases are not the bottleneck" due to his complete lack of understanding of the actual problem set (hint: I don't know anyone who's suggesting WP should be switched off MySQL). This then leads to a horribly ignorant article being published by a writer who is in the best case, lazy and doesn't understand what he's reporting (just show two equal sides and do a writeup) or in the worse case is simply looking for a manufactured conflict that only will serve to stir controversy and confuse the non-savvy reader.

(The reasons for alternative data-stores actually exist in a couple axes - one is for more development flexibility or the ability to change functionality w/o expensive downtime (schemaless), one is for issues of scale and availability (distributed), and then a whole bunch for supporting social queries that just are horribly suited to RDMSs (multi-attribute, inverse index, mq/pubsub, etc.). Many of the alternatives are a combinations of various axes.)

A Free Lunch: Just Do The Numbers

Thu, 24 Apr 2008 20:57:58 -0700

Yesterday Gordon commented on Silicon Alley Insider's back of the envelope calculations for Google's food costs. IMO they're not doing the right numbers. I spent a fair amount of time fighting (somewhat unsuccessfully) this sort of retarded thinking, so here's a copy of the back of the napkin numbers that I just did. Here's a copy of what I posted on Gordon's site:

I think they're not doing the right numbers. GOOG headcount is at 18,000. If half the workforce ends up working for an extra 1 hr/day (it's probably closer to 2) b/c of breakfast and dinner, and assuming that an FTE hour is worth $50 to Google (probably low since the fully loaded cost for the avg knowledge work is probably about $100/hr, and the Goog's profit margin is >25% and that's including all operating costs), that means:
9,000 employees * 251 days * 1 hour * $50 = $113M
If it does cost $75M (again, we'll use the worst case number, even if it's more likely 70% of that), we're talking about a 50% return on investment. Since we're using worst case numbers, the direct returns are probably much higher.

And that's before calculating *any* second order benefits like: better retention, increased loyalty/productivity, increased morale/quality of life, increased reputation/easier hiring and recruiting - and of course free write ups. By almost every set of these important (but harder to quantify) metrics, they is a non-trivial qualitative improvement. Even if it were a complete wash, any knowledge company that could^* would be (and are IMO) stupid for not following suit. Any proper calculation would properly include/calculate offsets for these externalities (certainly recruiting/retention cost projections would be easy to model).

Finally, speaking from personal experience, I doubt SAI realizes how disruptive and hard it is to replace (recruit/hire/train/etc) high performing engineers (especially w/ the huge ramp up w/ the custom systems and processes at a cutting edge tech firm)? Does anyone realize how non-linearly time scales and what it means in terms of development productivity (talking to lots of people, and certainly for myself, I find it takes me a couple hours for me to ramp up into a flow state, after which I get more productive (until a certain point - but again, depending on blood sugar)). Anyway, I'll stop railing against stupidity. I'm sure Google has run the numbers themselves. They do love doing that, so I hear.

Based on my napkin calculations, and the failure of other organizations to grasp the work environment tactics that Google employs, my conclusion is that everyone (employees, the corporation, and the shareholders) at Google are very much enjoying a "free lunch."

(* Where "could" means a company that could successfully profit from the additional man-hours and the reduced turnover costs. While the former may be a smaller slice, I would suspect that once you add retention, almost every company would be over the line.)

April 2008 Mix (The Way I Rock)

Thu, 24 Apr 2008 03:30:28 -0700

The other day, ph wanted to some new music suggestions. When I asked him to name some bands that he liked, he named ABBA and Michael Jackson. Now, while this seems suspiciously like the response of someone who doesn't need any new music, I decided to interpret it as a "call for help." It's time for a new mix anyway, so what the heck... Here's a quick try at putting together a mix of (mostly) not-too big artists/songs together. It's poppier and more major key than my regular mixes. :)

Sorting Lists of Lists in Python

Sun, 20 Apr 2008 11:22:25 -0700

No language is perfect, and Python is no exception. In Python 2.4, for example, sorting a list of lists sorted by a value in the nested list (more common than you'd think), requires a bit of work and a library:

import operator
max = sorted(list_of_lists, reverse=True, key=operator.itemgetter(1))[0]

Python 2.5 makes things easier:

max = max(list_of_lists, key=lambda x: x[1])

The official Python Wiki has a nice HOWTO on Sorting in Python (also a much less comprehensive one on sorting dictionaries).

(note: OS X Leopard comes w/ Python 2.5 while, Debian Etch remains on 2.4 by default)

Some Cygwin Setup Tips

Sat, 19 Apr 2008 14:55:25 -0700

I don't touch Windows much anymore (games and demos in Boot Camp, and EAC and a few other programs in VMWare), but when I'm in there, cmd.exe serves as a constant reminder that I'd rather be elsewhere (Console is an open source attempt to improve the default console). The obvious solution is to bring your UNIX tools with you, which is where Cygwin comes in. Here are some setup notes.

I make a setup folder in C:cygwinsetup, otherwise it just dumps the it's packages in a folder where you're running the setup.exe from
Some "essentials" that don't get installed by default: Devel/subversion, Net/openssh, Net/rsync, Shells/rxvt
The last one is important because by default, the default blue is too dark (of course). Here's directions on setting up a proper rxvt .bat file, and then the relevant lines you want in your .Xdefaults:
```
rxvt*color4: SteelBlue1
rxvt*color12: SteelBlue1
```
These change the dark blue and blue defaults respectively (I also like RoyalBlue -- here's a list of XTerm colors; you can also enter hex colors directly if you'd like) Also, for a good list of what color0-color15 are. (aside: this is the correct answer to an unanswered Ask Metafilter question from a few years back, but I guess there's no way to comment on old closed questions)
The other big thing is file permissions, which I've only run into when rsyncing (I use -avP as my default copy mode). I haven't done anything with that except for making aliases that chmod 755 and 644 before I need to make copies.

Also, while Fusion has it's own fast launch (cmd-L), if you're in Boot Camp, I recommend Launchy, which helps a bit w/ the Quicksilver withdrawals.

Flickr Favs: Sun Apr 13, 2008

Sun, 13 Apr 2008 09:00:02 -0700

Getting Started w/ Python

Mon, 07 Apr 2008 23:03:19 -0700

As you might have heard, Google AppEngine launched tonight, with Python as its initial (and only) programming language to interface with its services. I started switching over to Python (from Perl) a few years ago for general processing and daemon tasks (mostly for its sweet RPC bindings and its comprehensive, if still somewhat convoluted Unicode handling). Over time, as the libraries matured, I started moving more and more over - some things were long overdue, like a CPAN equivalent (pypi and EasyInstall have finally stepped up to the plate), but in some areas, like with cross-platform GUI toolkits, things like py2app/pyexe, or with libraries like Twisted, and SciPy, and Beautiful Soup, Python has long since blown past the competition.

Earlier this year, as I was wrapping up at Yahoo!, I knew I wanted a clean start, and after reviewing what was out there decided on switching to Python as my primary language and making a go of writing my new web apps in Django (deployment, performance, and decoupling being among the primary factors; less wankery in the development community was also a big part of it). I've been somewhat sidetracked by a slew of other projects, but so far it's been a good experience (and I hope to have some stuff to publish soon).

Anyway, all this is a very, very, long setup for a list of resources that may help those who are looking to get started working w/ Python. I'm still not as proficient as I'd like, so here are the references that I typically reach for:

PLEAC Python - PLEAC (Programming Language Examples Alike Cookbook) is a project that aims to port the Perl Cookbook to other languages. The Python port has been at 85% for years, but is invaluable when looking at basic constructs.
(the eff-bot guide to) The Standard Python Library - although a bit out of date and not comprehensive, it offers short and useful examples for most of the modules in Python. This is great because often times the official library docs while technically complete are also at times completely opaque. If I were to give any advice to people writing API docs, it would be to 1) have some simple real-world usage examples and to 2) allow user annotations (PHP was (and remains!) way ahead of the curve on this one. It's amazing how primitive the core language/library docs are.)
Dive Into Python - I waffle back and forth on how much I like Mark Pilgrim's book - it's oftentimes just short of useful and not organized so well (I'm still looking for a good language reference), but it also has really useful tidbits, like when I forget how to append the system import path
Python-by-example - this is a new one, and I haven't used it much (inline-search would do wonders for this) but I wholly approve of the intent: "This guide aims to show examples of use of all Python Library Reference functions, methods and classes."
Otherwise, I've found that doing a web search almost always turns up something on ASPN or on a mailing list somewhere.
Lastly, there are some interactive shells that are useful, specifically IPython. Reinteract is less of a tool that I use everyday and more of something that's damn cool. The same w/ Nodebox.

Of course, one of the biggest benefits of Python is how readable the source code is - it's definitely a big help for seeing how things works. Have any of your own favorite Python resources? Please post 'em up on the comments.

Getting up to speed on Django probably deserves its own post...

R.E.M.

Mon, 07 Apr 2008 00:56:59 -0700

I've never been a huge R.E.M. fan, most probably because I was introduced to them by their Out of Time radio hits (Losing My Religion and Shiny Happy People still annoy the shit out of me) -- I remember listening to their followup albums Automatic for the People and Monster, and then losing interest. I didn't have anything against them, and over the years they've popped up on my radar from time to time (more for their videos than their songs -- Everybody Hurts still holds up as an amazing work, and more recently, the Hammer & Tongs directed Imitation of Life was also quite good), but they they've just never resonated with me all that much.

That being said, after reading an interesting discussion on early R.E.M., I chased down their pre-90s work and will have to spend a day sometime giving it a listen. Also, their new album (Accelerate) isn't bad (single). I've only given it one spin so far, although I've been deluged w/ new stuff lately and I guess in a different headspace at the moment (new things on rotation: Styrofoam, Cut Copy, M83).

Here's a passage from a last.fm journal linked from the discussion:

Much like a hat I used to own, R.E.M. was perfect because it fit perfectly into the space where nothing used to be. They drew on the energy of punk without its harshness and nihilism, replacing growling distortion with chiming Byrds guitars and hoarse, angry shouts with pretty harmonies, and filtered out the cornpone from country/roots, leaving only its forthright beauty, in a way that made perfect and astonishing sense to white Southern kids of a certain age, to whom real country was a cliche beloved by redneck uncles and to whom real punk, however satisfying its aggressive pleasures, was music about New York, and London, and Southern California, exotic places that bore scant resemblance to the tree-lined streets of our hometowns. R.E.M.âs sound validated a kind of modern-South lifestyle that we were already living, and made it seem both mythic and earnestly real. You couldnât buy drugs that did that. Not consistently, anyway.