New Database Server

We made our first hardware purchase of 2009 - a new database server from Silicon Mechanics. I chose a 1U Intel Nehalem machine and filled it with 2.5″ 15K SAS disks.

I’m really excited to see how this will stack up against our current hardware (which is 18 months old, on average) I’ll be sure to run some benchmarks and post them. I also want to compare Xen to KVM to no virtualization. We started using Xen over two years ago and it looks like KVM may be pulling ahead performance-wise.

We’ve grown a lot during the last year and faster disks and more memory in the master database will give us a lot more breathing room. It will also allow us to turn our current master DB (which is still a nice machine) into a slave that can actually be used to take on part of the load. Our current slave is great for backup and feeding the Sphinx search engine but its SATA disks are just too slow to handle any of the site’s database traffic.

While I was looking at servers, I noticed that nice Intel SSDs are pretty affordable for machines that don’t have big storage needs (app servers, etc) or database/file server IO needs. ..especially when you compare them to RAIDing 2 hard drives for redundancy. I didn’t really understand why SSDs varied so much in price and performance until I read this LWN article: “Log Structured File systems: There’s one in every SSD“. Check it out.

Redis

We have our own ad-serving system at Ravelry - it allows advertisers to reserve their spots, target certain locations or groups, and change ad images on the fly. We like to think that our ads are interesting and cool enough that people actually want to return to one if they catch it as they leaving a page.

A long time ago, we had a feature that allowed people to “rewind” ads in order to bring back an ad that was previously displayed. We had to throw it out when it became too database intensive to make the feature work.

Thanks to Redis, I was able to bring this feature back with very little code. In one operation, I can refer to a list of viewed ads for a particular user/targeting and say “remove this ad from the list if it exists, push it to the tail, and trim the list to X items”. It’s really cool. The Redis page calls it “a data structures server” and I think that this is a very fitting description. Check out all of the operations: http://code.google.com/p/redis/wiki/CommandReference

If you are running Redis, here is a munin plugin that will graph connections and memory usage. (direct link to the script)

Amazon S3 sticker shock

Our Amazon S3 bill is getting really high. Too high - we spend a small fortune on serving up images.

I’ve been beginning to front our S3 buckets with a caching nginx proxy connected to the cheaper of our two ISPs. That way, Amazon will still handle the storage but we can serve the most-accessed files from our own bandwidth.

Swapping back and forth is easy because we just have to alter the behavior of AssetTagHelper.

It’s too early to tell how much this will save us because I can’t relaly handle the traffic right now. Turns out that Amazon is taking care of somewhere around 100 Mbps of file serving during peak times Once the new database server is installed, we’ll be able to do some shuffling and repurpose an existing virtual machine as a cache.

I posted an answer over on StackOverflow that explains how to do this: “How to set up Nginx as a caching reverse proxy?“

I looked doing this with Varnish but we are already running nginx and it was dead easy to set this up.

Google analytics API

We started using the Google Analytics API to create statistics tables for some of our data so that search results and other features can use the pageview information to better rank things, detect “hot” pages, stuff like that.

The API is great. It’s a little slow and a little strange, but it gives you access to all of the data that you can get through the web interface. Thanks Google, for processing gigantic amounts of traffic data for me so that I can query it later.

Adding a full featured API to the site has been a really good exercise. It’s really helped me to improve code quality and tie up loose ends and I’m glad that the iPhone has finally sparked the creation of an API for Ravelry.

I was surprised to find that the software support for API needs is lacking in places. Here are some of the missing or ill-fitting pieces that I’ve run into so far:

Routes, controllers and actions

Rails gives us formats and :respond_to. Rails 3 makes it even better…. but I don’t like it. I don’t like mixing my API actions up with my non-API actions because the logic is never the same except for the simplest methods. For now, I’m just manually adding routes that send my API (JSON) requests to X_with_api actions. Hopefully some sort of pattern will emerge and an idea will sprout… or maybe I’ll see the error of my ways.

Documentation

I couldn’t find any way to generate pretty API documentation from simple structured text or code comments. There are so many people creating and documenting HTTP APIs that I was a little surprised by this. For now, I made up a code commenting convention and borrowed heavily from YARD. I could try to figure out a way to get YARD to parse the stuff but I suspect I’ll end up hacking up a simpler parser that generates HTML documentation.

I like that this combined with the above results in methods dedicated to each API call with code comments that describe the HTTP parameters and JSON output. Add in a little setup code and some sample parameters and I could probably create tests from this.

Rendering the Results

For now, I’m only providing JSON. When I first started out, I was wishing for some XML::Builder/rxml like thing that I could use to write templates that generated JSON. My wish has since been granted by the Tequila RailsRumble project but I’m glad that I ended up going in a different direction.

I’m using my own take on the serialize_with_options plugin. This allows me to set up multiple serialization “styles” (for example, I might have “compact” and “normal” - for search results vs retrieving a full object) and declare what attributes, method return values, and associated objects to include. I’m liking this approach so far: it means that all of my API data has to be in a model as a method or attribute which will be easier to maintain. It also results in a little DSL that I should be able to use to generate API documentation that describes the return values.

That’s what I’ve got so far. It’s my first time building such a full-featured API and it feels a little like I’m inventing things that I shouldn’t have to invent. At least what I’ve got so far seems like it will be easy for me to maintain. I’m just getting started, so who knows where I’ll be by the end.

PS Ravelers - I know that you are probably excited to hear about the iPhone thing Check out this thread for more information: http://www.ravelry.com/discuss/for-the-love-of-ravelry/601827/76-100#82

It’s obligatory to be cranky about anything that Jeff Atwood and Joel Spolsky put out into the world together. I got over my initial grumpiness and discovered that StackOverflow is actually pretty cool. I’m casey #18424 over there.

SO is a question and answer site for programmers. People post questions, others answer, onlookers vote answers (and comments on answers) up and down. The question pages are well done and easy to read and digest.

I only wish that it were a little more… social and personalized. Seriously. This is not something that I usually wish. I’d like to use the site for learning and helping others but right now it seems (mostly) suited for asking questions and helping others if you are either really dedicated or lucky enough to stumble over questions that you know the answer to.

Here are some things that I’d love to see. I’d post these in meta.stackoverflow but I’m too lazy to see if they’ve been suggested already and I’m a little scared of the community over there. Ironic, eh? For the Love of Ravelry is similar. Participating on a totally different site is definitely helping me see things that can be improved on Ravelry when it comes to community and participation.

My wishes. It’s very possible that I’m either doing it wrong, missing something that exists, or just missing the point, but here they are:

• Friending or following - some way to watch questions and answers from people who consistently post things that seem interesting or educational (to me). I know that I can do it by subscribing to tons of RSS feeds but…
• Something other than tags for organization. Using piles of tags with no relationship or hierarchy makes browsing the piles of questions sort of painful.
• Some sort of recommendation or matching tools that shows me questions that might interest me. Maybe based on my votes or something. “Interesting tags” isn’t working very well because of high volume and huge variation in questions tagged with things like “ruby” and “javascript”.

I guess these all address the same problem - discovering questions that interest me is hard.

If you code for work or for fun and you haven’t seen it already, check it out… and if you are looking for the answer to a coding question, toss “site:stackoverflow.com” on the the end of your Google search before you try anything else.

Campfire and Propane - team chat

What would we do without team chat? Campfire is the web-based group chat that keeps our team connected throughout the day. With chat, we don’t feel like we are working alone, thousands of miles apart.

There are many web-based chat services but we like Campfire. It works well for our needs and there are some great ways to extend it. Propane is an excellent tabbed native chat client for Macs that we use when we are at our own computers. Ember is a Campfire client for iPhone… and we integrate Campfire with the rest of Ravelry through a Ruby API (Tinder) and built-in connections from our help desk / email support software.

Communicating without distracting

Unlike IM, having a team chat gives us a way to have the a communications channel that doesn’t always demand immediate attention. We can leave notes for others to see when they have a chance, jump into a full on chat or meeting when we need to, and page one another visually by taking advantage of the Growl notifications that Propane offers.

Email bad

The last thing that we need is more email. Chat has given us a way to share information and discuss things without adding to our already-overwhelming email load.

Integrating notifications

Ravelbot, our automated chat robot, pops in to let us know about things that have occurred in Ravelry or in our customer service email that we may want to know about. We don’t want to have to check screens of important information or run reports but we also don’t want to deal with email notifications.

By using the different APIs that Campfire offers, we can connect our own software to chat so that we know about important events as they happen.

Skitch - easy screen capture sharing

We love Skitch so much. A screen capture application doesn’t seem like a big deal but there are a few features that make it indispensable to us:

Upload your capture in one click. We share screenshots of things in chat so much that it really is important that it is easy. Talking about things is so much faster when you immediately share a image from the site, someone else’s site, email, twitter, whatever.

Upload to your own host. Since Skitch can upload any image (not just a screen capture) in one click, we save a lot of time by using it to store images that we post on the site, in forum posts, in blog entries, and in other places. When we do these things, we definitely want to store the images on our own server for more permanence and less bandwidth concerns. Skitch can upload to any host via SFTP, FTP, or WebDAV.

Easy annotating and drawing. Once you’ve got an image, it’s really easy to add arrows, text, draw, do whatever you want to do before you share with someone else.

Our Campfire chats are peppered with screenshots and annotated images and it really makes communicating a lot easier and a little more fun.

Dropbox - file sharing that feels local

Shared folders - sometimes you need them. Dropbox is the best file-sharing application that I’ve ever used. The client works on Windows, Mac, Linux and provides you with a real shared folder on your machine that looks and acts like a normal local folder. The web site works well when you aren’t at your own computer. There are RSS feeds and Growl notifications for updates. There is a revision history in case you need to recover previous versions of a file…

Most importantly, the syncing in the background is transparent and seamless. A copy of your files are stored on each machine that is configured to use Dropbox and the application takes care of the syncing for you in the background. If you aren’t connected to the Internet, you still have your files, and if you make changes they are synchronized when you next connect.

Email - Helpspot

Gmail is one good way of easily sharing general email boxes and linking to individual messages if you want to talk about them. We started out using shared IMAP accounts (regular email accounts) because all of us have mail clients that we prefer to Gmail - Pine for me, Apple Mail for Jess and MH. As we’ve grown, it has become harder to manage the volume of email and harder to keep track what is happening with all of the individual users and other people that we communicate with.

We now use Helpspot for most of our general mailboxes. Helpspot is a web-based email app that is aimed at customer support. Helpspot helps us work together in a few ways:

• We can share links to messages when we need to talk about them
• Much like Gmail conversations, we can see history so that we can easily pick up on an existing conversation
• I’ve integrated the Helpspot email with Ravelry itself so we can all see email histories in relevant places - this helps because a previous conversation that someone else had with a person may be important and this way we can know what is going on without asking (or discovering later)
• Helpspot has a Campfire API that we use to automatically push certain messages to chat so that we can all see and talk about certain types of issues/questions

…and I ran out of steam. I’ll just post this now before I relegate it to the draft pile. I suppose I don’t really need to talk about Google Docs or Skype anyway Hopefully someone will find these things useful or interesting.

Is there anything that your office can’t live without?

edit One little thing - we’ve been using oovoo for 3-way video chats. iChat seemed too bandwidth intensive and Skype only does 2-way video. oovoo has an ugly client but the service works pretty well.

• Ruby GC Tuning - handy GC tuning info with performance notes from Evan W. at Twitter. I adjusted part of Rav and got big response time improvements and CPU usage improvements that were in line with the numbers that Evan posted. I posted a little graph in his comments.
• Tokyo Cabinet and Tokyo Tyrant - a super fast key/value store and memcached-speaking server component (respectively). This came out of Japanese social network mixi.jp. Tokyo is file-backed but is nearly as fast as memcached for most needs. We’ve been using this for the last couple weeks to store long-lived cache data that is a little big for memcached. See also this presentation on Scribd and Plurk’s built-on-Tokyo LightCloud.
• Facebook’s growing infrastructure spend - really interesting post from Niall Kennedy. The numbers showing Facebook users vs % of people with internet access per country are staggering.
• Kindle 2 stuff - Jess recently got a Kindle and she is addicted to it. Some neat tricks: change the screen saver, build in conversion of PDFs and epubs
• For all Mac users who are interested in Ruby - check out the 2009 Rubyist’s guide to a Mac OS X development environment from the robots over at Thoughtbot.
• Sly - I haven’t really been following the Javascript selector speed wars lately but this thing looks crazy fast (well, circa April 2009 crazy)
• Speaking of crazy fast - the new Intel Xeon 5500 Nehalem processors look pretty awesome. Faster, lower power consumption, DDR3 memory with a faster bus, Power Boost (appears to be an auto overdrive/overclock?), Hyperthreading is back. This is a real upgrade and not just more cores and more GHz. I can’t turn up any nicely done benchmarks right now but here is an overview, PDF data sheet is here, and the Silicon Mechanics configurator will give you some idea of how much these things cost.
• OmniGraphSketcher - finally, nice Mac OS software that you can use to turn raw data into nice looking graphs. I’ve only tried it on simple things so far.
• Realtime Twitter Search Results for Google - this Greasemonkey extension inserts Twitter search results into the top of your Google searches. Install it immediately. The realtime aspect changes everything - having this information inserted into my normal googling has been immensely useful and interesting.
• Not purely technical - but kind of amazing. Individual musical performances in YouTube videos layered to create new audio & visual compositions. I’d call them something simpler like “mash ups” if I could bring myself to use that phrase THRU YOU | Kutiman mixes YouTube.

That’s it for this month!

Last March, I wrote a tiny post that listed a few of the pieces that run Ravelry along with my opinion of each. Here is a update to that list. The biggest change? I’ve been running Phusion Passenger + Apache as application servers for the last several months.

As of March 2009, Ravelry runs on….

• Ruby on Rails 2.2: Still love it.
• Phusion Passenger: Replaced Thin/Mongrel. Passenger plus Ruby Enterprise Edition uses less memory, avoids memory leaks, and it spawns and retires Ruby processes as needed. I also like that you can get a fair amount of status and debugging information if you need it. It looks like Passenger is the new standard for Ruby deployment. The only downside? Hot deployments of new versions of the app were causing lots of trouble for me (Google Groups thread here). I ended up putting together a Capistrano script that removes each Passenger from load balancer, updates and stops/starts Apache once it is cold, puts it back into the loop, and moves on the the next app server.
• nginx: Still rocks. fast, memory efficient, stable, trouble free. We’re running two now.
• haproxy: Still rocks. I’m using it to load balance HTTP requests across Apaches and provide failover for search and another service.
• Sphinx: Still love it. If your site has any search needs, you’ve got to try it.
• beanstalkd: beanstalkd + a few simple Rails runner script daemons are my new solution for running background jobs and processing file uploads. This is a huge improvement over backgroundrb: much simpler, much more stable. I’ve never had a problem with beanstalkd. Background jobs used to be a very brittle part of our architecture and now I rarely think about them.
• Percona MySQL 5.0.x build: MySQL 5 has been rock solid for me. MySQL 5.1 was not. I’ve been running MySQL 5.1 on my slaves and I’ve found that they do a better job of picking up replication after a restart or other problem. Although I’d love the fast index creation feature that the InnoDB plugin provides, I have no plans to upgrade the main database to 5.1. However, I would like to bump up to one of the newer OurDelta MySQL 5 builds.
• Other stuff: I’m still very happy with Nagios for monitoring, Munin for perfomance graphs, Postfix for mail, Xen for virtualization, and Gentoo as my Linux of choice.

Updated to add a little diagram that shows how the web/application tier fits together.

Look out - this post is going to be a bit dry.

Over the Thanksgiving weekend, I upgraded Ravelry from a crusty old nearly-2.0 version of Edge Rails to the latest 2.2.2. Other people’s upgrade war stories were really helpful so I figured that I’d post my own. (See also gusg.us’s 2.2 story, assaydepot’s 2.1 story)

Here are the things that had to be fixed or tweaked:

environment.rb

I moved a bunch of stuff out of here into the config/initializers directory. Much neater. I also config.gem‘d my gems.

cache_fu plugin

Broken. Upgrading to the latest source from github made it all better.

memcached gem

I use Evan Weaver’s juiced up memcached client. Upgrading to the latest source from github made it all better.

xss_terminate plugin

I’m using xss_terminate for sanitizing input and I needed to add a little patch to an initializer to make the plugin work properly: http://code.google.com/p/xssterminate/issues/detail?id=3#c0

exception_notification plugin

Wasn’t working right, but I’m using HopToad now so I just trashed it.

acts_as_versioned

I read that weird stuff would happen with older acts_as_versioned and the new dirty attributes/partial updates. Upgraded to the latest source. Unfortunately, the newer version seems to handle version columns differently. I was previously using “acts_as_versioned :version_column => ‘lock_version’” to base the “version” column in the versioned table off of the model’s “lock_version”. Now it wants both to be called “lock_version”. I just patched it for now.

RFPDF plugin

Broken. Changed init.rb from ActionView::Base::register_template_handler ‘rfpdf’, RFPDF::View to ActionView::Template::register_template_handler ‘rfpdf’, RFPDF::View

rails_asset_id in ActionView::Helpers::AssetTagHelper

Gone. Reworked some helper code - I didn’t need it anyway.

attachment_fu

Broken by ActiveRecord changes. Here is the fix: http://ar-code.lighthouseapp.com/projects/35/tickets/25-edge-callback-overrides-inconsistently-pass-arguments

active_form

More small breakage. Here is the fix: http://github.com/valda/active_form…

has_many_polymorphs plugin

Oh god - has_many_polymorphs was a nightmare (as expected). I upgraded to the latest and then had to fix broken stuff for a while. I don’t know what version I was using before but stuff changed. ack.

prototype.js

Sadly, I’m still using a pre 1.6 version of Prototype and now was not the time to upgrade that as well I had to implement Element.Insert in Javascript because the RJS stuff was changed to use that instead of Insertion (which was deprecated) You can see the change that I’m referring to here: http://github.com/rails/rails/commit…

That’s it! Also, HopToad was a great help in finding a few small bugs and glitches once I pushed the upgraded version out to production.

Now what?

I’m going to go back to fun Ravelry-improving work for a while. Still, I’d like to move to Git. …and maybe take the latest JRuby out for a test-drive. The total amount of memory that our Mongrels (Thins) consume is pretty embarrassing and things are heating up in the JRuby world with faster releases, a more threadsafe Rails, and so on…

Toolbox photo from Flickr/mamabarns .

Tracking down Ruby/Rails memory leaks

Memory leaks are no fun. We sprouted a big leak about a month ago: Ruby VMs would get all bloated, often hitting 300-400 MB and sometimes consuming even more ridiculous amounts of memory.

I run about 20 instances on each virtual server and a bunch of misbehaving instances could quickly eat up all of the memory. Because restarting my mongrels (Thins, I mean) when they consume too much memory is pretty painless, I put off fixing it for a while.

This week, I tried using bleak_house to figure out what sorts of objects were responsible for my VM bloat. I found nothing.

Then I learned something really important about Ruby’s garbage collection and heap maintenance. I don’t know how I missed out on learning this earlier. I’m going to quote a bit from hacking on ruby’s garbage collector at lloydforge.org since he explains it well.

• First important thing: “When ruby runs out of heap space, it first does a GC run to try to free something up, and then allocates a new heap. the new heap is 1.8 times larger than the last.”
• Second important thing: “Because of the way ruby works, objects may _never_ be moved around in heaps. That means from the time they’re allocated to the time they’re freed they may not be moved to a new memory address”

You see where I’m going with this? Force ruby to build bigger and bigger heaps and you’ll probably never get that memory back. A single non-garbage object sitting on one of this heaps will keep it alive prevent you from getting that memory back. There is no heap compaction.

I read about some of the work that the Phusion guys had done on their ruby fork called “Ruby Enterprise Edition” and noticed that they had added a handy debugging tool: ObjectSpace.statistics.

I installed Ruby EE, added some ObjectSpace.statistics logging to each request and ran it on one server for a half-day. Once I had my results, it only took a few minutes to spot the problem requests and fix them. Phluid’s Ruby EE prints out the number of objects, size of objects, number of heaps, and size of heaps so you can spot when a request is causing the creation of a new (large) heap. The culprit was just a stupid coding error in a rarely-hit action that loaded way too much data via Activerecord. The heap would balloon to a gigantic size and although the objects loaded would all become garbage, something prevented the memory from being given back.

This was a heap-related problem but I think that these ObjectSpace statistics would be equally valuable for finding the reference-related leaks. You probably wouldn’t be able to eyeball the data like I did but you could add up the growth in # of objects and bytes across a large number of requests and come up with a few suspicious actions. The really nice thing is that I was able to set up this logging in production without impacting the site in any way.

Tracking down MySQL performance problems

If you are running MySQL, you should consider using one of the Percona patched versions: http://www.mysqlperformanceblog.com/2008/07/16/mysql-releases-with-percona-patches/

The Google SHOW INDEX_STATISTICS patch is great - you can use it as part of your index tuning and pruning because it will show you which indexes are being hit.

The microslow patch turned out to be an extremely valuable tool for finding problem queries. We have far too much query traffic to log all of it and MySQLs “slow query” log only works as whole second precision. In our case, we had some slow queries that were beating up the IO subsystem and they weren’t easy to spot until brought up the patched MySQL and set the threshold to 300 milliseconds (instead of 1 second).

I ended up breaking up a very large table into an InnoDB table with all of the important information and indexes and a MyISAM table containing the blobs (text columns). I changed a few lines of code on the application side to make things work with the new split table, and voila:

okay. I’ve run out of steam…

In other news - we hit a milestone of sorts by nearly making the Alexa 10,000 this month. We’re at 10,032

As an aside - I’m looking into doing away with the beta label and having beta testing being an opt-in thing for users who want to be guinea pigs. We’re approaching 150,000 registered users and not everyone realizes that they often part of an experiment - this ends up creating more work for us in the form of questions (and sometimes, complaints)

Our users have been incredible - they report bugs of all kinds, come up with tons of great ideas and suggestions, enter things into our issue tracking system, and comment on existing issue. Here is how we do it:

Collecting feedback

Some sites collect feedback from beta testers with an email form. Don’t do this. All of your users need to be able to see and read the bug reports and ideas that everyone else is sending in. Working as a group is more efficient (less duplication), more productive (because small what-ifs can turn into great ideas through conversation) and more fun.

We set up one board in our forums that is specifically for talking about Ravelry itself. The board isn’t limited to feature requests and bug reports - it is a place for any kind of Ravelry talk like “Who has the biggest stash on Ravelry?”. In the last 30 days, 2300 people have posted on this board and 18000 people have read this board. hm… I guess I have to come up with some ideas to drum up participation - people who aren’t reading are missing out on their chance to shape the site.

Keeping Track

Our Ravelry discussion board contains 97,000 posts. We can’t use the board itself to help us track, categorize, and update all of the bug reports and suggestions. Even if we tried to do it by keeping the threads themselves excessively organized and moderated, we’d be sure to lose things.

So… we have an issue tracking system (The List) built in to Ravelry. Jess, Mary-Heather, myself, and 30ish other Ravelry users called “trackers” have the ability to turn posts in the forums into new issues in our system. When people post bug reports or suggestions we all try to either add them to The List or connect them up to existing entries. You can see that the post below has 1 agree vote and 1 disagree - sometimes the agree/disagree votes that are part of our forums can be helpful when we are looking at suggestions.

Now what?

When I am not actively working on them, the list items serve several purposes:

• Users can search The List to see if their suggestion or bug report has already been brought up
• Trackers can connect future forum posts to list items in case people have more or different information to share. This happens a lot and it is handy to have slightly different bug reports attached to bugs and further thoughts attached to suggestions.
• All users can comment on items and I read these comments when I set out to work on something.

Although I do have some basic prioritization type features to The List itself, I do all of my work-gathering and organizing outside of Ravelry. I tend to go into the list, gather a set of items that fit with what I am working on, and take those back to my virtual work area so that I can sort through them, read comments, and organize.

I’m really happy with this. I enjoy talking with Ravelers about the site and getting ideas and help from them. I’m also very glad that I don’t have to worry as much about losing good ideas and bug reports in a sea of posts. Ravelry Users: You can find the For the Love of Ravelry (FTLoR) board and a link to The List on the forums tab. As always - if you have ideas on how we can improve tracking and The List, we’d love to hear them. Just make sure to post them in FTLoR so that we can track them

MySQL 5.1 rc

I’m still having a problem with MySQL 5.1 hanging. I removed the innodb plugin as an experiment and now I’m pretty sure that it is a bug in MySQL. Not only is it suspicious that the same workload never caused deadlocks in 5.0, I’m pretty sure that either innodb’s lock detection or the fallback lock_wait_timeout should be kicking in.

I’ll probably downgrade to 5.0 soon. The before/after performance graphs will be interesting.

Reading

This week I read Clay Shirky’s “Here Comes Everybody” and Josh Porter’s “Designing for the Social Web“. Shirky’s book was great - try out some of his writings on media and community if you want a (free) taste. Porter’s book was more of an ego booster - sort of a catalog of lessons that we’ve already learned. If you are working on a web site that will have human users, do pick it up. The book is concise and well organized and you’ll get *something* out of it (I did).

Bookmarked this week

• Presentation Zen - Jess, MH, and I are going to be giving a 1 hour presentation at Knitter’s Connection in June. We don’t want it to suck, so I’ve been digging around for ideas.
• Google is hosting your favorite Javascript framework for free. Speaking of free stuff from Google - I love the Chart API.
• If you use last.fm, visualize the ebb and flow of artists with Lastgraph.
• Rails stuff: Akita’s 2.1 rundown, better_partials, acts_as_reportable, using git submodules to track plugins (I haven’t made the jump to git yet but I’ve been playing with it and it is tempting).