geek!daily

SPEC: Rails App Autostaging

2017-01-01T00:00:00-08:00

SPEC: Rails App Autostaging

Goal

Be able to push-button deploy an app to a cloud-based staging environment that may or may not already exist in a predictable fashion.

How I’ll do it

Set up a cloud-based staging environment and take notes on what I have to change beyond the basic provisioned OS (OpenSUSE Leap 42.1)
Make cap tasks to deploy the app to the existing staging environment
Make cap tasks to verify the status of the staging environment (not running, running but needs cold deploy, warm deploy)
Automate staging setup in a cloud-init script
Make cap tasks to create a new staging environment

I’ll know it’s done when

I can type cap staging deploy with no staging environment and see the staging environment be created and the app be deployed
I can type cap staging deploy again and see a warm deploy to the existing environment
I can destroy the staging environment, then type cap staging create_environment and see the new environment be created
I can type cap staging deploy and deploy to the fresh environment

I’ll need to verify

Dev environment is up to date
App is healthy
- All specs run clean
- All automated tests run successfully
Cloud account is healthy and can allocate resources via API/CLI

I’ll need to learn or relearn

EXPERIMENT: Spec'g My Projects

2016-12-31T00:00:00-08:00

If you’re at all like me, you’ve got upward of a few gazillion ideas for projects bouncing around in your notebooks, task management tools, and/or head, and you never get enough time to bring them all to a successful conclusion. I hope you’re not like me in the sense of having many of them half-started and silently nagging you to finish every time you wander into your garage, office, or ~/work directory.

For me, I’ve noticed that a lot of this ultimately goes down to losing track of what I came do. It usually goes something like:

“I’ve got a day free. It’s time to add that feature to the Foo project.”
Run the specs and get 971 failures, clustered around 12 core error messages for 3 different gems.
“Oh, right, I upgraded [something] … well, let’s get everything working again. It’s the right thing to do.”
Three days of sweaty debugging, learning, and relearning pass. This includes learning how to tune my shell prompt, tweak Rakefiles (like the one for my blog to make a new post), and do a lot of other non-essential things.
“Well, the specs run clean again and I learned a lot. Guess I’ll add that feature next month.”

Don’t get me wrong; I like the learning, and all the things I end up doing improve some aspect of my life, but I don’t like the unpredictability, the time that I sync into recovery (and not my family, or other projects like BBQ … mmm, BBQ …) and inevitable letdown of not getting the one thing done that I came to do.

I’m going to try a different approach. I’m going to try writing a very short, very lightweight spec for projects that I intend to do, now or sometime in the future. The format will be something like:

	PROJECT NAME

	Goal: A single sentence describing the intent.


	How I'll do it:

	- A short list of high-level steps
	- Each something I can measure the end of
	- And each of which can provide me focus to finish it


	I'll know it's done when:

	- A short list of acceptance criteria
	- Each being something I can demonstate, preferably automatically
	- And bonus points if I can automate the acceptance tests quickly


	I'll need to verify:

	- Things I believe to be true
	- That I need to have working before I can get started
	- So I don't waste time hoping that it all works out


	I'll need to learn or relearn:

	- Tech I don't already know
	- Tech I've used before but has changed a fair bit since last I dived in
	- Other skills I'll need.

I’ll publish some here, under the SPECS nav above. We’ll see how it goes.

MacOS Sierra, Homebrew, and rbenv

2016-11-23T07:20:00-08:00

I recently updated to MacOS Sierra, aka 10.12, and got to relearn my dev environment when I wanted to do a little work in Ruby and ran face first into

> irb
Bond Error: Completion file '/usr/local/var/rbenv/versions/2.2.5/lib/ruby/gems/2.2.0/gems/bond-0.5.1/lib/bond/completion.rb' failed to load with:
dlopen(/usr/local/var/rbenv/versions/2.2.5/lib/ruby/2.2.0/x86_64-darwin15/readline.bundle, 9): Library not loaded: /usr/local/opt/readline/lib/libreadline.6.dylib
  Referenced from: /usr/local/var/rbenv/versions/2.2.5/lib/ruby/2.2.0/x86_64-darwin15/readline.bundle

How rude! It took me a couple of minutes to even remember that I’d updated to Sierra (it’s been a week; I’ve slept since then). Once that synapse fired, I knew it meant that all the versions of Ruby I’d installed with rbenv were built against the older readline, so it was time to rebuild and reinstall them all, as well as all of their gems with native extensions. Ugh.

Luckily, that’s not so horribly hard once you sort things out, so here’s the short course on how to deal with this:

> xcode-select --install
[... popup dialog launches and you do the install dance ...]
> brew update && brew upgrade
[... many package defs update and upgrade ... ]
> rbenv versions
  2.1.5
  2.2.0
  2.2.1
  2.2.3
  2.2.5
* 2.3.1 (set by /usr/local/var/rbenv/version)
> rbenv install -f 2.3.1 && RBENV_VERSION=2.3.1 gem pristine --all
[... much building of ruby and gem native extensions ...]

From there, it’s repeat for every version of Ruby that you care about. It’s also a good time to remove any you don’t need anymore.

If this was useful, an upvote on my StackOverflow answer for this would be a kindness.

How to get started with a Minecraft Server, Redux

2015-09-06T06:00:00-07:00

Back when, I started a series on setting up a Minecraft server for kids which I never quite finished. As I mentioned in a post yesterday, it’s time to do a major update on our server, so I’ve got a chance to update and complete that series. And here we are.

Choices are better now

Paid Minecraft Hosting

One Google search for Minecraft Hosting shows a lot more viable alternatives than existed two years ago. There were easily a few dozen hosting options available, each clearly targeting Minecraft server hosting. All of them seemed to be ~$12-25 for a 2GB RAM server, which is pretty close to the going rate for a 2GB VM in any of several public clouds.

Containers!

Another excellent alternative is some solid Minecraft Docker images for running Minecraft. Very configurable, and gives you good control over what’s going on without needing to go deep into the dependencies of Java, etc.

Self-Hosting: Playbooks and Recipes and Shell Scripts, Oh My!

As was true two years ago, there are a host of ways to set up your own server Chef, Puppet, Ansible, and shell scripts. However, several have emerged as more popular and have thus matured a bit since then.

It’s Always Build vs. Buy

In the end, I’m making a tradeoff in how much we want to be able to customize the server; how much time I need to spend setting it up; how much time I need to spend administering it; and net dollar costs to set up and run. A fast sketch of this looks like:

Option	Customization?	Setup Time?	Admin Time?	Dollar Cost?
Paid Hosting	Low-High	Low-Med	Low	~$6-20/month + ~$15 setup fees
Containers	Med-High	Med-High	Med	$10-20/month for container hosting
Self Hosting	High	High	Low-Med	$10-20/month for instance hosting

The dollar costs of hosting are so close in all cases as to be a negligible motivator, so it really comes down to how much time I want to put in. Two years ago, I wanted to do this myself to learn how a server works and experiment with modding; now it’s much more about having a stable server with the right functionality (usually via plugins) and modding is something Sam and I do on laptops and an occasional cloud VM or container. So this time around, we’re going to look at paid Minecraft hosting first, then containers, then self-hosting.

What we need

I mentioned in my previous post the bare minimums; we’ll add a few more at the end as we’re looking at migrating to paid hosting:

Multiple worlds: the kids have invested a lot of time in the three worlds we have, so that’s got to continue. We’re currently using the Multiverse Bukkit Plugins.
Permissions management: I don’t want to op kids for the whole server, but I don’t mind letting them rule a world. We’re currently using the PermissionsBukkit plugin.
Different settings across worlds: some of the worlds are creative, some are survival. Various rules apply in each of them. We need to keep that true. This is inherent in the Multiverse
Chat across worlds: the kids frequently chat with each other while in different worlds, often calling for each other to come see and try stuff. That’s where the fun is.
Grief prevention and bad word management, because kids. We use the GriefPrevention and BadWords plugins at the recommendation of cheracc(http://www.sandlotminecraft.com/index.php?pages/about/#owner), who runs Sandlot Minecraft server for kids and families(http://www.sandlotminecraft.com/). He puts a lot into running Sandlot(http://www.sandlotminecraft.com/index.php?pages/about/) and it’s an excellent place if you just want a family-friendly server to let your kids play on.

Since we’re migrating, we’ll add:

Ability to import existing worlds: We have existing worlds that the kids have put a lot of time into.
Ability to back-up on a daily basis: It helps for moments where one of the worlds that’s not under GriefPrevention has accidental damage. Also, we need to be able to pull a daily backup from outside the service in case the service gets hosed.
Good tools for server administration: We like to divide up permissions so we can make kids into minor ops, and we often need user logs to see who was online when something happened to address behavior issues, etc., because kids.
Cost: way down at the bottom because, as I said, it’s not the most important factor.

Hosting Options

I’m going to look at three main options here:

Creeperhost.net, BeastNode, and BisectHosting all seem like reasonable alternatives, but landed in my second choice list for various reasons. Creeperhost is UK-based, though with US locations, and didn’t seem as oriented to opinionated Minecraft/Spigot hosting; neither did BisectHosting. BeastNode looks promising, and I wanted to limit the field to evaluate.

So with that in mind, I set out to set up worlds with each of the top three. I used the 2GB RAM server as a basis for comparison as that’s about what runs well for long periods with the crowd on our server.

Here’s what I found.

Minecraft Realms

Multiple Worlds: yes, limited to three worlds, but only running one at a time.
Permissions Management: no, only op or normal player.
Different World Settings: yes
Chat Across Worlds: no, only one world running at a time.
Grief Prevention: no
Foul Language Management: no
Import Worlds: no
Daily backups: yes
Automatically copy backups externally: no
Good server admin tools: 5/10
Cost: $13/month, discounted if you either enable auto-renewal or buy 3 or 6 months in advance.

Best if you want to run “vanilla” Minecraft and always have the latest improvements. It’s pretty easy to get started; you pay on Mojang() then set up your new Realm through your Minecraft app. Maybe I’ll write a post with a quick step-by-step, but in the meantime Wikihow has a pretty good one.

I was pleasantly surprised to see you could have three worlds (plus a selections of “mini-games”); however, switching worlds seems to require a restart of the server for your Realm so you really only get access to your worlds one at a time. Interesting a bit, but that’s a deal breaker for the server we run.

The server admin tools are okay, but not great. You can invite players, choose whether or not to op them, and see when they’ve been online (but not if they’re online now). You get automatic backups and can “download” them manually, which makes them one of your single-player worlds on your local machine.

Not the right long-term choice for us.

MC Pro Hosting

Multiple Worlds: yes, with Multiverse plugin.
Permissions Management: yes, with PermissionsBukkit plugin
Different World Settings: yes, with Multiverse plugin.
Chat Across Worlds: yes
Grief Prevention: yes, many plugins available
Foul Language Management: yes, many plugins available
Import Worlds: yes, via FTP
Daily backups: yes, set up as a repeating task. 1 GB quota for backup storage; 10 GB quota for $2/month
Automatically copy backups externally: yes, backup via FTP as often as you like.
Good server admin tools: 8/10
Cost: $20/month, $15 plugin setup fee if you need help, discounted if you either enable auto-renewal or buy 3 or 6 months in advance.

MC Pro’s signup experience is top-notch. They walk you through eight steps to get a sense of what you’re looking for; helping you size your server; locate it where you want it in the world (they’ve got a fair number of choices, too); choose the add-on services you want, including mod packs, premium support, etc.; and even getting the name, hostname, IP and/or port set just right.

They use Multicraft control panel, which they’ve integrated very well into their system, right down to authentication and styling. Multicraft includes an FTP server, so you can easily download backups and copy up configuration, etc. They also have good docs to help you get started if you’re importing worlds or installing custom plugins.

It took me a short while to merge config data from our server with the defaults at MC Pro, but fairly quickly I had a mostly working server, lacking only the core worlds. I filed a support ticket to help me solve that, then figured it out myself about 6 hours later and closed the ticket, which had gotten no response.

The only complaints I have would be that they don’t take American Express (which is the card I use for online transactions) and don’t specifically say so; I had to figure it out when their CC entry field couldn’t handle the 4-digit CVV code. Also, the discount code from their Facebook page didn’t work, despite being roughly three weeks old. Minor bummer.

ggservers.net

Multiple Worlds: yes, with Multiverse plugin.
Permissions Management: yes, with PermissionsBukkit plugin
Different World Settings: yes, with Multiverse plugin.
Chat Across Worlds: yes
Grief Prevention: yes, many plugins available
Foul Language Management: yes, many plugins available
Import Worlds: yes, via FTP
Daily backups:
Automatically copy backups externally:
Good server admin tools: 7/10
Cost: $12/month, $15 plugin setup fee if you need help

Right off, I was inclined to like GG Servers as they handle Amex properly and the discount code from their Facebook page worked. The signup was fairly easy for someone who’s experienced and knows how to size their own server; it was much less of a concierge experience when compared to MC Pro Hosting.

GG Servers also use the Multicraft control panel, but it’s not as well integrated into their client control panel so there are bumpy bits. This turns out to be the source of several issues as I was looking to get set up. They’d sent me an email with a separate password for the Multicraft CP; that, of course, ended up in the spam folder so I was blissfully unaware of it.

For one, I wasn’t able to use FTP service right away, so I put in a support ticket. That was relatively straightforward and within a couple of hours I had a response. From there, it was a fairly typical support interaction, by which I mean the very busy support person is working through tickets as fast as they can and thus asks me to do something I’ve already done (and partly, poorly described doing in fairly specific terms). We iterated along those lines a couple of times before the support tech realized what I’d said and pointed out I needed the password from the email they’d sent.

To be clear, their support was very responsive — most replies came within 10 minutes once they picked up the ticket, and on a holiday weekend, even. They were never rude. They ended up solving my problem. Yet it was frustrating to re-explain what I’d explained at the beginning, and twice, to get to the right answer.

Decision: MC Pro by a smidge

I have no doubt our server would run fairly equally well on either MC Pro or GG Servers. In the end, I’d already resolved all of my issues and had a working, identical server running on MC Pro by the time we reached the end of the GG Servers support ticket, and that was the deciding factor. As I mentioned above, this is more about how I save my time than anything else.

GG Servers is high on my list should I need to re-examine Minecraft hosting providers again, but for now we’re happily up and running on MC Pro.

Happy crafting!

Compiling Spigot Minecraft Server for CentOS 7

2015-09-05T09:00:00-07:00

A quick cheat sheet for those who don’t want to read the entire post with all the instructions:

Make a CentOS 7 server, then execute these three commands in it:

$ sudo yum -y install git java-1.7.0-openjdk-devel
# TONS of output and dependent packages install

$ wget https://hub.spigotmc.org/jenkins/job/BuildTools/lastSuccessfulBuild/artifact/target/BuildTools.jar
--2015-09-05 18:42:24--  https://hub.spigotmc.org/jenkins/job/BuildTools/lastSuccessfulBuild/artifact/target/BuildTools.jar
Resolving hub.spigotmc.org (hub.spigotmc.org)... 198.41.205.94, 198.41.204.94, 2400:cb00:2048:1::c629:cd5e, ...
Connecting to hub.spigotmc.org (hub.spigotmc.org)|198.41.205.94|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3411286 (3.3M) [application/java-archive]
Saving to: ‘BuildTools.jar’

100%[======================================>] 3,411,286   2.65MB/s   in 1.2s

Last-modified header invalid -- time-stamp ignored.
2015-09-05 18:42:27 (2.65 MB/s) - ‘BuildTools.jar’ saved [3411286/3411286]

$ java -jar BuildTools.jar
# TONS of output
Success! Everything compiled successfully. Copying final .jar files now.
Copying craftbukkit-1.8.8-R0.1-SNAPSHOT.jar to /root/.
  - Saved as craftbukkit-1.8.8.jar
Copying spigot-1.8.8-R0.1-SNAPSHOT.jar to /root/.
  - Saved as spigot-1.8.8.jar

Now you’ve got the latest jars for spigot and craftbukkit. Enjoy!

RIP Craftbukkit, Moving On

2015-09-05T08:00:00-07:00

The Death of CraftBukkit

Craftbukkit kinda died a year ago in a big dramatic flare-up. That sucked. We were running a server based on it, and continued running the last public version, 1.7.9-R0.2 while hoping things might resolve well. They didn’t.

It’s time to call it: CraftBukkit, and the underlying Bukkit server, are well and truly dead. It’s time to migrate the server. There are other pressures: I want to change hosting providers; make the server creation, backup, and monitoring more robust; and make it easier to admin.

Moving On …

In good news, it allows me to finish the blog series I started back when, and update it for smarter things. I’m using some time this weekend to look into what’s best, so here’s what I’ve learned.

Requirements

Multiple worlds: the kids have invested a lot of time in the three worlds we have, so that’s got to continue.
Permissions management: I don’t want to op kids for the whole server, but I don’t mind letting them rule a world.
Different settings across worlds: some of the worlds are creative, some are survival. Various rules apply in each of them. We need to keep that true.
Chat across worlds: the kids frequently chat with each other while in different worlds, often calling for each other to come see and try stuff. That’s where the fun is.
Grief prevention and bad word management, because kids.

Ideally, it would just support the CraftBukkit APIs so that CraftBukkit mods would continue to work. Least pain to migrate and all that.

The Alternatives

So of course I first started by searching for posts from folks who’ve made this journey in the past … and found nothing significant. I think my search-fu must be inadequate. So, I spent a couple of hours just reading forums to see what was possible.

Vanilla Minecraft with Forge: promising, especially since Arun Gupta and son authored a book about modding with Forge. Not API compatible with Bukkit, but has reasonable docs and there are some efforts to add Bukkit APIs via plugin though very old; also, there’s a plugin to add Bukkit functionality.
Sponge: Promising, and there’s a short migration guide but no CraftBukkit API compatibility, so we’d need to find new plugins to handle the job and Ore, the plugin directory for Sponge seems not to be ready for production.
Spigot: Spigot, an early fork of CraftBukkit, aims to be a more efficient version of Bukkit for modding Minecraft. That means API compatibility is a given, and explains why it’s the leading server according to MCstats.org*. Very promising, and very active
Craftbukkit?!: Yup, you read right. Since Spigot is an early fork, they still contain the forked CraftBukkit code and avoided the DMCA takedown issues. Building Spigot also builds CraftBukkit, so I could just use that and move on, in theory. CraftBukkit is dead, long live CraftBukkit.

*MCStats requires a plugin to send data, so Spigot is the leading server among folks who run the MCstats plugin when the MCstats servers are up to collect data, yada yada.

There are many others, but these are the serious contenders for us to migrate our server to. I’ll be exploring them and try to do a decent post on how we went from where we are to there.

Moving Your TypePad Blog To Jekyll: The Re-Translation

2015-04-26T00:00:00-07:00

In my first post about moving from TypePad to Jekyll, I discussed a few ways you can translate your TypePad.com blog into Jekyll, noting that the Jekyll Importer would be the best way to go but for its lack of an MTIF importer. As I got further into the post-install cleanup, there were enough little things like lost permalinks and the like that I decided to go the whole way and add MTIF as a Jekyll importer.

Right now it’s just my fork of the jekyll-import project with a wholly untested MTIF converter implemented on the typepad_converter branch. As a bonus, it’ll convert your old posts to Markdown using Johannes Opper’s reverse_markdown gem.

If you’re desperate to use it before I get it tested and merged into jekyll-import, you could do something like this:

$ cd ~/work
$ git clone git@github.com:purp/jekyll-import.git
$ cd jekyll-import
$ git checkout typepad_export
$ RUBYLIB=~/work/jekyll-import/lib jekyll import mtif --source path/to/your/mtif

Yes, I have shame for not TDD’g this. I’ll confess that I don’t TDD when I’m exploring something new as I’m usually learning so much that my test hypotheses are crap at the beginning.

Cue discussion of ends and means and justifications.

Announcing the MTIF gem

2015-04-25T00:00:00-07:00

As I got deeper into the blog move, I knew I’d need a better translation; to do that, I needed a better way to read and write MTIF files. Since there wasn’t good, reusable code out there, I wrote the MTIF gem (source).

It’s rough and certainly incomplete but fully tested. All contributions welcome.

Moving Your Blog To Jekyll: The Translation

2015-04-19T00:00:00-07:00

UPDATE: I’ve written an MTIF gem and have a working jekyll-import with MTIF though it’s not yet merged into the main project.

tl;dr: If you’re planning to move or export a blog to Jekyll, first look at Jekyll Importer. Hopefully, you’ll save a fair bit of time. If you’re moving from Typepad, it’s a bit complicated.

Moving from Typepad is relatively straightforward, but not yet a smooth path. Step 1 is universal: export your blog to MTIF format. From there, you’ll find a number of ways to do the translation.

I first found https://github.com/dams/typepad_to_jekyll, a perl script which does a good chunk of the work. I found a couple of burning issues with this one:

It used 1 or 0 for the Jekyll Front Matter published tag. Jekyll requires this to be true or false. Since 0 is truthy in Ruby, this means all your drafts would be instantly published. Turns out I had an embarrassingly high number of drafts.
It didn’t quote post titles. If your post titles contain quotes or colons, this could cause Jekyll to fail to generate these properly.

I forked and fixed those, but there were other things I wanted to get working like carrying the permalink and the date into each post. In the end, I decided to move on. This is when I discovered jekyll-import, which handles a Movable Type instance by attaching to the database but doesn’t handle a Movable Type export file.

Okay, so all I need is a parser for the export file and writing the importer for Jekyll will be a snap. Ideally, there’d be a gem I can import, but life is rarely ideal. Instead, I found two other projects who’ve written parsers in Ruby:

Typepad to Jekyll (blog post). Chris found the same perl script and wrote this instead. It does a fair job, but the code isn’t segmented in a way I could use directly in a jekyll-importer.
Marc Chung’s ruby-mtexport. This has a well-segmented parser, but isn’t a gem. I’ve written Marc a note to see if he’ll gemify it; if not, I’ll write up an MTIF parser gem and make the importer.

So hopefully this is a good jumping-off point if you were looking to move from Typepad to Jekyll. I’ll either update this post or add another when/if I get back to the jekyll-importer project.

Goodbye, Typepad; Hello, Github Pages

2015-04-18T00:00:00-07:00

I’ve been using Typepad to blog for nearly ten years now. When I started, I was a geek with typographic and design skills but very little feel for how to employ them on the web; Typepad made it easy to focus on the content and plug in a reasonable design.

Since then, I’ve learned more about front-end web engineering, gotten busier, and have generally meant to move onto something more controllable. That always seemed to mean setting up a full app, which seemed a bit of overkill for my blog … so I never got to it. Then along came the combination of Jekyll and Github Pages and now I’m making the jump. I’m using the So Simple theme as a starting point, but who knows where it will go? I don’t.

I’m sure things will be a bit bumpy around here for a bit. Hopefully I’ll blog a bit more about some of those bumps, and a bit more about leadership as well.

No promises.

Setting Up A Minecraft Server For Kids, Part 1: Preparation

2013-04-12T09:00:00-07:00

I’ve got a little more time on my hands than usual at the moment, so I decided to set up a Minecraft server for my son and his friends. I figured it would force me to play around with cloud APIs, some linux configuration that I haven’t done for a long while, and other various bits of geekery that would just make me happy.

As I started to do some research on how to go about it, I found about a gajillion posts on how to set up a server (including this slightly outdated gem), including a few from smart friends of mine. After a bit of reading, it became clear that this could be a much larger undertaking so I decided to chunk out the project into four steps:

Preparation: research and learning
Installation: set up the instance and server
Configuration: tweak the server config and add plugins to get what you’re looking for
Productionization: make the installation reproducible and automatable; set up backups; etc. And yeah, that’s not a real word. I’m okay with that.

This post will focus on the preparation required to get as many bits sorted out as you can before installing. I’ll write about the rest in the next few posts.

So, Minecraft …

If, like me, you came to this with little practical Minecraft experience, the first thing you should do is try playing a bit. There are a few free ways to do this — such as Minecraft Classic in your browser or Minecraft Pocket Edition Lite on an iOS device — or you can just spend the ~$30 and jump in. I figured that to be an op I’d want my own account, so I went for the full experience.

I died a lot. I crafted a bit. I created a bunch in single player. I joined a few multiplayer servers. I died some more. Through all of it, I got a sense of the game and what my son and his friends would be in for. This helped a lot as I started to figure out what it would take to start up and run a server, as it identified some key questions, like, “How can we have a creative world as well as a survival world?” and “What’s the difference between ‘easy’ and ‘peaceful’?”

These led me to the Minecraft Wiki. Frankly, had I read much of it before playing I would have died less and crafted more. However, it helped me get clear on a few things that matter when you’re considering running a server for others.

Game Mode

The default is survival mode, which means that, in order to survive and craft things, you have to walk around and collect block types required by recipes while avoiding being killed by monsters (“mobs” in the old MUD parlance). There’s also creative mode, where you gain the power of flight, you have access to every block type instantly, and you can make anything your heart desires. My son is currently constructing a flush toilet in the hotel he’s making. His heart desires strange things.

Difficulty

This ranges from Peaceful, where any aggressive mobs are removed or don’t spawn, through Easy and Normal, right into Hard. The wiki does an excellent job explaining.

World Types

There are a few settings in there that weren’t obvious on the face of it, such as what Large Biomes is or why Superflat would be interesting. It turns out that Superflat is a great canvas for Creative worlds, and Large Biomes is pretty much just like the default biome-based terrain generator but each biome takes up more real estate.

It was clear to me that we’d want a Superflat Creative world and an Easy Default Survival world.

Servers, Servers Everywhere

There are a lot of available servers out there. I could tell you that I did an exhaustive search and evaluation, but the truth is that I trusted my smart friend, Sarah, and chose Craftbukkit (CB) for many of the same reasons she did: we both know and trust Tom Enebo’s opinion, it’s easy to extend using JRuby, and it seems to be the leading server beyond vanilla Minecraft.

Another bit of validation comes from cheracc, who runs Sandlot Minecraft server for kids and families, and who also uses CB with many plugins. More on that later, though you can read his excellent Reddit post on setting up a Minecraft server for kids now.

One other minor note: since CB is a third-party server, it takes them a bit of time to catch up when Mojang releases a new client version. This means you’ll want to wait on client updates until there’s a beta version of CB that handles the new client version. As of this writing, the current client requires a 1.5.x server which means craftbukkit-beta.jar

Does Size Matter?

Finally, I wanted to figure out how big an instance I’d need to host a decent server. I flailed my way through several posts trying to figure this out until I discovered Can I Host A Minecraft Server, which makes it easy to take your bandwidth (both up and down) and the available RAM and figure out about how many people you can host. Remember to use the bandwidth figures for the cloud provider you’re planning to use. For me, it turns out I can handle ~12 players in a 1GB instance with the usual cloud host network speeds of >5Mbps, which is what I was hoping to cover.

So that’s it for prep. Now it’s time to instantiate and install.

Bashing tcsh

2012-03-26T22:45:04-07:00

After 15 years of non-stop tcsh action, I’ve finally made the jump to bash. It was painful and I miss all my lovely variable expansion (^X $), wildcard expansion (^X*) and term completion (ESC-/), but there are so many tools like rbenv that expect that you’re using bash with no tcsh-friendly alternatives.

On the upside, I’ve got functions now. I guess that’s something.

Oh, and hi again.

Advanced YAML: Tricking Out Your database.yml

2010-08-13T16:03:33-07:00

I’ve been cultishly carrying this snippet around for years:

login: &login
 adapter: mysql
 host: localhost
 username: myuser
 password: mypass
 encoding: utf8

For a couple of reasons around running DB migrations in differently configured environments, I wanted to factor this into bits common to all environments and bits particular to the config on the local machine. I came up with something like:

# Provide default local block
local: &local
 socket: /var/run/mysqld/mysqld.sock
 
common: &common
 adapter: mysql
 encoding: utf8
 reconnect: false
 pool: 5
 username: myuser
 password: mypass
 <<: *local

development:
 database: myproj_test
 <<: *common

This works swimmingly, but I was tired of ritually mimicking and extending the original snippet without actually understanding it. I decided to dig deeper into what was actually going on and found, unsurprisingly, that I stand on the shoulders of giants like Doug Alcorn, James Duncan Davidson, and Ben Bleything. To render proper credit, Ben’s post from June 2006 builds on a missing post from James’s blog, both of which were preceded by Doug’s observations in March 2006, and all of which draw on tricks originally found in the Typo database config. None of them seems to have detailed what’s going on with the &login/*login construct, so I thought I’d dive into that a bit for posterity.

YAML is nothing more than a human-readable way to serialize basic data structures like lists and associative arrays (aka hashes), which makes it ideal to represent the configuration values needed by Rails. It turns out that YAML provides anchors (&), references (*), and associative array merges («), all of which allow you to include by reference either by assignment or by merging a referenced array into another array.

Here’s an example of each:

# Generate a reference
mammal: &mammal_ref
 warm_blooded: true
 lays_eggs: false

# Define via reference assignment
beaver: *mammal_ref

# Define including a hash merge
otter:
 cute: true
 <<: *mammal_ref

# Define including a hash merge, overriding a value in the reference
platypus:
 <<: *mammal_ref
 lays_eggs: true

I found that the Wikipedia entry on YAML was a good, quick overview; the official YAML spec has all that and more, but it’s not a quick read. Meanwhile, both Doug and Ben observed that bringing ERB into the mix let’s you seriously customize things. I took a bit from both of them; here’s my database.yml now:

# Provide default local block
local: &local
 username: myuser
 password: mypass
<% if File.exist? "/opt/local/var/run/mysql5/mysqld.sock" %>
 socket: /opt/local/var/run/mysql5/mysqld.sock
<% elsif File.exist? "/var/run/mysqld/mysqld.sock" %>
 socket: /var/run/mysqld/mysqld.sock
<% elsif File.exist? "/tmp/mysql.sock" %>
 socket: /tmp/mysql.sock
<% end %>
 
# Allow for local DB configuration
<%= File.read(File.join(File.dirname( __FILE__ ), 'dblocal.yml')) if File.exist?(File.join(File.dirname( __FILE__ ), 'dblocal.yml')) %>

common: &common
 adapter: mysql
 encoding: utf8
 reconnect: false
 pool: 5
 <<: *local

development:
 database: myproj_development
 <<: *common

test:
 database: myproj_test
 <<: *common

production:
 database: myproj_production
 <<: *common

Works a treat. Thanks, Doug, James, and Ben.

Live Blog: Comet & Other Browser Stuff

2009-10-15T13:27:25-07:00

It’s a big component of what we call “real-time”

This is the kind of stuff that puts up a banner on a Twitter search (“# New Tweets Found”) and makes things feel real time.

List of what’s being done to move notifications from server to browser:
* Ajax polling: Anybody using the JS set-timeout function, then refetch. Used by GMail, Campfire, et al
* Long polling (typically Comet): Used by FriendFeed, keeps an http connection open as long as possible (keep alive), reopen when timed-out. Keep a local thread pool.
* Flash sockets: Same as long polling, but from Flash. Don’t have the same-origin policy. No limitaions on number of connections. Does gzipping.
* Reverse HTTP: hosting a little webserver inside a JS connection. Long pushing?
* Silverlight (“MS’s version of Flash”)
* HTML5 websockets: part of spec, doesn’t exist and not implemented. Expected to be like flash sockets.

(things discarded as too old/obscure/painful/absurd):
* Java applets (David Weekly is about to get himself kicked ;)
* Big ugly JS kludges (iframe tricks, etc)
* What’s that Opera thing … unite? ubiquity? Unite. Every browser is a webserver. Sounds like reverse HTTP.
* Using XMPP format, arbitrary JSON structure, Atom.

Libraries:
* Orbited (?) tunnels TCP thru HTTP, treats each end like a socket. JS front-end, Python backend. Often paired with Twisted. Specifically meant to be Comet.
* stropheJS: javascript, can use flash sockets
* Tornado: python
* Cometd: Java, made by Dojo to work with dojo
* Dojo: javascript, can use flash sockets
* APE project (?)

What formats are people using to send data:
* JSON
* XMPP

Apps and their libs/methodologies:
* Meebo: ??
* Google Wave/GTalk: GWT RPC, long-polling, their own JS
* FB Chat: ??
* FriendFeed: Tornado server with long-polling, their own JS
* Superfeedr: uses BOSH, stropheJS. BOSH is kinda long polling, bidirectional (two cs open all time). Very similar to comet, more friendly than strict proxies
* Collecta: uses BOSH
* Twingly: orbited
* PBWorks: long polling on network dashboard to see updates come in live; wrote their own libraries, also use stropheJS
* StatusNet - identi.ca and ??, orbited and cometd (you can do either)

Flex and Air apps? The most interesting stuff is what used to be called Flash Media Server (been renamed). They’ve got their own P2P protocol. All sorts of funky stuff you can do. OpenSource version of FMS is Red Five.

In what situations are these libraries breaking down?
* Transparent/Opaque proxies that give repeats/dupes/hangs. Keep buffer on server and check for these. Sometimes have to close connection to flush thru proxies.
* “Everyone focuses on the newest hottest event-based framework, but the hardest part is that HTTP wasn’t designed for long-polling.”

Have to hold open request/response pairs. Connection setup and teardown is expensive

Guy at UK telecom has only ~3K IP connections available in London area. Comet is going to force an upgrade of their hardware, which will be expensive. Real world constraints will always present.

Real-time at UI can be distracting at best, horribly annoying at worst. Charts work great, but text moving too quickly becomes hard. UX is challenging. Would be nice to add items while autoscrolling relative to the focussed item

When you want to add a pause button to your site, that’s FAIL. But the hover-over-conversation to pause semantic is pretty good.

Go to jschat.com to see a bad resize.

FB pioneered notifications really well in UX. You get a “toaster popup” that doesn’t disturb your screen and it fades after a few moments plus a bar at the bottom to persist the aggregated notifications/count. Really nice.

Growl is also mentioned as a good model. Is there interest in a notification aggregator with contexts in the browser? Meebo wants to go that way as a notification aggregator. Adobe wants to do this on your desktop. BrowserPlus also hits Growl. Prowl == Pushed Growl.

If everyone started using XMPP, it would increase the message load (“three bazillion individual messages”). No one is bundling XMPP updates; no reason you couldn’t. ActivityStreams could also serve.

Why are people using Atom/JSON for this? They’re so extensible, but when you extend them so far they’re just as verbose as XML. “But we just don’t like XML.”

Would be nice if you could select things to pull out of the stream and hold onto.

Seems to be consensus that chat belongs at the bottom of the browser, append at end. Everything with a permalink (blogs, tweets, etc) gets added to the top (?). Seems to be related to height issues.

Infinite scroll vs. “More” button

Titlebar flash (Gmail and GTalk)

Some sites make a tiny flash widget to play a sound

Haptic: twitter dmesg to cellphone buzz

Notifications can become another form of info overload/noise

(We flailed at making a 1D/2D chart to represent frequency and value … FAIL)

Phonetop apps are like desktop apps. Leah feels strongly that they’re headed down-and-to-the-right and browser-ish apps are taking supremacy due to interoperability.

Apple’s surprise: phonetop apps took off. (?) vs. John Gruber: “The most used app on my iPhone is Safari.”

You can find the whiteboard pics in my Flickr stream.

Live Blog: Web Aggregation, What Works and What Doesn't

2009-10-15T12:17:59-07:00

[note: I originally scribbled this on paper thinking I could hand it off immediately, preventing the obligation of typing, posting, etc. Turns out I don’t get off that lightly, so here’s the spew in electrons.]

Scraping isn’t a scalable model.

There are biz issues around aggregating data: many businesses don’t want you to get their data, though many are becoming more open.

Doing aggregation right:
* minimize latency
* maximize engagement

When latency is high, it causes confusion and takes you out of real-time

Doing conditional gets can be somewhat useful.

Plaxo had to shard their crawlers, which lands you in the shared state/sync problem of any stateful system you want to scale horizontally.

Gnip integration has been good:
* Offload the long-running processes
* Gnip offers alerting or “fat ping” (ping includes update data)

Plaxo likes using the alert to escalate the priority of the crawler which fetches the rich data related to the update. This approach allows you to use a consistent model for content ingestion vs. get info from fat ping, then augment later.

Smarr: “Brad Fitzpatrick said, ‘Make polling a special case of push.’” He attributed this to someone but I missed the attribution.

(Don’t try to keep up with Joseph Smarr on paper. He’s thinks too many cogent thoughts too quickly to preserve legibility)

Plaxo uses TripIt’s RSS feed as alerting, grabs item ID, then uses their APIs to fetch rich data.

There’s a move to homogenize the info from sites, which may not be a good idea. It suppresses the distinctive look and feel/experience of the publishing site. Allowing for these differences means more labor spent on making one-off shims, which increases maintenance. Still, right choice in order to provide value to the user.

Activity streams seek to provide more rich data in a somewhat normalized, extensible format.

Many/most sites aren’t yet perfectly architected for real-time’s push, ping, etc.

PubSubHubBub and Activity Streams are externally represented data shards

Plaxo’s Pulse started with known architecture issues (in order to ship) and hit the wall sooner than expected. Threw hardware/software optimizations at the problem to move the wall far enough to give time for rearchitecture, sharding, and working out how to propagate changes throughout the system properly.

None of the NoSQL alternatives are quite ready for prime-time. Smarr: “It should be something that’s just a primitive.”

Conversation platforms are slightly different sorts of aggregation platforms. There are UI diffs (e.g. pause the stream when indicating interest). Handling the transition from slightly-latent/passive real-time to synchronous real-time/active not yet well-developed (think: when a comment inspires a conversation)

90-99% of the value of the real-time web is realized in not-real-time [unreal-time? ;] This is a big deal for discovery. Twitter and FB make this harder by obscuring history.

Ideal scalability/performance would be an index per user. This would be grossly inefficient due to the number of duplicate entries.

No one has nailed reader-controlled aggregation (Show me Joe’s tweets and blogs but not his photos) quite yet.

Smarr: “If we’re all kinda [sharing], we’re all making each other smarter”

The firehose of info is a hard model to scale to. Ben Metcalfe proposes the garden hose – a firehose filtered at the source according to your interests, which helps aggregators by allowing them to request the superset of all filters from a given publisher.

We really want to push contexts to the publishers and let them determine which content fits that context. Context shifts over time: Joe doesn’t normally read my tweets (and why would he?) but when we’re at a conference together, he’s much more interested (thus the popularity of hashtags). This is a geographic and purpose-driven context (the conference) as well as Joe’s context on me (Jim knows where the good bars are).

Folks like Twitter are so overloaded with info that they might not recognize non-immediate contexts that are interesting to me.

There’s also the risk of exposing users to the amount of correlatable public data they have. Many don’t want you to apply a transitive closure to identify them in all spaces even though doing so allows you to present a much more convenient UX around what they want you to aggregate.

Someone likened the real-time aggregation problem to a bar conversation: you get snippets here and there and follow your own thread of interestingness.

Three fundamental themes:
* How to specify contexts to data provider/publisher
* How to control access to private data (and carry ACLs with that data)
* How to do all this efficiently

Plaxo implemented polling-back-off (poll infrequently updated sources less frequently). Turns out this is a bad idea, as it introduces latency which makes it feel broken.

There’s also the issue of aggregating conversation about web objecs (like blog posts) and how not to divert the conversation from the publisher’s site. However, sometimes you want a private discussion of a public object (cf. LinkedIn company groups discussing an article)

Q: What’s the state of open standards around this?
A: PubSubHubBub and Activity Streams are very exciting. OAuth as access delegation. There’s still a lot of ground to cover.

Stepping Into BDD with Cucumber and WebRAT: Structuring the Work

2009-09-13T14:45:32-07:00

This is the first of a series on my adventures in going BDD with Cucumber and WebRAT with a greenfield project. Watch for the next article coming soon.

I’m working on a tiny project for the dads club at my son’s school and using the opportunity to plunge fully into BDD with Cucumber. I’ve been tinkering with it a bit here and there, but this time I’m committing to fully specifying my acceptance criteria before I start writing specs for my implementation. I have to admit it feels a bit onerous right now, but I can see the value coming so I’m sticking to it.

My usual working methodology is:

Write a quick set of use cases
Infer features from the use cases
Divide the features into logical delivery phases
Start spec’ing the implementation
Code to the specs
Profit!

… so really I’m just adding a “Specify features using Cucumber” step between 3 and 4. Sounds small (but it’s not).

The use cases for this project are pretty easy:

Use Cases:

NON-MEMBERS:
* Join

MEMBERS:
* See news postings
* See calendar events
* Subscribe to a calendar feed
* Read and send email to the group
* Maintain a member profile (email, phone, location, kid info, etc.)
* Browse and search a member directory (by grade, class, proximity/map, etc.)

PREZ FOR LIFE:
* Write news postings
* Post new events
* Send news/reminders to various publishers (group mailing list, school newsletter editor, etc.)

(Yeah, we have a President For Life. He’s awesome.)

Time to break these into features. I try to scope each delivery to be a useful increase in features and something I can delivery quickly (I don’t get much geek time these days ;). Some of this is old hat (it’s mostly just a CMS), but some of it calls for me to learn about things I’ve never played with (Google Maps APIs and Geolocation, for example). Additionally, it was important to get something up fast last week so the main URL could be included in some printed materials for back to school week.

Here’s the feature rough divided into phases:

Phase 0: Site
Non-Members can see front page

Phase 1: Members
Non-Members can join
Members can sign in with email and password or OpenID
PFL can manage news items
PFL can manage events
PFL can manage pages
PFL can suspend accounts of non-PFLs
Members can view news and events
News snippets and event titles show on top page

Phase 2: Admins
Members can be made admin by PFL
Admins can manage news items
Admins can manage events
Admins can manage pages

Phase 3: Feeds and Mailings
Members can subscribe to news and events feeds
PFL can set reminders to be sent by email
Members can recover passwords by email

Phase 4: Profiles
Members can add email, phone, address, availability, notes, profile pic
Members can add kids: name, age, picture, teacher
Members can mark info private, admin-only, or open

Phase 5: Directory
Members can browse directory info

Phase 6: Maps
Members can see people, events on map

Phase 0 was “get something up fast;” we made it a simple redirect to the existing Google group. Phase 1 is the biggest chunk of features, but they’re very straightforward. Each phase after that introduces enough new functionality to be interesting for the members, but in small enough chunks that I should be able to knock ‘em out in a weekend, plus-or-minus.

Okay, so now I’m ready to generate the app and cucumber-ify it. I’ll cover that in the next article.

Generate the regex for a TLD hostname from Perl

2009-08-18T19:05:19-07:00

This was a quick, fun exercise to remind me that I can still write Perl. It fetches the list of TLDs from IANA, does a quick bit of munging, then renders a regex which should match any valid FQDN:

#!/usr/bin/env perl
use strict;
use warnings;

use LWP::Simple;

my $fqdn_regex;

if (my $content = get('http://data.iana.org/TLD/tlds-alpha-by-domain.txt')) {
  $fqdn_regex = '(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+(?:';
  $fqdn_regex .= join('|', grep (!/^(#|xn)/i, (split /\n/, lc($content))));
  $fqdn_regex .= ')';
}

my $regex = $fqdn_regex . '(?:\s|\/|$)';
print "$regex\n";

Several caveats:

It doesn’t match IPv4 dotted quad nor IPv6 ::-notation
It intentionally ignores Internationalized Domain Name in Applications (IDNA) domains
It borrows from my favorite reference for this, regular-expressions.info’s page on email address regexes.

Maybe I’ll extend it for completeness and/or rewrite it in Ruby someday. Until then, it’ll always be ~/bin/tld_regex for me.

Gmail no longer beta? True dat.

2009-07-07T10:12:31-07:00

I guess it’s no longer under construction, or so sayeth the GMail blog which also shows those who can’t live without that ubiquitous beta tag how to bring back their old friend:

Michael Jackson's Obit via Wordle

2009-06-25T18:40:55-07:00

Story by Yahoo!, picture by Wordle.net:

LiveBlog: 10+ Deploys A Day: Dev and Ops at Flickr

2009-06-23T13:47:46-07:00

Update 1: Slides are now on SlideShare.

Update 2: Video now available on blip.tv

John Allspaw (Ops) & Paul Hammond (Eng), Twitter

“Actually work together and aren’t huge assholes to each other.”

(omitted: photo stats … that’s a lot of kittens)

Dev vs. Ops

It’s not my {machines,code} it’s your {code,machines}

Spock v. Scottie analogy

Ops as grumpy old man, says no all the time, cycle of “no all the time because no one tells them anything because they say no all the time”

CW: dev job to add features, ops job to keep site stable and fast

Flickr: Ops job is to enable the business (Dev’s, too)

Business requires change, otherwise you’ll be overtaken by the new guy … but change is the root cause of most outages.

Discourage change vs. Allow it to happen as often as it needs to (via tools and culture)

Lowering the risk of change via tools and culture.

Increase confidence in change goodness
Increase ability to react to those changes

You need {ops,devs} who think like {dev,ops}

Role and Config Mgmt
Shared Version Control: everyone looks in the same place for everything

Code and config in same place
Everyone has access—transparency
Everyone knows how to use it

3a. One-step build

Everything you need to do to convert svn co’d code into what goes to the site—one command
They have “Perform Staging” button at bottom of a page with stats on latest commit

3b. One-step deploy

Top of page is deploy log with notes: who, when, what (link to changes)
Bottom has “I’m feeling lucky!” button to deploy
Continuous deployment

You can’t pretend to deploy 10 times a day if you go down 10 times a day. That’s not being agile, that’s being retarded.

They use Hudson to generate packages which can be deployed by ops

Small frequent changes make it easier to see what went wrong and recover when needed

[missed that tag]

Branching is all about managing bugfixes
Always ship trunk
Branch in code instead: use conditionals to block out pre-release features and configure off/invis—provides an operational lever for adjustment
Makes these open for private beta in production on production hdwe, etc.
Allows dark launches, which allows you to size appropriately, fix major oversights, take the fear out of major new launches

They have a couple hundred “free contingency switches” to turn things off/throttle things down. Gives broad operational control to minimize effects on the site.

Tend to fail forward using these and fix the problem.

Shared metrics

You can see mine, I can see yours
Use ganglia as console
Devs know where dashboard is, and watch as obsessively as ops
Includes app-level metrics (which exposes them to Ops)
(helps drive accountability in both directions in the org—both can see and feel ownership)
This begins to create opportunity to gracefully collaborate to back off an oversub’d resource/degrade/throttle as needed
Show last site deploy info on every page/graph; you can corellate a change in the graph

IRC and IM bots

Heavy IRC users for ongoing dialog between dev/ops, remote/local
Last.fm wrote a tool to inject events into IRC (monitoring, events, deployments, builds)
Log it all and put it in a search engine

Culture

All the tools in the world won’t help if you have a contentious culture

Respect

No stereotypes: not all devs are lazy/cowboys, not all ops are obstructive
respect their expertise and opinions and responsibilities (they influence priorities)
Don’t just say “no”—it’s like saying “I don’t care about your problem/perspective”
Best solutions are collaborative. Memcache is an excellent example; written to solve the problem of DB overheat, which impacted both
Don’t hide things: share your solution even if you think they’ll say no; you deny their expertise and input

Talk about the impact of your code push

What metrics change
what are the risks
what are the symptoms of somethign going wrong
what are the contingencies (rollback)

Trust

When you walk up with all the above on hand, you demonstrate that you care enough about them and the business
“I don’t want to tell X …” == you’re a cowboy, and “cowboys are losers”
Have shared playbooks and contingency plans so all understand.
Provide as many knobs and levers as you can so Ops can tweak to match the env
Ops: be transparent, give devs insight and access to the systems. Playing telephone around shell commands is dumb.
It’s hard to help if you can’t directly see

Have a healthy attitude around failure—it’s going to happen.

Think about how you’ll respond more than you think about how you’ll prevent it
Would you rather be treated by a GP who deals with heart attacks infrequently or an EMT who handles them weekly?
Fire drills: when ops and sr engineers are fixing a problem, have others diagnose live in parallel (but make no changes!)

Avoiding blame

they have a rule of no-finger-pointing; it doesn’t need enforcement, folks step up
fixing blame wastes a ton of time, why not skip it? Feel guilty afterwards if you must.
They’ve got a bit of a potlatch culture as people try to assert responsibility in order to fix things.
Remember that when your code breaks, someone’s going to have to wake up to fix it. Own it and apologize, at least. Otherwise, you’re back to not respecting each other (“Screw you … aren’t you getting paid to do that?”)

Ops should provide constructive, continual feedback on how it’s going. Point out interesting things before they’re critical

LiveBlog: Q&A with Twitter's John Adams

2009-06-23T13:28:57-07:00

A short pre-lunch session to absorb a few moment:

John Adams (Twitter) Q&A

Q: How do you log all the info from your APIs?

A: syslog, looking at scribe, generally summarize and toss

Q: How do you control abusive clients?

A: Rate limiting, apply feature limits to abusers, etc.

Q: What would you do differently?

A: Implemented change controls much sooner. Process is much better now with more control, predictability

Q: How does your on-call team work?

A: More people reduces length in rotation. Nagios with alerts and aggregation of alerts. Make alerts actionable (db fails? see one page for db down, not 500 webservers). Also prevents burnout

Q: Carry a real pager?

A: Some, mostly SMS. There are escalations if you don’t answer. Always someone from Ops and Eng on the pager chain.

LiveBlog: PageSpeed

2009-06-23T13:19:27-07:00

Bryan McQuade (Google), Richard Rabbat (Google, Inc.)

What’s Page Speed

FF/Firebug addon
http://code.google.com/p/page-speed
Optimizes images, minifies JS, tells you what you should defer
100K downloads in 10 days, 1000s of tweets, 100s of blog postings

How’d it start

wanted to help stop relearning lessons in new apps
keep you from unintended consequeces (e.g. vary header, squid won’t cache anything with params, etc.) Google map tiles used to have ? in URL; removing it gave huge boost in perf, huge reduction in requests
Sourced from many smart people on the web

Prioritizes according to importance/savings, gives easy way to see detailed info about any given rule, what’s violating it, docs on why and how of each rule

Defer loading JS as much as possible

Rule looks to see what’s not been invoked before OnLoad completes
Not enabled by default, has perf hit
Load multiple times in multiple scenarios; some JS is triggered in different

Inefficient CSS Selectors: based on David Hyatt’s post on inefficient selectors

Activity Panel

Show where most time is spent, where you should focus optimization
Shows DNS, network, connection, latency, data available for use, JS parse/exec, cache hits
Coming: paint events, screen snapshotting

BBC “has lots of room for optimization” gmail waterfall is hugely vertical!

Just released FF3.5 compatible version

LiveBlog: 2 Years Later, Loving and Hating the Cloud

2009-06-23T13:16:21-07:00

LiveBlog: Fixing Twitter

2009-06-23T13:07:14-07:00

John Adams, Twitter Ops

Ops

Small team
SW perf
availability is their primary focus

All on managed services with NTT - No clouds—too high latency - NTT runs the NOC - Frees them to deal with real thinking compsci probs

752% growth in 2008, trend happens ~11/2008 and keeps climbing

Growth Pain Fear of what’s gonna happen

Mantra:

Find the weekest point (metrics + logging + analysis)
Take corrective action (process)
Repeat

Find weak points

Collect metrics and graphs (individual metrics are irrelevant)
Logs
SCIENCE!
Instrument everything! More info is better

Monitoring

Keep critical metrics as close to realtime as possible
Using RRD, Ganglia + gMetrics, MRTG
Mostly on 10s interval, some 5s, some 60s
Everyone in company has access to dashboard
“Criticals” view
Use google analytics for failwhale and other err pages

Analyze

Turn data into info
Are things better/worse post-deploy
Create env of capacity planning, not firefighting—no more cowboys in the wild west

Deploys

Ganglia shows final deploy info for twitter, summize, and search

Whale-watcher

simple script with massive win
503 is a whale, 500 is a robot
Whales per second exceeds whale threshold then “There’s whales!”
Darkmode: selectively disable portions of site with automatic notification to product and eng teams to let them know

Config Mgmt

You need an automated cfg mgmt system NOW. Else you won’t scale
It intros complexity, with multiple admins, unknown interactions
Peer review solves most of this; they use reviewboard with svn precommit hook requiring “reviewed by” note in comment and postcommit hook sends note about what changed to people

High communication

They use chat (campfire) with docs, graphs, logs, etc.
skitch into campfire is a frequent working methodology

Subsystems

Many limiting factors in request pipeline
Oversubscribe mongrel 2:1 vs. cores
Attack plan per ssytem (e.g. bandwidth? bottleneck: network, vector: http latency, solution: servers+; timeline? db, update delay, better algo; search? db, delays, dbs+ and code; etc.)

CPUs:

switched to Xeon +30% gain
replace 2x and 4x core with 8x core +40%

Rails:

Stop blaming rails
Analysis: caching/cache invalidation, AR makes bad queries, queue latency, memcache/page corruption, rep lag
Not so much about Rails

Disk is the new Tape

Social networks is very O(n^y) oriented
Disk is too slow
Need lots of RAM

Lots of caching is possible. Moving libmemcached to native C gem was bigtime helpful.

Nick’s CacheMoney AR plugin: readthru/writethru caching with memcached!

Caching everything not smart, either

Cache evictions
Cold cache after host failure/new host spinup
Cache smarter: get rid of cache busting behaviors, varnish with failover, etc. RDBMS vs message queues
Not everything needs ACID
message queues help
Most MQs suck at high load
They wrote Kestrel for this; looks like memcache
Starling was earlier version

Asynch == Good

They lean on mongrel heavily (they know it well)
Keep external service requests out of the pipeline via daemons which process message queues
Size worker daemons appropriately, have them kill themselves off rather than long-run

DB replication

Multiple functional read/write masters
never read from the master—slows it down too much
watch your slow queries
use mkill to kill long-running queries before they kill you.

Put up a status blog on some other service—transparency stops armchair engineering

LiveBlog: Next Web Challenges: It's Still All About UX

2009-06-23T12:59:01-07:00

Velocity 2009, the conference about performance, is very high-performance about getting people on and off stage (and high density around content)

Umang Gupta, Vik Chaudhary (Keynote)

(omitted: Keynote history, 15 years of continuous improvement, etc.)

Debuting Transaction Perpective 9.0 (TXP9)

Embeds real IE browser for monitoring
Adds “screen sensing” technology
Esp. useful for “next web” apps: flash, video, voice, SMS/mobile—composit transactions or flows

(demo: reservations site for The Broadmoor in Colorado Springs, very flash-integrated with lots of client-side action. “Challenge of screen sensing what’s going on on the screen is non-trivial”. Also http://espn.go.com/video/ and Mini-Cooper flash site)

Using KITE platform/desktop environment to record what you’re doing. You click around, type, etc. and it records a script.

(This is somewhat like what they do at [DeviceAnywhere http://deviceanywhere.com/] for mobile device testing. They don’t focus on UX or perf; they’re more on QA testing side)

Script runs and collects UX and Network times. UX time is net time + client-side execution + rendering. Also shows augmented waterfall inclusive of client-side computation, etc.

LiveBlog: The User and Business Impact of Server Delays, Additional Bytes, and HTTP Chunking in Web Search

2009-06-23T09:58:10-07:00

Eric Schurman (Microsoft/bing), Jake Brutlag (Google)

Experiments

Server delays (MS and Google)
Page weight variance
Progressive rendering

They have platforms for experimentation which allow fractional experiments

Divide users into small buckets
use good methodology (control group, experimental group(s))
Way better than usability tests

Server Delays

Goal [missed all of this due to an IM. Lesson learned]

Results

No statistically significant change @ ~50ms delay
Observable and fairly linear impact on delays 200/500/1000/2000ms.
Time to first click took ~2x delay—theory: user has opportunity to get distracted

Google Search Delay Experiment

Varied type of delay, magnitude, and duration (number of weeks) per user group
Pre-header delay: pause server processing upon receipt of req
Post-header delay: pause after sending on header, but before sending results
Post-ads delay: (ads are structurally first in page, can render before search result) put ads in separate http chunk, delay between ads and search results

Results:

Measure average daily searches per user
50ms pre-header delays show no significant impact
100ms pre-head, 200ms post-heads, 400ms post-head, 200ms post-ads (and others) showed linear progression in decreased avg daily searches
Also saw increase in internally monitored “abandonment rate”
Active users are more sensitive
drop-off continued to trend down linearly beyond 4 weeks; effect becomes more pronounced over time, and additive—200ms and 400ms groups diverge more strongly
Stopped injecting delays at week 7; recovery was significant immediately, but not fully realized at week 12—there was still a drop in activity for these groups

Page weight experiments

injected incompressible comments into various places of page
varied size of comments from 5% of page to 500% (most of larger loads were below the fold)
small payloads weren’t worrisome (tho stat’ly significant)
perf suffered slightly, but was US only experiment; global exp planned, will likely show significantly larger drop in perf
Click metrics were hurt more than query metrics

Progressive rendering experiment

Goal: determine impace sending visual header before results
Build page in phases, send using HTTP 1.1 chunked transfer encoding
Results: Large improvement due to parallelization. Time to first click was ~9% faster, more likely to refine query, more clicks, more likely to page thru results

HCI may state that 100-200ms isn’t perceptible; it still has effect.

Getting something to your user quickly is more important than when they receive their last byte

Experimentation platforms make all this research and hard numbers possible.

LiveBlog: After the Click

2009-06-23T09:35:53-07:00

More from Velocity 2009. Going really fast, sorry for all the sloppiness.

Jonathan Heiliger, VP Tech Ops FB

FB Mission: give people the power to share and make the world more open and connected.

2004: launch in MZ’s dorm room 2004-5: new apps launched (events, photos, mobile) 2006: news feed and open reg 2007: platform launch 2008: crowdsourced translations; reached 30 langs quickly (spanish 2 wks, french ~24 hrs)

[nice map viz for growth: colorize market penetration]

Radio took ~150 yrs (?) to reach 150M TV: 13 yrs Computers 4 years FB: 3 yrs

How FB deals

Classic battle of Ops v. Eng

Ops wants no change—stability
Eng wants lots of change—driven by users and site
Do you really want to fight it out? Teamwork is required
Enable individuals to reach goals, chase team success
Make it transparent to users and safe for employees to fail
Make it a point of pride: you don’t want to be the one who took down the site (but there’s some cache in that war story)

It’s the people

Everyone hires the smartest people
It’s about organizing and leading

Tuning the Operating Pipeline (Eng -> QA -> Ops aka Dev -> Test -> Deploy) (this isn’t how they did it)

Engineering is responsible for the efficacy and reliability of their code, writing their own tests, and full lifecycle of code including pushing it live.

Ops provides guard rails to keep eng safe from itself, prevent site downtime. Feature can go down, but rest of site is safe.

Complaints back in the day: Ops: Eng is way too unstructured, lobbing crap over the wall. Eng: Ops is not nimble

Make the problem joint; Eng owns the problem

Continuous build, code review, peer review, perf testing has kept things moving fast while moving to 200+ eng org.

Put engineers in operations

Site reliability team: stewards of the site
Operations engineering: tooling and glue apps (workflow/pipeline)

Put ops in engineering (consulting engineers)

Partners with backend service groups to think about architecture, scaling, reliability
Helps mentor into full SDLC responsibility—really understand complete DEV to PROD function of code

Software launch has warroom with PM, Eng, NetEng, SRE, Perf Eng, Site Integrity staff around. Always the right person on hand, physically present.

Getting it done

If you can’t work as a team, you’re done
Design is awesome, but it needs execution to succeed

Three things they did live expecting to break the site

See how the team worked, who would step up, etc. CNN livefeed
Group of 20 some folks came together, marketing, eng, product, ops, etc.
Added much capacity, made warroom
Written from scratch in ~3 weeks
Replicated (and improved) for Oscars, etc.
Knew there would be point load much like DoS attack
Added throttles to direct features, as well as throttling things like chat, number of thumbnails shown on site, etc.
Friends had to be shown on the fly
Common content was cached in CDN; didn’t anticipate delay/latency from CDN
Didn’t expec users to maddly twiddle “Everyone” and “Friends” tabs (they did) – learned “cache everything”
During inauguraton 2M status updates, 8.5K spike at start
Dark launched everything with users exercising the stack without any visible UI to users
Also built perf framework to see what real user experience would be like
Used data from both to appropriately size

Simple “I like this” on wall/status
Didn’t expect it to get a lot of traffic at first; totally wrong
- 4.1m users liked 7.1 M times first day
- 16.3/46.2 1st week
- 39.6/226.8 1st month

Username allocations

Was initially to be auction (codename: hammer)
Decided to go first come first served, kept codename—it was going to hammer the site
Had to have blocked list of trademarks; didn’t block “asp.net”
Dark launch, found issues, delayed initial launch
Launched at 9p; huge cache hit within moments, no increase in idle latency (means they got it right, maybe a little overprovisioned)
Made pages as light as possible
Tiny blip in overall load

Datacenter infra/organization is hugely important

Untidiness reflects bad organization
DC/infra is 2nd biggest exp after people
Invest where appropriate

Distribute accountability Test with users “The only place success comes before work is in the dictionary” – Vince lombardi

Expects org to look different in a year—evolution is the key.

LiveBlog: Surviving the 2008 Elections at DailyKos.com

2009-06-23T09:03:07-07:00

LiveBlog: Hadoop Operations

2009-06-22T16:39:44-07:00

I’m at Velocity 2009, sitting in on the “Hadoop Operations” talk.

Jeff Hammerbacher, Chief Scientist, Cloudera (email is first six of his last name at his company dot com). He has an ambitious agenda for this session and talks very fast, so sketchy notes and abbrevs for me. Pardon the crappy formatting.

slides are here.

Built data team at FB. ~30 ppl when he left. Built Hive and Cassandra.

Good resources:

“Hadoop: The Definitive Guide” by Tom White (must have)
“Hadoop Cluster Management” slides by Marco Nicosia’s 2009 USENIX talk

Hadoop: OSS for WSCs (warehouse-scale computers)

Typical cluster: 1U 2×4 core, 8GB RAM, 4×1TB SATA, 2×1 gE NIC; one switch per rack with 8 Gb intfc to backbone. Think 40-node-rack as unit.

HDFS: breaks files to 128MB, replicates blocks across nodes. W1RM design. checksumming, replication, compression included (tell you three times). Hooks in via Java, C, command line tools, FUSE, WebDAV, Thrift. Not usually mounted directly.

[how does it handle many small files? see HAR files below, see Common problems below, no statements about performance]

HDFS looks to diversly write blocks (across racks) using topology info.

MapReduce uses HDFS api to assign work to where the data is.

Avro: cross-language serialization for on-wire/RPC and persistence, includes versioning and security

HBase: Google’s BigTable lookalike on top of HDFS

Hive: SQL-like interface to structured data stored in HDFS. Replace DWH.

Pig: lang for dataflow programming.

Zookeeper: manage a distributed system

Good ways to dip your toes with Hadoop:

Projects:

Log or msg warehouse
DB archival store
ETL for DWH
Search team projects (autocomplete, did you mean, indexing)
Targeted web crawls (market research, etc)

Clusters:

use retired DB servers
use unused desktops
use EC2

[skipped a lot about how the project runs, apache voting, etc.]

Don’t run Hadoop across two data centers; one per and communicate at the app layer. [this sounds a lot like the rules for MPI et al ca. 1999-2000]

Make sure to use ECC RAM. High volume mem churn requires it.

Linux/CentOS “mildly preferred”

Mount local FS “noatime” for performance.

Recommend ext3 over xfs. Local FS performance improvements (e.g. xfs) don’t necessarily translate to global perf improvements (network bottlenecks consume it). Mentioned an xfs long-write problem.

JBOD over RAID0; slightly better performance and losing a disk doesn’t suck as much.

Java 6 update 14 or later (update 14 makes 64-bit pointers as cheap as 32-bit).

Installation: http://www.cloudera.com/hadoop

“In our distribution we put [things] where they ought to be.” Register with init.d, etc.

Configuration: http://my.cloudera.com/

You spec topology and whether JT/NN live on same machine, it spits out the rest. Hangs on to it for you, too.

Config modes

Standalone mode:

Everything in one JVM
Only one reducer, so you might not be able to find the bug

Pseudo-dist mode:

All daemons on one box using socket IPC

Dist mode:

For production

Config files

xml based
org.apache.hadoop.conf has Configuration class
Later resources overwrite earlier; “final” keyword prevents overwrite
common-site.xml, hdfs-site.xml, mapred-site.xml
Look in .template for examples

Cloudera admins their soft-layer cluster with Puppet “with varying level of success”. He’s seen Chef, cfengine, bcfg2, and others.

Problems in config:

“The problem is almost always DNS”—Todd Lipcon
Open the necessary ports (many) in firewall
Disting ssh keys (Cloudera uses expect)
directory permissions (writing logs)
Use all your disks!
Don’t try to use NFS for large clusters
JAVA_HOME set right (esp. on Macs)

Nehalems ~2x performance improvement

HDFS NameNode (“the master”)

VERSION file specs layoutVersion (negative number, decrements for each new). You hope this doesn’t change much; upgrade is painful

NN manages fs image (inode map, in mem) and edit log (journal, to disk).

Secondary NN (on different node) aka checkpoint node (v0.21): replays journal and tells primary to forget some history to prevent the edit log from becoming ridiculously large.

Backup node: write same data to NFS to recover if local node blows up

DataNode: round-robins blocks across all nodes.

Heartbeats to the nodes
dfs.hosts[.exlcude] to allow/deny clients

Client:

Use Java libs or command line
libhdfs c library lacks features and has memory leaks (and FUSE interface uses it)
Client only contacts NN for metadata
Client keeps distance-ranked list of block locations for data reads
Client maintains write queues: data queue and ack queue (writes three times, can’t forget request until all three are ack’d).
First datanode in write takes responsibility for pass-down-the-line write requests rather than having client spray data at all 3/n data nodes expected to write.

Can’t seek and write, nor append. So you create new each time.

HDFS Operator Utilities

Safe mode

Loads image file, applies edit log, creates new (empty) edit log
Datanodes send blocklists to NN
NN uses this during startup, will only service metadata reads while in safe mode
Exits safe mode after 99.9% of blocks have reported in (configurable); only one replica of block must be known (can rereplicate)

FS Check (hadoop fsck)

Just talks to NN to look at metadata
Looks for minimally rep’d, over/under rep’d blocks
Identify missing replicas and rereplicate, blocks with 0 replicas (corrupt files)
hadoop fsck /path/to/file -files -blocks to determine blocks for file
Run ~1 hr in production, store output

dfsadmin

admin quotas
add/remove datanodes
ckpoint fs image
monitor/manage fs upgrade

DataBlockScanner

cksum local blocks (with bandwidth throttling)
Runs ~3 weeks (configurable)

Balancer

goes thru cluster, makes disk utilization scores per datanode
rebalances if nodes are more than +/- 10% (with throttling)

Archive Tool

HAR file: like tar file, many entries in one HDFS namespace
Makes two index files and many part files (hopefully less than # of files you’re har’g)
Index files are used for lookup into part files
Doesn’t support compression and are W1RM.

distcp

Move large amounts of data in parallel
Implemented as MapReduce with no reducers
Can move data between data centers with this; can also saturate the network pipe

Quotas

apply to directories, not users or groups
namespace quotas constrain your use of the NN resources
diskspace quotas constrain your use of the datanodes’ resources
No defaults (can’t make new directories pick them up)

Users, Groups, Permissions

Relatively new
Very UNIXy
Executable bit means nothing on file
Need write on dir to add/remove files
need exec on dir to access child dirs
identity of NN process superuser

Audit logs

Not on by default, but useful for security

Topology

Uses to compute distance measures for replication
Node, Rack, Core Switch
Some work to infer from IP

Web UIs

There are many
NN @ port 50070: /metrics /logLevel /stacks
2NN @ port 50090
Datanode @ port 50075

HFDS Proxy: http server access for non-HDFS clients

ThriftFS: thrift server for non-HDFS clients

Trash:

Helps recover from bad rm’s (indavertent rm -rf happened on FB cluster)

Common Problems

Disk capacity: crank up reserved space, keep close eye on space, watch hadoop logfiles
Slow disks which aren’t yet dead: can’t see as fail, but you have to watch
NIC goes out of gig-E mode
ckpoint and backup data: keep an eye on 2NN node, watch NN edit log size
check NFS mount for shared NN data structure
Long writes (> 1 hr) can see things get freaky; break them down
HDFS layoutVersion upgrades are scary
Many small files can consume namespace: keep an eye on consumption

Turn on fairshare schedulers (Cloudera rus it out of the box)

Use distributed cache to send common libs to all nodes

JobControl: good way to express job depedencies

Run canary jobs (sort, dfs write) to test functional status

Upgrades are scary. This will be less true as it reaches 1.0

One admin can easily carry a medium (100-node) cluster. Most activity is around commission/decommission.

Try not to lose more than N nodes, where N is your replication factor. You could hit the jackpot on those being the only three replicas of some needed block.

LiveBlog: Intro to Managed Infrastructure with Puppet

2009-06-22T12:26:47-07:00

I’m at Velocity 2009 and very happy to see Luke Kanies presenting around Puppet. If you’re not familiar, Luke’s a hard-core sysadmin type with the typical sysadmin bent for highly automated systems management solutions which require as little maintenance as possible. If you’re looking at things like cfengine, cft, etc. then you should be looking at Puppet as well.

Meanwhile, it’s been a while since I’ve live-blogged a session, so let’s see if my stream of consciousness technique still flows (or has stagnated):

It’s a workshop, not a talk; to follow along git clone git://github.com/reductivelabs/velocity_puppet_workshop_2009.git

You’ll also want the slideshow gem to see the slides (which are in the git repo): sudo gem install slideshow

You also probably want to install puppet. You can get it from github, through MacPorts, but easiest is sudo gem install puppet

Seek help: #puppet on freenode, puppet-users Google group

The usual problems obtain: keep everything configured correctly all the time.

Puppet provides a resource abstraction layer. Do you remember which command removes gems? Is it the same as the command to remove a package (via MacPorts, rpm, yum, et al)? Why work to remember all of those.

ralsh (resource abstraction layer shell) gives you direct access to the abstraction layer

ralsh package – list all pacakges on your machine via any known package installer; responds in puppet code.

(it chokes a bit if you have rpm installed via MacPorts and don’t run it as root. sudo ralsh package is a good workaround. That’s a bug in the rpm port; it should be able to query as non-root users.)

ralsh user – provides you the info around that user

ralsh user <p> ensure=present shell=/bin/tcsh home=/Users/</p> – be sure the user exists. Runs idempotently. First run creates; second takes no action. Change args (shell, home) and it will change to agree. Change ‘ensure=absent’ and it’s blown away.

Luke now uses ralsh for interactive administration to avoid having to remember all the various details of which args for what command, etc.

The language is mostly declarative and is very simple. No loops; only recently getting conditionals from “this crazy French guy”.

Aliases allow you to think of things by alias/title rather than technical name:

package { ssh: name => $operating systems ? { debian => "ssh", openssh => "openssh", default => "sshd" },ensure => installed}

… and now you can just talk about the “ssh” package, abstracting you from naming strangenesses. You’ll never have a DB of all the strangenesses, so you can at least to insulate yourself.

Executables: puppet, ralsh, puppetd, puppetmasterd, puppetca (there’s more, but that’s what we’ll touch today)

puppet executable allows you to fiddle and iterate via -e and –noop

>puppet -e 'file { "/tmp/foo" : ensure => present}'notice: //File[/tmp/foo]/ensure: created

(note: puppet barfed for me here because ~/.puppet/var didn’t exist; mkdir -p ~/.puppet/var fixed that up)

>puppet -e 'file { "/tmp/foo" : ensure => absent}'notice: //File[/tmp/foo]: Filebucketed to with sum d41d8cd98f00b204e9800998ecf8427enotice: //File[/tmp/foo]/ensure: removed

I was curious about the filebucketed thing, so I looked:

> ls ~/.puppet/var/clientbucket/d/4/1/d/8/c/d/9/d41d8cd98f00b204e9800998ecf8427e/contents paths

Turns out contents is the file itself, paths is the path from which it was removed.

With noop:

> puppet --noop -e 'file { "/tmp/foo" : ensure => present}'notice: //File[/tmp/foo]/ensure: is absent, should be present (noop)

You should keep a single repo of your config/code (see http://github.com/albanpeignier/gepetto/) which you can treat as an application.

A puppet “module” is related config and code (classes, plugins, etc) to handle a particular function – “Why do you have this?”

(jump into repo/modules)

puppet –configprint modulepath – what modules will be loaded? (also: confdir and vardir)

>puppet --modulepath $PWD/modules -e 'include foo'notice: //foo/File[/tmp/foo]/ensure: created

Put your nodes into the site manifest: manifests/site.pp … for simpler sites. As site gets more complex, there are ways to hook puppet to a DB.

default node matches all unmatched nodes. You can also inherit:

node my_host inherits default { notice "I'm your host!"}

puppet makes it easier to capture the many little uninteresting twiddles you forget you did in the middle of the night; you also capture them as somehting you can execute to repeat them.

puppet uses ssl certs for client and server to allow identity verification. Particularly, it uses self-signed certs

Running puppetmasterd in dev:

>mkdir /tmp/server>puppetmasterd --verbose --no-daemonize --modulepath $PWD/modules \ --confdir /tmp/server --vardir /tmp/server \ --manifest $PWD/manifests/site.pp --certdnsnames localhost

… and it’s all rock-and-roll from here.

Luke (paraphrased): “Puppet uses SSL just like your bank uses it – so most SSL errors are not Puppet’s fault. Be sure you know what you’re doing before you twiddle your Puppet conf around SSL; most of the errors people report turn out to be their own attempted cleverness biting them.”

Now use puppetd to talk to puppetmasterd to get config, etc. without knowing about anything:

> puppetd --test --confdir /tmp/server --vardir /tmp/server \ --no-daemonize --server localhostinfo: Caching catalog at /tmp/server/state/localconfig.yamlnotice: Starting catalog runinfo: Creating state file /tmp/server/state/state.yamlnotice: Finished catalog run in 0.01 seconds

It takes about 15 seconds for code changes in the repo to propagate through puppetmasterd.

… and then it’s a blur as he crammed the rest of the prezzo into the last 15 minutes, so I quit typing and just listened.

LiveBlog: Death of a Web Server

2009-06-22T10:40:00-07:00

The first Velocity 2009 session for me today. Semi-useful, but seemed very .NET focused. I didn’t take a lot of notes around it for that reason.

For me, most of his prezzo reduced to:

Make sure your instrumentation has a light touch
Know what you’re caching
Be sure it’s getting hit (don’t cache singleton queries)
Be sure your TTL is well-set (short, perhaps sliding)

Otherwise, most of the interesting bits came from Q&A:

Everyone has a favorite load generator; many love the appliances. They “make a webserver cry”. Mercury LoadRunner seemed like a crowdpleaser.

Log playback can be challenging as you frequently don’t have all the post data, response data, etc. Hard to know how “real” it is. An alternative is to inject a transparent proxy on the front end and capture everything both ways for short periods.

One fellow in the audience is using “Siege” and EC2 instances. [I think this is the Siege he meant. I wonder why no one mentions using Tsung, formerly Tsunami, and EC2 … here’s the one decent ref I could find]

Realisitic tests: be sure what you’re using to load it resembles your production load or it’s not very useful.

All rely on scripted sessions; there’s a gap in converting log data into that script.

Got the "can't activate" Gem::Exception Blues? gem cleanup!

2009-06-14T17:52:03-07:00

What to do when rubygems can’t activate a version of a gem because it’s already activated a different version of the same gem? My Google magic wasn’t good enough to find this one quickly, so I’ll happily point out Jesse Hu’s nearly one-year-old post about the same problem, descended from a Ruby on Rails thread on ruby-forum.com where, unsurprisingly, it’s Jeremy McAnally who knows that gem cleanup is the way to go.

Took nearly 45 minutes. Freed untold amounts of disk. Much happiness.

Cucumber, Webrat ... Who Names These Things?

2009-06-10T20:20:24-07:00

I’m tinkering a bit with Cucumber and Webrat for my day job and am very excited by some of the prospects for our QA group’s automation efforts. Along the way I’m finding that I have to explain how all the moving parts relate to each other, so I made this diagram:

Cucumber, based on RSpec, uses Webrat to drive Selenium. Seems to make it all make sense to the folks with the questions.

(and thanks, brynary + all the webrat contributors, aslakhellesoy + all the cucumber contributors, and dchelimsky + all the rspec contributors, and the whole selenium crew … this is very, very cool stuff).

Generating with Ruby OFF Rails

2009-06-03T22:49:50-07:00

I’m spending a fair amount of my time working on non-Rails Ruby projects lately and I’ve been missing the fine code generation toys in Rails. So tonight, for the fun of it and because I was hankering to play with cucumber, I hacked up the minimal set of bits that gets you script/generate in your non-Rails project.

This is how we do it:

> mkdir my_non_rails_proj> cd my_non_rails_proj> mkdir script config> touch config/environment.rb> cat > config/boot.rbRAILS_ROOT = "#{File.dirname( __FILE__ )}/.." unless defined?(RAILS_ROOT)RAILS_ENV = nilrequire 'rubygems'require 'initializer'Rails.configuration = Rails::Configuration.new^D> cp -p /path/to/gems/rails-2.3.2/bin/generate script/.> ./script/generateUsage: ./script/generate generator [options] [args]Rails Info: -v, --version Show the Rails version number and quit. -h, --help Show this help message and quit.[...]> ./script/generate cucumber create features/step_definitions create features/step_definitions/webrat_steps.rb create features/support create features/support/env.rb create features/support/paths.rb create lib/tasks create lib/tasks/cucumber.rake create script/cucumber> touch Rakefile> rake features[... large lack of FAIL ...]>

Voila! Not guaranteed to work for all Rails generators, but it bootstrapped rspec and cucumber just fine, which was the point of my exercise. Someday I’ll make a generator:bootstrap rake task or gem something if I need to do it again.

Re-finding my rubyligion ...

2009-05-03T11:06:10-07:00

Life’s been busy. Good, but busy. My gig at LinkedIn, two hours of commuting each day, plus our four person plus dog family certainly keeps me awash in things I should be doing. Thing is, ever since I stepped away from managing LinkedIn’s rubyists, I haven’t had much to keep me practicing my Ruby skills or tied to the Ruby community. That’s made me a bit sad.

So, for the last few months, I’ve been finding spare-time, non-urgent Ruby projects on github to play with, forking them when I’m ready to make contributions. The one I’m currently twiddling most is a fork of Jon Maddox’s IMDB library. He got the very basics in to deal with movies; I’m expanding it to handle TV series and episodes as well. In the process, I’ve added testing and some heavy refactoring, mostly to arrange the libraries in a more idiomatic fashion but also to swap out Hpricot for Nokogiri (sorry, why, but Nokogiri has better support for complex XPath and XSLT fun that simplifies some of the code dramatically).

I’m also digging into the guts of the Confluence4R and Jira4R libraries (and I am gratefully indebted to Ben Walding for setting up a github mirror of the Jira4R main git repo at Codehaus); for them, I’m mostly adding spec code to demonstrate what I think is true as I’m learning. Jira4R is driving me a bit nuts – I’m having a love/hate relationship with its wsdl2ruby underpinnings. Slowly I learn.

But, then, that’s the point, isn’t it?

Seven Ways To Approach An Issue

2009-01-29T12:07:40-08:00

I was chatting about this with someone yesterday and made a total hash of recounting it, so I thought I’d do a quick blog post to clarify my thoughts.

This requires that you understand the Circle of Concern/Circle of Influence concept from The Seven Habits of Highly Effective People. There’s a good summary of these concepts at Cut The Knot.

Now imagine that inside your circle of influence is a smaller circle of responsibility. That’s the things you’re directly responsible for doing; the area outside of responsibility and inside of influence is what you can talk others into doing. ;]

With that in mind, here’s a good framework (from a USN SSBN commander by way of Covey’s book, “The 8th Habit”) for how to approach an issue which you believe you know how to resolve depending on your level of influence and/or responsibility:

1. Stand Mute
If it’s completely outside of your circle of influence, stand mute. There’s no constructive course you can pursue. Instead, consider whether or not you’re genuinely concerned; if you are, channel your energies into expanding your circle of influence to include issues like this in the future.

2. Ask Questions
If it’s just outside the edge of your circle of influence (inside or outside), ask questions which tend to confirm your assumptions about the issue. You’ll learn things, you may provide focus and insight, and you might get invited to the solution. You’ll likely expand your influence in the direction of the issue at hand … as long as your questions are pertinent and well thought out.

3. Offer A Solution
If it’s just inside the edge of your circle of influence, propose a solution. This isn’t a lot different than asking questions as noted above; rather than putting your solution up bit by bit, you’re presenting it as a holistic thing, complete with reasoning. It’s really just a change from participant to potential thought leader.

4. State Intent
If the issue lies just outside your circle of responsibility but well within your circle of influence and you believe that you can take or coordinate positive action, make a plan and present it as your intent to the person or people who have direct responsibility for the issue. This provides them insight into your proposed actions complete with a plan and the opportunity to question underlying assumptions, impact of your proposed actions, etc. They might put forth an alternative proposal, suggest modifications to your plan, or instantly agree and ask you to carry out your intent.

5. Take Action And Report Immediately
If the issue’s at the outside edge of your circle of responsibility, you have a solution which you firmly believe in, and there’s an impact to others, take action and report that action immediately. This allows them to see the impact of your action, judge the effectiveness of your solution, and will increase your circle of influence and responsibility over time as you stack up more and more successes.

6. Take Action and Report Periodically
If the issue’s clearly inside your circle of responsibility and there’s not an immediate impact to others, take action and report that action periodically (e.g. at a weekly checkin or in a weekly summary blog post, hint hint ;-) . This again exposes the impact of your action and, with continued successes, will grow your influence and responsibility.

7. Take Action
For those things clearly in the center of your circle of responsibility, just do it. You’re good at it, we already know it (that’s why we made you responsible for it =), and we trust you completely. Go!

_ How do you know where the boundaries of your influence and responsibility lie? _ Your responsibilities should be clear (and if they’re not, you need to sit your boss down and make that so). As to influence, that’s a bit of an art and a bit of intuition; mostly you judge it by thinking about who’s coming to you with what issues, questions, and concerns.

The Values Driving My Value-Driven Life

2008-09-29T17:04:36-07:00

I had the great good fortune to attend FranklinCovey’s FOCUS training with several members of my team last week. One of the exercises they give you is to select 4-9 core values and write positive clarifying statements for each. I just finished up and, since one of my statements is “I openly share my values and invite discussion,” here they are:

These are the values which drive my life. On my best days, this is who I am; on my worst, they still mostly apply (I hope …)

Passion

I actively seek challenges that excite me.

I convey my excitement openly, genuinely, and infectuously.

I urge others to identify their values, tap into their passions, and get going!

Respect

I consider the circumstances of others in my dealings with them.

I assume the best and most constructive motivations on the part of others.

I demonstrate my ignorance of others’ circumstances by asking positive, open-ended questions and listening carefully to the answers.

Integrity

I live by consistent values in a transparent manner, without regard of what’s at stake or who’s watching.

I make decisions based on my values and what’s right for the situation and people involved, not what’s most personally beneficial or most expedient.

Candor

I share my values openly and invite discussion of them.

I express compliments and concerns freely and frequently, in ways that will be heard and is not hurtful to others.

I make a point of complimenting the good work of others. I share compliments I’ve heard with the subject of those compliments.

I share concerns in ways which make them actionable for others.

I hold myself and others accountable for mistakes. When appropriate, I fully acknowledge my role in bad choices and work to make things right.

Leadership

I step into unowned problems and help find them a home.

I constantly seek to improve my vision of the best possible future and articulate it to others in a way which allows them to act independently toward that end.

I look for opportunities to mentor others in growing in line with their personal values.

I help others engage in long-term critical thinking and help them recognize that they are fully empowered to act as a force for positive change.

Compassion

I take opportunities to do small kindnesses and good deeds whenever I can.

I relieve pain wherever possible.

I seek first to understand.

Courage

I do not allow concerns about my personal well-being to prevent me from addressing a problem.

I take positive, direct action even when I dread the work or the outcome.

I seek creative solutions to intimidating problems, pursuing the most positive outcome for all involved.

Reliability

I meet my commitments. I do not take on commitments which I know I can’t meet.

I manage expectations openly and clearly.

Humor

I use humor to defuse tension and relieve pain.

I always look for and share the humorous aspects of challenging or grim situations.

Mozilla's TraceMonkey Avoids Trampling

2008-09-03T10:16:41-07:00

Mozilla’s TraceMonkey JIT Javascript compiler climbed down from the trees last week only to be somewhat lost in the thundering herd of Google’s announcement of the Chrome browser and its accompanying V8 Javascript Virtual Machine.

Based on Andreas Gal’s 2006 HotPathVM paper [PDF, 10pp] as well as his doctoral dissertation on bytecode compilation in VMs [zipped PDF, 138pp], TraceMonkey is offering speedups to rival those of the V8 JsVM. This is very worthy of note as both sides are open source and so can borrow learning and code from each other. Competition is good; open competition is even gooder. ;]

I’m not educated enough to offer in-depth commentary, so here’s a quick collection of the announcements and discussions from those who are:

Mike Shaver’s original TraceMonkey announcement
Brendan Eich’s announcement, complete with benchmarks and metrics galore
Andreas Gal’s discussion of the underpinnings
David Anderson’s in-depth discussion of what trace-based compiling is
InfoQ’s interview with John Resig, jQuery creator and Mozilla’s Javascript Evangelist

I’m starting to wonder if Chrome’s accidental release wasn’t as much a mailroom mistake as it was a tactic to not be an also-ran.

Google Chrome: The Next Big Thing?

2008-09-01T16:13:04-07:00

A friend at work recommended I have a look at Google’s explanation of Google Chrome as illustrated by Scott McCloud [via Google Blogoscoped where they were kind enough to scan the comic they received] and I couldn’t wait. You see, Scott McCloud is the author of two of my favorite books, “ Understanding Comics” and “ Reinventing Comics” … both must-reads if you’re interested in comic history, the vocabulary of comics, or even just in thinking about how we communicate and interact with things on pages.

The Google Chrome strip [via Google Books] is an engaging 38 pages which lightly explains what they’re trying to accomplish and how they’ve gone about it so far. Amusingly, it seems the comic went out a bit earlier than planned, but Google is taking things in stride, with grace and good humor, showing the rest of us how it should be done.

Some of the more interesting choices:

Webkit-based: Google Chrome is based on the Webkit browser engine, which is also used by Safari, Android, and a number of other mobile device browsers. Google’s developers liked it because it’s fast, memory-efficient, embeddable, easy to adopt, and generally keeps things simple.

The V8 Javascript Virtual Machine: written by “the V8 team in Denmark” specifically to address the shortcomings of previous JsVMs in a number of ways, including compilation to machine language and real garbage collection. I’m pretty excited about this, particularly that they’ve made V8 browser-independent.

One process per tab: They’ve implemented this as a protection from crashy apps as well as a way to clearly reveal what’s chewing up resources in your browser. This concept also seems to be foundational to their anonytabs and anti-popup measures; it’s definitely the basis of their sandbox security measures.

The New Tab Experience: When you open a new tab in Google Chrome, they assume you want to go somewhere rather than see your default page (or a blank page, which I’ve used for years to avoid unnecessary page loads). A nice UX touch that shows the depth to which they’re thinking about this.

Gears Inside: It’s no surprise that the Google Chrome browser builds in Google Gears, which has been aiming to bring the Web 2.0 experience even when offline for over a year now. I couldn’t find any estimates of how widely installed Gears has become in the past fourteen months, but I’d guess it’s low at present despite MySpace and Windows Mobile having jumped on board in the past six months.

There’s lots of other good bits in the strip, like the anti-phishing measures page (which makes an analogy between malware and dog poop which couldn’t have been made nearly so excellently in any other medium, imho) and a late appearance by Chris DiBona to sum up the open-sourcedness of it all.

I have some minor concerns about the extra resource consumption and possible need for interprocess communication (IPC) in their model. However, this is probably because my last significant experience with IPC dates back almost a decade to SGIs running IRIX. It was my experience then that IPC is hard to implement cleanly as well as the most frequent source of race conditions. It’s likely times have changed since then. I hope.

The biggest disappointment? When the beta is officially released tomorrow, it’ll be Windows only for now. The official blog post notes that they’re working hard on versions for MacOS and Linux, too (and I suspect they would have liked those to be available on launch day), so when the Google Chrome project page lights up tomorrow, it won’t be for the rest of us quite yet.

But I’ll be eagerly watching, just the same. You don’t see a new browser born every day.

Update: Paul Thurrott’s WinSuperSite has screenshots purported to be Google Chrome [via Blogoscoped]

Reflections on moderated blog comments

2008-08-31T09:35:00-07:00

Ran into another blog today with misleading, subjective assertions about topical content and which moderates all comments. It’s made me realize that, if you intend to blog about current events with any integrity, you absolutely must unmoderate comments and be ready to fight the spam. Otherwise, you’ve risk starting a monolog, not a conversation, and the web has enough monologs.

At best, the practice of moderating comments on opinion pieces is ill-advised and inconsiderate; at worst, it’s self-serving dishonesty — you should just turn off comments entirely so we all know it’s really a monolog.

Reports of the Business Card's Death Are Greatly Exaggerated, or why bzCards != biz cards

2008-08-20T17:40:02-07:00

Caught the article “rmbr launches mobile app to get rid of business cards” on VentureBeat which covers the announcement rmbrME, the new electronic connection tool from rmbr. (note: the rmbr.com site, which launched as a photo organizing site, currently redirects to rmbrme.com) The general idea is to trade vCards over email, IM, SMS, etc. and get rid of those pesky piles of pasteboard. If the ease of use isn’t incentive enough, they’re mixing in the funware idea via contests and leaderboards.

You may have guessed by now, but I think they’ve got the beginnings of a good idea and a poor implementation. I’m particularly amused by rmbr founder Gabe Zichermann’s assertion that the business card’s time has come and gone. Here’s why I think he’s wrong:

Business Cards Are Infinitely Customizable: I can quickly extend or personalize the information provided by my business card*. I can also correct bad information on the spot*, e.g. my title’s changed, or I have a new phone number. And the only device required is any writing instrument.
Business Cards Don’t Require Information Exchange: I don’t have to ask you to give me contact info in order to give you my contact info. This preserves a level of anonymity which should not be undervalued. I wouldn’t give contact info to everyone I’ve accepted a business card from; I’ve later chosen to contact some of those people.
Business Cards Are Trivial To Distribute: I can hand you a business card in just a few seconds, less if I’m already handing them out. I can place a stack of business cards in a tray on a counter to be taken by those interested without having to make any contact at all. I can drop them in the fishbowl of an excellent bakery and cafe in Wilson, NC in hopes of winning some goodies for the next time I’m there. I can hand it to someone who doesn’t have a device. These examples all demonstrate the business card’s continued practical utility.
Business Cards Are Static: They provide a clearly limited, time-sensitive set of information about me. If I change jobs, companies, phone numbers, email, or what-have-you, your data isn’t current anymore. Effectively, the data ages and, in doing so, provides me some additional privacy with regards to those people with whom I’ve not formed a more permanent relationship than the original business card exchange (which, if I have, I likely want to provide a more dynamic link to my contact info). This is also true of a vCard exchange, but unclear with regards to bzCards. In my opinion, the lack of an analog to this aging process is a flaw in social networks which is becoming evident; it’s currently all or nothing.

That said, I applaud the attempt toward a more dynamic contacts list and easier connections. There are some pieces still missing that, imho, are being overlooked by folks headed in that direction, but they’ll come soon enough. And I’m utterly underwhelmed by the idea of competitions to send out the most business cards. It’s quality, folks, not quantity.

* The business card with hand-written phone number comes from a 1997 Mother Jones article. Ironically, that lovely corrected business card image is from the “Why Use It?” page of MyDetails.biz, another digital business card replacement.

Cuzillion: 'cuz there's a zillion pages to check

2008-08-12T23:26:40-07:00

Once in a while I get lucky. Most recently, this means that I was asked to write a blog post for LinkedInabout the tech talk Steve Souders gave for us last week: Even Faster Web Sites. Unfortunately, in the interests of brevity, I couldn’t dive on something incidental to the talk which fascinated me: Cuzillion.

Cuzillion is one of those headslappers Steve seems to produce on a regular basis; it’s a tool to model web pages and experiment with load order and methodology in order to better understand performance bottlenecks and find your way to a solution. He uses it to illustrate examples in his talks of late (as he did in the slides for his talk at LinkedIn), which highlights another feature: the ability to share a precise example among the team seeking a solution. It fascinates me because it’s so straightforward and so obviously useful, yet you can twiddle it to great depth, willfully violating all the well-known best practices, and see clearly why that’s not a good idea. It’s like checkers: you get the basics instantly, and yet there’s so much more there.

After playing with it for a while, it occurred to me that I very much want to say, “Cuzillion, go model the way this web page loads so I can play with it.” When Steve announced Cuzillion, he mentioned that he intends to set up a Google Code repo for Cuzillion; when he gets around to that, it might be fun to spend a little time adding that feature.

Check it.

Stupidity In The Name Of Security

2008-08-12T12:34:37-07:00

Just created an account with WordPress.com and find it amusing that they show a password strength meter to encourage your to choose a cryptographically strong password (which is good), then display your password in large type that’s clearly readable from 12-15 feet away on the unsecured activation page. Oh, and they email it to you, too.

Bruce Schneier would be amused.

Old & Rusty to Shiny & New, Part 1

2008-08-07T09:52:17-07:00

I’m feeling particularly masochistic today, so I’m going to tackle a project I’ve been putting off for a while: moving a Rails 1.2.6 app in our subversion repository into github, then updating it to Rails 2.1.

The first part, getting from svn to github, should be relatively painless according to github’s “Importing From SVN” guide. It is, as long as you have already created your github repo (I hadn’t) and already have git-svn installed (I didn’t). So here’s the expanded tutorial

Create your github repo
Be sure you have git-svn installed. I didn’t, so here’s what I had to do:

sudo port uninstall git-core
sudo port install git-core +svn

I had a bit of trouble with the serf port breaking during build time:

--->  Building serf with target allError: Target org.macports.build returned: shell command " cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_www_serf/work/serf-0.2.0" && make all " returned error 2Command output: /opt/local/share/apr-1/build/libtool --silent --mode=compile /usr/bin/gcc-4.0 -O2 -I/opt/local/include -DDARWIN -DSIGPROCMASK_SETS_THREAD_MASK -no-cpp-precomp -I. -I/opt/local/include/apr-1 -I/opt/local/include/apr-1  -c -o buckets/aggregate_buckets.lo buckets/aggregate_buckets.c && touch buckets/aggregate_buckets.lolibtool: compile: unable to infer tagged configurationlibtool: compile: specify a tag with `--tag'make: *** [buckets/aggregate_buckets.lo] Error 1Error: The following dependencies failed to build: p5-svn-simple subversion-perlbindings serf p5-term-readkeyError: Status 1 encountered during processing.

… so I got all medieval on it, building it manually, then finishing the port install:

BarCamp RDU: Fun and Done

2008-08-03T15:51:37-07:00

Was a lot of fun. Did the Werewolf and Value-Driven Life sessions and got more interest in both than I expected. Also pitched in on Nick Napp’s “10 Ways To Accelerate The Failure Of A Startup” which drew a big crowd and was good, but that was because Nick did all the work and his usual consummately excellent job presenting.

I’ll be sad to miss it next year; we know our family vacation will be earlier and in Orlando to celebrate our 10th anniversary with the one that brought us together, Mickey Mouse. =]

BarCamp RDU in 8 hours!

2008-08-01T21:39:25-07:00

Was checking in to get the details on where and when to be there and, courtesy of a semi-random chat with my friend Evan, decided to propose not one, but two non-technical sessions tomorrow:

Werewolf 101: Rending Your Friends For Fun And Profit!
Taking a Leap of Faith: Letting Your Values Run Your Life, A Story in Three Acts

And then, being the nervous sort, I had to write up some fast notes on what I’ll say in either.

The second one surprised me a bit and, frankly, I’ll be surprised if anyone bites on it. But it’s a topic which I’ve studied to considerable depth and about which I’m passionate … and sometimes people actually want some ideas on how to think about their lives and consider what they want to make of them.

Six hours to sleep, up and shower, and an hour to drive. Wheee!

(p.s. yes, the WW101 class is really just a ploy to play some Werewolf. And maybe get a game together for that night. Would be fun to go back to the Velvet Cloak Inn …)

Let Our Congress Tweet? Who's Stopping Them?

2008-07-14T22:19:35-07:00

A quick follow-up: When Ellen Miller posted to further promote Sunlight Foundation’s Let Our Congress Tweet” petition, she made this statement:

Under the current system, members of Congress are forced to break rules to use new technologies and services to do what their constituents ask of them: connect, listen and be held accountable. So, that YouTube video you saw on a lawmaker’s Web site? Illegal! […]

She also linked to the “Unsolicited Mass Communications Restrictions” section of the House Administration Committee’s Member Handbook, presumably indicating that this was the section that supported her statement.

I read the section (and the bulk of the rest of the handbook for good measure. So I asked for clarification in the comments of her post. What I got was a response from John Wonderlich, who authored the original post, saying in essence that the rules are unclear and they’re seeking clarity. Much softer stance, it seems, than the original hard line that our Representatives were being forced to break the law.

I tried to respond, but it seems the blog isn’t accepting further comments from me. Here’s the comment that wasn’t:

Fair enough. I share your interest in transparency. Having spent the time to read these rules, I also heartily endorse any efforts to update and clarify them.

That said, I take issue with the sensationalist nature of the copy on http://letourcongresstweet.org/ as well as the assertion that it’s outright illegal rather than unclear. Furthering your cause by scaring people with incomplete information and vague possibilities demeans your mission. The means count as much as the end.

I’d also assert that transparency is more about being able to reach in for information than those inside being able to push information out. A commenter at MetaFilter summarizes it well (http://www.metafilter.com/73194/Let-Our-Congress-Tweet#2178791):

“How is Yet Another One Way Communication Medium “sunlight”? Sunlight is when I can see what they are doing, even when they don’t want me to. It’s not a press release.”

Good luck.

Disappointing that they had to resort to unclarity to fix the clarity thing.

Using mysqldump with subqueries

2008-07-14T10:27:46-07:00

I needed to dump some very specific portions of a large MySQL database; some of the data were second-degree relations (aka a Rails has-many-through). mysqldump won’t allow a join in the –where clause, but it will allow a subquery, so I ended up with something that looked a bit like this:

mysqldump --lock-all-tables --where 'id in (select user_id from memberships where site_id = 123)' my_database users

Handy, that. Thanks to Austriaco who was doing this in a similar context.

Let Our Congress Tweet? You've been had.

2008-07-09T23:54:08-07:00

If you’re like me, some large number of the people you follow on Twitter have tweeted something like this today:

Congress, change the rules. Talk to us on our social networks. http://LetOurCongressTweet.org Let our Congress Tweet! #LOCT08

You may be tempted to join in. It seems like a noble cause, and whoever’s trying to keep Congress from tweeting must be evil, bad, and wrong, wrong, wrong. Right?

Maybe not.

John Wonderlich of The Sunlight Foundation — an organization which professes the admirable goal of, among other things, more transparency in government — published a blog post today titled, “Member Use of the Web Reconsidered”. I know this because they also registered and launched the site LetOurCongressTweet.org today, whose content implies that Congress fears the internet and that the quality of discourse with our politicians would be vastly improved by allowing them to use “the communication technologies that we already use” as if they were still writing with quills on parchment and dispatching their missives via a coach-and-four.

Now let’s be clear here. I’m a tree-hugging San Francisco Democrat who firmly believes in transparent government and free speech; of course my first reaction should be to tweet this every hour on the hour. But I’m also a rationalist who likes to understand the issue reasonably fully before I lend my voice, so I clicked through to Wonderlich’s post, which felt low on facts and high on rhetoric. At one point, Wonderlich engages in this reductio ad absurdum argument:

“If Members can use whatever brand of inkpen, or any brand of paper, or buy whatever shoes they want, they should be given radically expanded freedom to use the Internet […]”

He could just as easily have said, “If Members can drive whatever brand of car they like on whatever roads they please, they should be given radically expanded freedom to drive on sidewalks and in public parks” with just as much semantic content. But at least we’re clear that they’ve moved beyond quill and parchment. And they apparently wear shoes! Hooray!

However, he does give good links, one of which led me to section 7 of the Open House Foundation’s analysis of the restrictions on House members’ web usewhere I found (once I learned to read just the parts excerpted from other reports and ignore the opinion bits) some excellent historical context regarding why those regulations came to be (hint: concerns over using public resources and time for personal purposes and members linking from their official sites to commercial and political sites they favored).

With this fresh in my mind, I read Rep. Michael Capuano’s letter. To my eye, the letter recommends the CHA give more freedom than is currently enjoyed, specifically by allowing the posting of video content on non-House.gov servers as long as the spirit of the current CHA rules and regulations are observed. It makes no reference to tightening any restrictions; instead, in precise language that’s refreshingly devoid of posturing, it clearly recommends:

Official content posted on an external domain must be clearly identified as produced by a House office for official purposes, and meet existing content rules and regulations;
To the maximum extent possible, the official content should not be posted on a website or page where it may appear with commercial or political information or any other information not in compliance with the House’s content guidelines.
Any link from a House website to an external site on which the Member video is hosted must contain an exit notice.
CHA, the Office of Web Assistance (OWA), or other designated House entity should maintain a list of external sites that meet whatever requirements are established by CHA.

In contrast, House Republican Leader John Boehner’s “Internet Freedom Alert” to “Online Community & Activists” nearly oozes with emotionally charged phrases as he writes “to alert you to an attack on free speech” and raises the concern that the proposed changes “would prohibit Americans from viewing content published by Members of Congress on websites that are not ‘approved’ by the Committee on House Administration.” Doesn’t mention that it’s a limitation on official content. Doesn’t mention that the limitation already exists under current CHA rules and regulations. Doesn’t mention that the recommendations apply only to video content. Pretty much leaves the facts behind early and never looks back.

Finally, I found my way to Rep. Capuano’s rebuttal of the earlier “Freedom Alert”. Indeed, he makes very clear just what I’d read in his original letter: loosening restrictions, video only, intended to prevent subtle abuses. Stand down, mighty Twitter militia; the country’s freedom is just as safe as it was yesterday. No, really.

I won’t comment on whether I think the current rules are too restrictive; I’ve not read them all, so I can’t say. Instead, I’ll note that Senators Clinton, McCain, and Obama all engaged in many electronic town halls, and are even engaging (by proxy) in a Twitter-based debate. I get emails from Rep. Pete Stark all the time; he comes by at least twice a year to listen to me and my neighbors gripe and answer our questions. I can get updates of action on the floor of the House of Representatives via Twitter. _ I don’t feel like my Congress is limited in any meaningful way from communicating with me _ and, frankly, I’m not sure I want them to be tempted by the distraction that Twitter can offer. Maybe that’s what got Rep. Boehner all fired up in the first place; seems he’s been able to tweet just fine for more than a year and even post not-official-House content on YouTube. He doesn’t seem at all limited, really.

I’ll also observe that I believe we should be properly thoughtful in changing regulations that limit the freedom of action of our elected Representatives. If there’s one thing this whole episode has made abundantly clear to me, it’s that transparency and openness are better served when we reach in for information of our choosing, not when they push out the information they want us to have.

Update: Erin Kotecki Vest (aka queenofspain) at BlogHer got the same vibe and came to much the same conclusion.

Update: I didn’t realize when I wrote this that Tim O’Reilly feels differently. His post adds one unverified fact: he states that it’s currently against the CHA rules to tweet from the House floor. It also reduces the purpose of CHA rules to “decorum.” I find that difficult to swallow.

Update: I’ve found the CHA’s Member Handbook. The relevant sections seem to be pages 44-47; I don’t see anything there which prohibits tweeting from the floor. I’m going to keep looking for a more complete ruleset.

Long Time, No See

2008-03-28T09:03:04-07:00

I’m on the plane to Salt Lake City for Mountain West Ruby Conference and am fairly excited to get the chance to see several people I haven’t seen for months. In all fairness, the only reason I’m not “really excited” is not seeing my family for two hard travel days with very little sleep likely to happen; sorry, Carl, Cobie, and the rest of you, but Beth, Sam, and Cassie have my heart.

I also realized that, since I joined LinkedIn’s Light Engineering Development group*, I’ve been so busy that I haven’t blogged anything significant. While I’ve been silent, my team and I created LinkedIn’s first fully public-facing Rails app, LinkedIn Mobile; I’ve also been thinking and learning about OpenSocial, microformats, digital identity, web services and discoverability in social networks, and scalable web architecture for both Rails and J2EE.

Over the next few weeks, I’m going to start posting a bit more again; if we’re both lucky, there’ll even be some code. =]

*Light as in “lightweight and agile,” though it’s fun to say, “Yeah, we make light. There’s another team responsible for making dark.”

LinkedIn Mobile is Live!

2008-02-25T00:12:19-08:00

It’s been a busy few months since I joined LinkedIn, but some of that work is finally out there for the world to see: LinkedIn Mobile is now in public beta. You can get more details from the LinkedIn Blog.

Today’s going to be an exciting day. =]

If

2008-02-02T22:44:57-08:00

by Rudyard Kipling

If you can keep your head when all about you
Are losing theirs and blaming it on you,
If you can trust yourself when all men doubt you,
But make allowance for their doubting too;
If you can wait and not be tired by waiting,
Or being lied about, don’t deal in lies,
Or being hated, don’t give way to hating,
And yet don’t look too good, nor talk too wise:

If you can dream - and not make dreams your master;
If you can think - and not make thoughts your aim;
If you can meet with Triumph and Disaster
And treat those two impostors just the same;
If you can bear to hear the truth you’ve spoken
Twisted by knaves to make a trap for fools,
Or watch the things you gave your life to broken,
And stoop and build ‘em up with wornout tools:

If you can make one heap of all your winnings
And risk it on one turn of pitch-and-toss,
And lose, and start again at your beginnings
And never breathe a word about your loss;
If you can force your heart and nerve and sinew
To serve your turn long after they are gone,
And so hold on when there is nothing in you
Except the Will which says to them: ‘Hold on!’

If you can talk with crowds and keep your virtue,
Or walk with kings - nor lose the common touch,
If neither foes nor loving friends can hurt you,
If all men count with you, but none too much;
If you can fill the unforgiving minute
With sixty seconds’ worth of distance run -
Yours is the Earth and everything that’s in it,
And - which is more - you’ll be a Man my son!

Data Portability In A Nutshell: You Own Your Data

2008-01-10T17:54:23-08:00

I’ve already gotten pings (don’t you people have other things to do? ;) about this morning’s post on the LinkedIn blog, mostly around a more detailed explanation of the data portability goals with some privacy overtones, so I’ll give a terse summary here:

Some useful jumping-off points:

Data Portability “Action Packs” (aka summaries, executive and other)
DPW Design Goals and Principles
Work To Be Done/How To Contribute
(Incredibly Rough Working Draft) Technical Blueprint

I’m only just getting caught up with what’s gone before on this effort, so I don’t have much more to offer than to say openness is good as long as it clearly respects privacy. It’s that last bit about privacy which I think is getting lost in the thundering herd of press coverage, but that’s what motivated me to get involved. It’s my firm, personal belief that portability must account for privacy; you own your profile and your connection to me, but you don’t get my profile and personal data in the bargain (unless I offer it).

While I don’t agree with Mr. Howlett’s title assertion, I absolutely adore the UK Data Protection Act (hey, Congress, you might have read this prior to the DMCA …) and its intent and I’d expect it can and will be fully embraced in this effort; in that regard, I think Danny Ayers has some ideas heading down the right path, and in fact Robert Scoble had thoughts along these lines just after the debacle that shined a spotlight on data portability in the first place.

Context matters. Context always matters.

Textile, RedCloth, and MonkeyPatching

2007-12-19T00:36:09-08:00

Okay, I need to go through ~2700 articles, some large, some very large, and programmatically decide which of them use an interesting amount of Textile formatting. Don’t ask why; it’s not polite to stare at the unfortunate. Just smile and nod and think about how you’d puzzle this one out.

So I’ve got a bunch of content, some of which may be in Textile format; how can I tell without looking at every piece and making a judgement call? First, I decided to try comparing the text before and after RedCloth processing, but that didn’t work since RedCloth does some whitespace cleanup (among other things) which mean that feeding it a string devoid of Textile markup doesn’t guarantee you’ll get the same string back. A small part of me is offended by this – it breaks the principle of least astonishment – but I’m enough of a situational ethicist to value the good in what it’s doing for me and move on.

I started poking around in RedCloth 3.0.4, found the bits which were perpetually going to alter some of my strings (for the better!) and discovered something wonderful – you can pass an ordered list of rules into RedCloth#to_html, allowing you to tweak it just the way you like it. Do you like using – gasp! – long dashes? The :glyphs_textile rule converts them to an HTML entity (— to be precise) along with a bunch of other conversions. And there are even handy shortcuts to specify all of the textile markup rules (:textile) and all of Markdown (:markdown). Yay! Flexibilty rewards to those who read the code closely!

But why, then, if I do something like:

RedCloth.new('foo -- bar -- baz').to_html(:glyphs_textile, :textile)

Do I get this:

foo - bar - baz

It turns out that the glyph and inline rules are run in the same method, inline, which looks like this:

    def inline( text ) 
        [/^inline_/, /^glyphs_/].each do |meth_re|
            @rules.each do |rule_name|
                method( rule_name ).call( text ) if rule_name.to_s.match( meth_re )
            end
        end
    end

See the problem? Because inline comes first in the ordered list of regular expressions, all inline rules come straight to the head of the line. It should look more like this:

    def inline( text ) 
        @rules.each do |rule_name|
            method( rule_name ).call( text ) if rule_name.to_s.match(/^(inline|glyphs)_/)
        end
    end

… and that’s exactly what I monkey-patched into my app.

Meanwhile, judging from the Textile demo, Textile should handle my double-dash example above; this is also a bug in the Ruby implementation – and all this time I blamed Textile … sorry! I think I’ll send in a patch as penance.

Things I’ve Learned

Making your code flexible is good for other developers.
Being good to other developers means they might send you patches when they find problems.
There’s a hidden feature in Textile which allows you to disable Textile for a chunk of text: <notextile>-not struck-</notextile> -or- ==-also not struck-==
Monkey-patching is a nice way to get something fixed in your app quickly.
It’s handy to be able to silence constant setting warnings when you have to monkey-patch a constant.
Even smart people make bugs (so what hope is there for me?)

How Quickly We Forget

2007-12-14T10:19:22-08:00

I’m working to set up my MacBook Pro to properly allow email from the Terminal command line (e.g. “generate_big_text

Mail -s ‘Your Big Text Has Arrived’ joe@example.com”) and, in the process of running down what needed, found the following line in Alan Kennington’s postfix config pages:

Who on earth is “fnet”??
This must be left over from SuSE 7.1.
This is in the passwd file:

root@dog# fgrep fnet /etc/passwdfnet:x:49:14:FidoNet:/var/spool/fnet:/bin/bash

Who on earth is FidoNet???

From his profile pic, he’s old enough to have at least *heard* of the largest non-commercial pre-Internet message forum on the planet. Really.

Wiimote Makes Your Screen A Canvas

2007-12-13T21:51:47-08:00

Johnny Chung Lee of Carnegie Mellon University decided that the infrared camera inside the Wiimote could be put to interesting uses … and he’s right. This is some of the coolest stuff I’ve seen for a while. If some company out there doesn’t build a $50 kit that converts your monitor to a poor man’s Cintiq they’re missing a really good opportunity.

You might also recall one of his earlier efforts, the $14 DIY Steadicam bracket featured in Make Magazine #1. I’m starting to be a serious fan of his. =]

Ruby Benchmark Shootout, Round 2

2007-12-05T14:13:28-08:00

Back in February, Antonio Cangiano did a fairly impressive run at benchmarking the various Ruby interpreter implementations and comparing the results. He’s just done it again and, just nine months later, the improvements are fairly dramatic.

I’d summarize here but frankly I hate trying to reduce the complexity of a benchmark suite to a pithy conclusion. Instead, I’ll inline one of the several results tables here; this one presents the benchmark execution times in seconds:

Befor you draw any conclusions, go read the rest for yourself.

I hope he’ll provide the raw numbers in some format I can pull into a spreadsheet and play with myself.

Growl for Entourage (via Applescript)

2007-12-04T13:06:28-08:00

So I went looking for a way to connect Microsoft Entourage (yeah, I’m in Exchange land now) with Growl for notifications. In theory, there are many ways, all of them Applescripts. In practice, I couldn’t find one that worked properly … so I grabbed the best one I could find and hacked it from there. I’ve put it in my public subversion repo, so if you need it either ctrl-click and download or use svn to nab it.

Make Terminal.app start with a default set of windows

2007-12-02T11:31:24-08:00

Due to recent circumstances, I’ve moved from my old MacBook Pro with Leopard to a newer, faster MacBook Pro with Tiger … but not soon enough to avoid addiction to the Leopard Terminal.app’s Window Groups feature, which allows you to position, size, and color any number of windows, make them a group, then have that group be what appears every time you start Terminal.app (back in my old 4Dwm days we called this “session management” … glad MacOS is catching up. ;)

Turns out there’s something similar in previous versions of Terminal.app, just not so nicely exposed. Setting up a particular startup configuration is pretty straightforward, though:

Open Terminal.app
Open windows, put ‘em where you want ‘em, change their colors, etc.
Select “Save As…” from the File menu; you’ll see this dialog with different settings:

4. Change the settings to match the dialog above:

Type any name you like in the “Save As:” field
Change “What to save” to “All Windows”
Check the “Open this file when Terminal starts up”

Note that by default it will save this file in ~/Library/Application Support/Terminal, which seems like a fine place for it to me but YMMV.

Me @ LinkedIn.com

2007-11-26T09:10:15-08:00

Today is my first day as the manager of light engineering development at LinkedIn and I’m incredibly excited about it. Why’s that, you might ask? Because LED is the group that’s bringing agile development and Ruby on Rails to LinkedIn.

Who says Rails isn’t ready for the enterprise? ;p

My New Toy: Pentax K100D Super

2007-11-25T22:03:10-08:00

I’ve been staring at the specs for the Nikon D40 (and D40x), Canon Digital Rebel XT/350D (and XTi/400D), and the Pentax K100D (and K100D Super) for over a year. 6 megapixels vs. 10, shake reduction in camera, dust removal, weight, lenses, reputation … I’ve been mulling this one pretty heavily, more than on our last car purchase (Toyota Sienna with all the geek family features like navigation and dvd … a no brainer ;)

I had a target rig of camera, basic 18-55mm lens, zoom lens somewhere between 200-300mm, polarizers for both, rechargeable batteries, a couple of reasonable-sized SD cards, and a bag; my budget was ~$750 for the whole deal. Thus, I waited. And waited. And waited. And it was getting closer and closer to working out as long as I mail ordered from someplace in BFE … then Black Friday came.

Pentax is offering a $100 rebate on the K100D Super kit. Ritz Camera offered a free Quantaray 70-300mm zoom lens. And now I have my camera.

Expect pictures soon. =]

OLPC: You Know, For Kids ... Theirs and Yours.

2007-11-24T23:44:40-08:00

I just tripped over this video:

… which leads to the OLPC Laptop Giving website. The gist of it is that for $400 you can get an XO laptop for your kids … and give one to kids who’ll use it to change the world.

I think Sam’s going to have a very happy holiday.

Cosco Busan, SF Bay, and Kurt Schwehr

2007-11-16T12:24:41-08:00

Sometimes I’m dense. I’ve been reading Kurt Schwehr’s excellent electronic work log (wlog? ;) for months and didn’t even think to share it in the wake of the Cosco Busan allision (a term I wouldn’t have known with Kurt’s help =). If you’re a geek, you should just read his whole blog as it’s rich in very cool stuff you don’t see every day … but if not, here’s his postings to date regarding the whole affair:

Alameda’s beaches haven’t been hit hard (surprising, really) but there’s a lot of cleanup still to be done. Grrr.

Post-Leopard Checklist

2007-11-12T13:35:44-08:00

I’m resettling into my laptop post-Leopard upgrade as a background task today and thought it might be useful (mostly to future-me) to note all the bits I’m twiddling:

Uninstalled and reinstalled SpanningSync to get iCal and GoogleCal talking again
Installed MacPorts 1.5 for 10.5; sudo port selfupdate yielded no changes.
Started installing Ruby gems: rcov uuidtools … I’m sure there are more coming.
Installed QuickSilver for the first time. I’ve been watching folks like Evan speed along using it, so I thought I’d give it a try. I’ll probably take time to watch the 10 minute tutorial on YouTube. Happily just discovered that this eliminates the need for SizzlingKeys to control iTunes. Excellent. Installed a metric buttload of plugins for it. If I can achieve a graceful way of adding tasks and appointments, I’ll be a happy camper.
Kept Twitteriffic 2.1 rather than upgrading to 3.0. I keep hearing grief about 3.0 and ads, which is uninteresting to experience firsthand.
Installed RMagick according to the usual instructions; however, rather than port install all the supporting libs, then compile the sources I just did sudo port install imagemagick which seems to have worked, so the sum of the instructions amounts to:

Install X11, Xcode, and X11 SDK (there are people who don’t? ;)
install MacPorts
sudo port selfupdate && sudo port install imagemagick
sudo gem install rmagick

That’s about it for now.

Update: Round 2 … fight!

Resubscribe to US Holidays and Movie release calendars. (I don’t usually track the DVD releases, but you can if you like)
Partitioned my drive with BootCamp Assistant and upgraded to latests Parallels in preparation for tonight’s Vista installfest. Found it interesting/amusing that Bootcamp suggested that I could install Vista Home although this is against the MS Vista EULA.
Try installing mysql5 from MacPorts and get it to start. Fail miserably. Oh well.
Install the MySQL 5/Intel DMG from MySQL.com
Install /Library/LaunchDaemons/com.mysql.mysqld as root, courtesy of Joannou Ng
sudo launchctl load /Library/LaunchDaemons/com.mysql.mysqld
Install the mysql gem. This was tricky (or I don’t know how it’s supposed to be done); the usual method failed while building the Makefile. In the end I had to read code to discover that I should install like so: sudo gem install – –with-mysql-config … and even so, it wanted to build a Universal bundle and kept failing due to ‘-arch ppc’ compiler and linker flags. Ripped those out of the Makefile and did a make install … and voila.
More gems: parsetree capistrano feedtools fastercsv ferret flexmock mongrel piston rails_analyzer_tool rspec rubygrowl rubyprof sake tzinfo

Still need to work out the Address Book missing thing.

RubyConf 2007 Slide Roundup

2007-11-12T09:28:06-08:00

Haven’t been able to find a definitive source for the slides from RubyConf and, judging from my referrer logs, many others are also looking. Here’s my attempt to gather a reference to them in one place; I’ve just copied the schedule from the RubyConf site and am adding links as I find them. I’ll update this as I find more (and please tell me if you know where to find them =)

UBER-UPDATE: Ryan Davis (aka zenspider) keeps an annual repository of the RubyConf presentations. He hasn’t posted 2007 yet, but here’s links to 2006, 2005, 2004, 2003, and 2002. Very nice.

UPDATES:

20071119: Added link for Luke Kanies’ “Essential Incompleteness” talk
20071117: Added link for Andrea O.K. Wright’s “Building Games with Ruby” talk
20071117: Added link for Ben Bleything’s “Controlling Electronics” talk
20071115: Added link for Charles Nutter’s “JRuby” talk
20071114: Added link for Eric Ivancich’s “Ropes” talk

RubyConf 2007

Charlotte, North Carolina

Friday November 2

Morning

9:00	Welcome
9:30	Marcel Molina	What Makes Code Beautiful? [note: from Ruby Hoedown, anyone seen a more recent deck?]
10:30	Jim Weirich	Advanced Ruby Class Design

Afternoon

Room 1

1:00	Nathaniel Talbott	Why Camping Matters
2:00	Shunichi Shinohara and Kiwamu Kato	Introduction to AP4R, Asynchronous Processing for Ruby
3:00	Nathan Sobo	Treetop: Bringing the Elegance of Ruby to Syntactic Analysis
4:00	Paul Brannan	Avoiding Pitfalls in C Extensions

Room 2

1:00	Ben Bleything	Controlling Electronics with Ruby [note: blog post with slides and other goodies]
2:00	Andrea O.K. Wright	High Art on Top of Low-Level APIs: Building Games with Ruby
3:00	Eric Ivancich	Ropes: An Alternative to Ruby’s Strings
4:00	Ryan Davis	Hurting Code for Fun and Profit

Room 3

Ruby Clinic, an expert-guided afternoon of troubleshooting, fixing, and learning. Bring your questions and problems! (Experts to be announced.)

Evening

Town meeting with Yukihiro “Matz” Matsumoto, creator of Ruby

Saturday November 3

Morning

9:00	John Lam	State of IronRuby
10:00	Charlie Nutter	JRuby: Ruby for the JVM
[note: you may need to ctrl-click/right-click and download the link manually]
11:00	Evan Phoenix	Rubinius 1.0

Afternoon

Room 1

1:00	Ed Borasky	Profiling and Tuning Ruby 1.8
2:00	Phil Hagelberg	Tightening the Feedback Loop
3:00	Eric Hodel	Maximizing Productivity
4:00	Francis Hwang	Conversations vs. Laws: What do we mean when we say Ruby is dynamic?

Room 2

1:00	Luke Kanies	Essential Incompleteness in Program Modeling [note: blog post with link to slides]
2:00	Bruce Williams	Writing Client and Desktop Applications in Ruby
3:00	Michael Neumann	Efficient Ruby to Javascript Compilation and Applications
4:00	Laurent Sansonetti	Mac OS X Loves Ruby

Room 3

Stuart Halloway presents Refactotum: Ruby, an afternoon presentation and workshop on the skills you need to prepare and contribute code to open source projects.

Evening

Keynote by Matz

Sunday, November 4

Morning

9:00	Dr Nic Williams	Use Ruby to Generate More Ruby - RubiGen is Everywhere
10:00	David Chelimsky, Dave Astels	Behaviour Driven Development with RSpec
11:00	Jay Phillips	Next-Gen VoIP Development with Ruby and Adhearsion

Afternoon

Room 1

1:00	William Bereza	Enhancing Embedded Development with Ruby
2:00	Kyle Maxwell	JRuby in the Wild
3:00	Ben Scofield	Cleanliness Is Next to Domain Specificity

Talk Room 2

1:00	Justin Gehtland	Ruby and Identity: OpenID, CAS and Information Card [blog post,not slides]

2:00	Erik Hatcher	solr-ruby: the best open source engine + ruby
3:00	Aaron Bedra	Sploitin’ with Ruby (Point, Click, Root)

Reading my referrer log ...

2007-11-12T09:07:15-08:00

… and if I were a writer in search of material, I’d write about:

rFacebook and Facebooker
Scoring more points playing Werewolf on Facebook
Blocking notifications from the Werewolf app on Facebook
The location of the slides for RubyConf 2007 (okay, I’ll do that one next)
MP3 streaming servers (huh? Oh, yeah, right.)
Infinite sessions in the Facebook API
Mock Ajax
The Superwall API (oh, wait, I did that one)

Such googlebaiting. Tsk.

A Mild Leopard Hangover

2007-11-12T08:51:25-08:00

I installed Leopard last night and, after a few hours, left it doing its TimeMachine thing. It was an overall good experience. Here’s my quick notes:

I did an old-style backup of my disk by rsyncing it to an external drive. This doesn’t make the Migration Assistant a happy camper, so I got the joy of hand-copying the bits I wanted back over. I now know more about the contents of the Library than I ever wanted.
My basic four apps (TextMate, Terminal, Firefox, and Thunderbird) all just picked up where they left off, prefs and all (see above). Yay!
My calendar, tasks, and address book all reverted to some earlier time. I think I missed some Library things. Joy. Well, they’re backed up (and on my Treo and Google Calendar). I’ll work it out this week.
I’ve finally got Spaces! Used to use You Control Desktops, which did one thing I can’t seem to do in Spaces: different wallpaper per space. Someone’s got to have figured this out.
No iPhoto, iDVD, iMovie, or iWeb? I must have missed something (maybe the memo)
Had to be sure I got all my dotfiles over (I’m a pretty shell-oriented kinda guy)
Changing my login shell to tcsh turned out to be long and painful, but it’s done for now … until next time …
My /etc/hosts is pretty heavily hacked, so that had to come, too
MacPorts: copy or reinstall? I opted for the latter. I’ll be refinding things for months, I’m sure, but I want to try to start with the Leopard Ruby I’ve heard such joy about and go from there.

Haven’t yet gone through the fun of BootCamp, though I’m oddly looking forward to it. I’m going to see if that plus Parallels can cope with Vista on a partition yet (and how bloaty it’ll be to have an Office install on the Vista partition).

Rails 101 by Peter Marklund

2007-11-10T22:42:20-08:00

On the off-chance you’re not already reading either Peter Marklund’s blog or Peter Cooper’s excellent RubyInside blog: Peter the Latter points out that Peter the Former has posted his 341 slide (!) deck used to teach a week-long Ruby on Rails basics course. I’ve only skimmed but it looks pretty tasty.

It's Official: I'm a Rails Contributor

2007-11-09T11:14:13-08:00

Okay, big deal. Time to get back to work. (he said with a little grin)

RubyConf, Day 2: Maximizing Productivity

2007-11-03T12:51:59-07:00

Don’t reinvent it if it doesn’t exist; extend what exists.

Don’t be a coding superhero – you don’t need the latest, greatest

Be selfish: fix your own problems first, and don’t even feel like you need to release (but do share that you did something like that somewhere that others might notice)

KISS and YAGNI. Don’t distract yourself.

YAGNITAGAFI: …they aint’ gonna ask for it. Invest in your own features, don’t make guesses about what they’ll want. They’ll tell you. They may even write it themselves.

Test: tell your future self (and others) what you know is necessary and what you think might break.

Autotest to keep yourself from forgetting to test related things.

Keep everything in sight: Eric likes to be able to see his four shells: code, test, autotest, terminal) in quadrants of the screen. Keeps the edge of chat window handy. Uses SSH Keychain and SpiritedAway to keep essential bits on the menubar/tab accessible.

Use Heckle.

Automate everything: setup, migrations, common tasks. Automating setup is good self-defense against support requests.

Know (and love) your editor. [me: it’s your primary tool, of course].

Pair programming [me: Eric pairs with zenspider; I’ve heard them called “peas and carrots” ;]

Ping-pong pairing: write a test, other person fixes and writes next test.

Polish your docs just before release (but rdoc along the way). README == quick start guide, synopsis and feature overview, links to bug tracker and rdoc. [I’d add contact info for developer]

Set up a bugtracking with no anonymous submissions. Don’t keep it in your email or a mailing list. It opens up the debugging and fixing to the community of users.

Have a review partner beat on your code just before release. Local ruby user’s group is a good source. =]

Releasing with Hoe: packaging uploads emails and blogs all taken care of. This makes it possible to release lightly and often. Small changesets. Less complexity, easier to pin down introduction of problems.

Go have a life! =]

Maintenance

Bug trackers allow collaboration and reduce the load on you. Except when they don’t and instead send it to your email (or mailing lists, or blogs, or somewhere). Find them and move them into the bug tracker.

Version your tools meaningfully and consistently. (e.g. v1.2.3 == major minor bugs)

Don’t fear breaking your code into the natural components when it grows too big to be easily dealt with. Find the natural divisions (and dependencies) and pull it apart.

Inspiration

“I don’t owe you” … “right now”. You released it to the world so you agreed to help them, but it’s okay not to drop everything. Be okay with criticism; the critic might dig in and fix it themselves.

Go to a HackNight/HackFest. Talk to people. Help them with their projects; they’ll help you with yours. Go to the pre-/post- dinner/drinks where your laptop isn’t out so you’ll think and talk.

Look to other domains for interesting ideas. autotest was inspired by Smalltalk.

Answer your own annoyances – automate or change your workflow. IMAPCleanse was written to read ruby-talk and flag messages that are interesting, and delete aging, unread messages.

Q: What bugtracker do you recommend?
A: Eric tends to use RubyForge, doesn’t have a favorite.

Q: IMAPCleanse for …?
A: ruby-talk, primarily. Very hard in that volume to keep track of a thread. Then use smart folders to highlight types of messages. [me: would be interesting to see email divided by tabs/sets like for a given mailing list or interest like “New Threads”, “Old Threads I Replied To”, “Old Threads I Didn’t Reply To”]

Q; How much time do you spend communicating with the community? Do you close out the world occasionally?
A: Think week-to-week, not day-to-day.

Q: How do you pair program?
A: Ryan and Eric ping-pong at one screen, one keyboard. Turns into a contest to write the perfect failure test to force the other to spend maximum time coding the fix. “Coding Gang Bang” … give a person a big window on the projector and all they can do is type what they’re told. [me: yike! I don’t wanna be the typist!]

Q: How much time do you spend on open source projects?
A: Sometimes a lot, sometimes a little. Depends on interest. Used to do a lot more but then hit burnout. Balance is very important. Smaller, simpler changes.

Q: What happened to Robot Coop?
A: I left and worked for LimeWire for a while, left in August to decompress (worked on RubyGems) and now working on a top secret project.

Q: You work closely with Ruby Core and have been thinking about productivity … what do you think needs tweaking?
A: Hard to say. The biggest problem is language barrier. ruby-dev’s titles are usually in english, so you can know what they’re talking about but not what they’re saying. ;]
A: (from bug manager on RubyForge project): Would love to see unification on one bug tracker.

Q: You talked about the importance of bugtrackers. The impression is that Ruby Core wants to have bugs/patches submitted to the mailing list, not a tracker.
A: (Matz) It’s easier for me […] because I’m mail based.

Q: Do you have a method size limit?
A: Yes! Roughly one window high (~26 lines). [me: shades of what Drew’s been

Q: Do you lmit how long your rdoc is?
A: If it’s that long, the code should likely be refactored as it’s too complicated and needs that much explanation.

Q; What if you want to document mindset or why you chose a particular implementation?
A; I make it a short note at the end of the line(s) where the implementation choice was made.

Q: What methods do you use to increase coding speed?
A: I type on QWERTY (Ryan: “No, you don’t!”) Oh yeah, what am I saying, I type Dvorak (audience: Yay for pair programming!). Don’t change your keycaps or you won’t learn to touchtype. Autotest shaves a lot of time, ~2-10s per test run.

Q: Using autotest, do you just not save interim (e.g. broken) code?
A: No, autotest has trained me not to. Instead, I write small pieces. Small test. Small method. Saves every 30 seconds or so.

Q: How do you make it easier to pick up what you working on some time ago (a day or more)?
A; I break the work into chunks of a day or less, or spend a day in exploration.

Q: How do you go about becoming a better coder?
A: Pair programming. Test-first. Seeing what other people do and challenging their beliefs. Keep learning.

RubyConf, Day 2: Essential Incompleteness in Program Modeling

2007-11-03T10:50:25-07:00

Ahhhh … sleep makes all better. I came late today, caught the Rubinius part of the plenary and so didn’t have any comfortable way to live blog it … so I didn’t.

Walking our way to Godel’s incompleteness, in a hand-wavy sort of way.

This statement is false.

This statement is true (but not provable as English has no axioms)

Godel theorem requires completeness and consistency

Completeness: all statements are true and provable

Consistency: all statements relate/are built on other statements.

Godel says: Any sufficiently complex system cannot be complete or consistent. No sane person sacrifices consistency, so all (useful) modeled systems will be incomplete.

Godel’s quine: This statement is not provable.

Godel’s theorem is general, not specific to mathematics.

Barrow’s first law: Any Universe simple enough to be understood is too simple to produce a mind able to understand it.

As folks were investigating set theory and real numbers, hoping these were a system which modeled reality completely and consistently. Bertrand Russell theorized “the set of all sets which do not contain themselves” which is paradoxical. This was disturbing, so they worked to define the system (Principia Mathematica) to eliminate the self-referential nature. Three volumes into this, Godel busted them with his theorem.

Applying this to Your Code

Don’t make inifinite stacks.

Three parts: reality, your model of it, and your implementation of your model. You know your model will be incomplete *and you control in what ways it’s incomplete*. Exercise your power consciously.

Applying Ramachandran energy modeling theory: Think of it as potential vs. kinetic energy. Predecision is potential: high energy state, etc.; post-decision starts you in motion, low energy and low tension, high movement. [me: he should have given a one sentence summation or Ramachandran theory at the top of this section; not doing so bogged him down].

He made some observations re: high vs. low energy states, largely urging that you stay in high energy state as long as you can/need. [me: His observations seem to relate to some interpretations of Wilfred Bion’s work re: group psychology re: groups in working state being high tension]

Decisions == energy deltas. Unmade decisions are high energy; made, low. Many unmade is expensive but flexible, most made is cheap but less flexible.

Make decisions as late as possible – you’ll know more. And now we’ve derived agile programming as a good idea.

Asymptotes: lines or curves which approach a line or curve arbitrarily closely without touching.

“You can’t perfectly match reality but you can asymptotically approach it”

Spending time on your model progressively increases the resolution of the model. Move the asymptote closer; don’t take “iteration” to mean “new approach” unless you can’t get closer with your current approach.

Math hurts your brain in new and interesting ways, provides new ways to think about our problems, and is wicked fun. =]

In 1979, Doug Hofstadter blew our minds with “ Godel Escher Bach” … over a quarter century later, the underlying ideas are still engaging a roomful of people. Excellent. Meanwhile, in 2__001, JP Lewis gave a similar talk at SIGGRAPH which was poorly received (and horribly, boringly delivered, despite him being a fundamentally interesting person and having a pile of humorous anecdotes from production). Luke’s presented it in a better context and with richer, more interesting sources.

RubyConf, Day 1, 4p: Hurting Code for Fun & Profit

2007-11-02T13:48:55-07:00

Ahhh, a nap makes all well. Last session of the day. Okay, you want to see these slides and the video once it’s posted by ConFreaks.

“Legacy code is any code you haven’t written.”

“People will press charges if you hurt them. Code won’t.”

“My name is Ryan and I like to hurt code. And I’m OK.”

Ruby Sadism: deriving pleasure from inflicting pain on software, esp. bad software.

Ruby Asceticism: cahracterized by or … [not fast enough, in the slides] … self-discipline

unit test is self discipline, YAGNI is abstention

“Technical debt: skipping design is like borrowing money, refactoring is like repaying principal, interest is paid as time dealing with complexity.” – Ward Cunningham, ComplexityAsDebt, c2.com

Use the Pareto Principle (aka 80/20 rule) to focus efforts

Increase the signal:noise ratio. Big alligators, little alligators, swamp.

“A developer’s obligation is to make sure that the code as written makes the clearest possible statement as to how the solution was understood …” –Ward Cunningham, TechnicalDebt

Introspection Oriented Design: Find the “I” … how did I miss that bug, where did I go wrong.

[HE QUOTED COVEY’S FOUR QUADRANTS! I’M VERY EXCITED! COULD YOU TELL?]

Expand your mind; more knowledge is better.

Write a lot of code. Make a lot of mistakes. Learn.

Push yourself, then push yourself more.

Mentarbator vs. Code Cowboy … find your sweet spot in the middle.

Tools to Help

Continuous integration and Autotest (with the doom guy)

Flog: complexity reporter based on ABC metrics; the higher the score, the harder to test.

Coverage tools: good to spot gaps, but don’t tell you the quality of your tests. 100% coverage with bad tests means nothing

Heckle: test mutator; if your tests pass after heckle, your tests are bad.

Be passionate. If you love your code, hate it, hurt it.

Q: Doesn’t using emacs violate your complexity rules?
A: Hey, I need an operating system, don’t I?

Q: When does [code-generation] violate complexity?
A: When it gives you a bad feeling.

Q: How much does your Evergreen background influence your philosophy?
A: Probably inverted my philosophy influenced the school I was compatible with. Stalker.

Q: You talked about design and about how you should always be thinking about it. Should you design the whole thing in advance or what (Chelimsky)
A: I design iteratively, TDD-based. If you try to solve the whole problem up front, you’re going to have issues that many have written about.

Q: How do you work with people who write bad code?
Hoy: I’ve seen him make somebody cry.
ZS: Who did I make cry?
Hoy: It was in private and I’m not going to say. It wasn’t me.
A: I try not to. Sometimes I’ll throw down the gauntlet and write something competing. Other times I’ll take advantage of ruby’s dynamic nature to just include what I need … I call it duck-raping

Q: WRT tech depth, how do you not revisit code flags?
A: I use a tag system (fix refactor hack doc and others). It lets me keep my brain on track and move on. [me: this is important]

Q: Have you found a way to automate heckle into your build process?
A: I have one lib (the png lib) that’s fully heckle-proof. Last iteration I hooked heckle into autotest; when it went green, it’d invoke heckle. The problem is the geometric expansion of tests.

Q: Given that only one of your libs passes heckle, do you just seek a better score?
A: Heckle shows you the diff of your test and what it changed without making the test fail (e.g. replace an assignment with a nil). A rats nest of code with tight dependencies where it breaks if you change anything will pass Heckle.

Q; are there tools you need as an ascetic that don’t exist yet?
A: Yes. Can’t name one off the top of my head.

Q: Pair programming … useful for you?
A: Ask Eric. (laughs) I find I get a 250% output increase.

Q: What’s your test:code ratio?
A: I don’t measure it often … generally 1.5:1 -ish but with artificial inflation of individual tests per argument, method, etc.

RubyConf, Day 1, Plenary Part 2: Advanced Ruby Class Design

2007-11-02T08:31:08-07:00

No coffee outside. How do they expect me to survive? However, the signs are out for the box lunches and they look to not suck.

Jim starts with a slide of a Flame Maple guitar back. Very beautiful.

Jim’s journey: FORTRAN (Physics), Lisp (in a fortran class taught by Daniel Friedman), C, Modula 2, C++, Eiffel, Java, FORTH, TCL, and Perl.

On learning Lisp: “Yes, these are your father’s parentheses”

Out to show interesting techniques that aren’t Java-esque.

Box 1: Master of Disguise, Rake::FileList

It’s essentially a specialized array which can be init’d with a GLOB, has some extra methods including a custom to_s, and is a lazy evaluator. First cut inherited from Array (very non-Java, this inheriting from a concrete class)

Be sure to call super in subclass initializers.

Overrides Array’s various accessor methods to cause them to all auto-resolve on access. This makes fl + […] work, but doesn’t help with […] + fl … so implement to_ary which is automatically called. Which would be peachy except that inheriting from Array causes this not to be called. So don’t inherit from Array; instead contain one (and cause all your previously overridden accessors to access the contained array)

def [](index)
  resolve unless @resolved
  @items[index]
end

(and he showed an excellent metaprogramming method to handle defining all the accessors)

Box 2: The Art of Doing Nothing, Builder::XmlMarkup

uses method_missing to construct tag accessors, but what about name collisions (e.g. class)?

Inherits from BlankSlate (not Object):

class BlankSlate
instance_methods.each do |name|
undef_method name unless name =~ /^__/
end
end

Kernel is a module included on all Objects, so this is icky:

require ‘blank_slate’

class Kernel
def foo
‘foo’
end
end

And now BlankSlate is not blank. So use method_added hook (in Kernel) to catch when anyone reopens the class, then use that to have BlankSlate hide/remove the method if someone adds something to Kernel. Excellent, until:

module Foo
dev foo
‘foo’
end
end

class Object
include Foo
end

@#%&! Time for the append_features hook and a nearly identical implementation.

Box 3: Parsing without Parsing,

Wouldn’t it be nice to do this:

User.select { |user|
user.name == ‘jim’
}

The Naive Implementation

class User
def self.select(&block)
find(:all).select(&block)
end
end

… perhaps a bit inefficient as you do a full table scan. ;] Might be good to offload the work to the database (but how to avoid SQL?)

class User
def self.select(&block)
cond = block_to_condition(block)
find(:all, :conditions => cond)
end
end

Use ParseTree: a la Ambition

user = TableNode.new(“users”)
result = user.name
puts result.to_s
users.name # <– that’s the SQL fragment to reference the db column

class TableNode < Node
def initialize(table_name)
@table_name = table_name
end

def method_missing(sym, *args, …

[I can’t keep up. We’ll find his slides. I’m going to nap at lunch.]

RubyConf, Day 1, Plenary Part 1: Welcome and Beautiful Code

2007-11-02T07:23:10-07:00

Apologies in advance for quality issues; our crack team of writer just stepped off a red-eye and into the conference. It’s 9:10a and they’re still working on the AV connex. Ah, here we go.

Conference this year is ~15x as big as the first RubyConf. Dave: “For some reason, people keep coming back for more.”

Plenary sessions in the morning, breakouts in the afternoon with Room 3 for work sessions and other non-presentations.

Despite a late start, Dave finished the welcome 10 minutes early. Time for more coffee. And now acquired, having seen Evan (with our awesome Werewolf tees) and Tony Devlin, who say last night’s Werewolfery was good but long. Now up: Marcel, with “What makes code beautiful?” … I’ve seen him do this talk before, but I’m looking forward to seeing how it’s changed over the last few months.

What’s beautiful? Someone volunteered, “My wife!” which drew applause; someone else volunteered, “His wife!” which drew even louder laughter.

Plato asserted that the properties of beuaty are deep and structural which appearances hint at. Discovering what those properties are gives you an understanding of what beauty is.

Marcel’s history: came from literature background, which led him to artificial languages, and here we are. He was early-on interested in looking at ways to make meaning (two sentences can be identical to the last few words and have widely variant meaning) and choosing the properly expressive form for the purpose. Realized over the years of learning and coding that he was really interested in writing what he sees as beautiful code. After evaluating many languages, Ruby rung most true at a subconscious level.

He desired to write software in search of beautiful code, so pursuing code beauty actively gets him to the desired end faster.

Pythagoras, passing a blacksmith, noted that the hammering sound had an appealing quality. As he investigated, he discovered that there were mathematical relationships between the size of the hammer and the pitch of the strike, as well as a relation to musical intervals. This eventually became the basis of the golden ratio.

Thomas Aquinas’ definition of beauty seems the most applicable to software.

Elements of beauty:

Proportion: Marcel invites a roomful of people, largely men, to consider his body. But not in depth. Specifically, his hand and arm. They’re very suited to their purposes. If you modified their proportions in any way, they’d be less suited and thus less beautiful. Two parts of proportion: ratio of parts and economy of size. If everything were 10x, the ratio of parts would be undisturbed, but the economy of size would differ. [I’d argue that there’s something about the system within which the thing exists that determines how economy of scale fits, and that it’s also arguable that form evolves to suit purpose as we cannot efficiently approach purposes for which we are not suited but if they are critical to our survival, we’ll evolve such that we can]

Integrity: Well-suited to its purpose. A crystal hammer might be beautiful, but it would shatter upon use.

Clarity: Something that’s clear and simple (no more complex than required).

Coerce example (in the slides). Lines of code required can be a measure of proportionality; less is more. How long your unit tests take to run is another [me: and perhaps a better one as it’s a real-world metric which affects how often you’ll run your tests … and the more you run your tests the better your code, right?]. Integrity is how well it behaves while running (does it leak memory?). Clarity is how expressive and readable.

[Rick DeNatale comments that his refactor of the coerce example ended up monkey-patching the String class rather than creating a new class. That might be another indicator of simplicity and how Ruby enables it.]

Kent Beck’s “ Smalltalk: Best Practice Patterns” may not express it as being “beautiful” but it expresses it well.

Rousseau equated “good” with “beauty in action”.

Beauty is a moving target. It’s contextual: it’s not your parent’s beauty.

Bohr: “An expert is a person who has made all the mistakes that can be made in a very narrow field.” [me: this is another demonstration of my “make a lot of small mistakes fast” philosophy]

Ruby is optimized for beauty.

Try to imagine better modes of expression. Ugly things reveal mistakes. Make enough ugly things and you’ll make beautiful ones.

Q: At what level can we apply this?
A: All levels, from a line of code to user experience

Q: [Paraphrased: What’s the role of symmetry?]
A: [Paraphrased: Wide and varied]

Do Not Watch This While Eating or Drinking

2007-11-01T09:17:00-07:00

Let my mistake be your forewarning. As I’m settling in to finish a last bit of serious work before heading to RubyConf tonight, my eyes fell on The Escapist’s Zero Punctuation Review of Valve’s Orange Box.

</embed>

This is blow-coffee-out-your-nose-FUNNY. Now I just wish they had a Wii version of Portal; until then I’ll have to settle for this.

A Plethora of Rails Podcasts

2007-10-26T15:55:13-07:00

The guys at Rails Envy (makers of fine Rails spoof commercials) started an audio-only podcast at the beginning of this month. I really like their podcast; it’s short (< 10m, mostly), a bit like a news-snippet fest, and the collective sense of humor that is Jason and Gregg is undiluted. It’s a touch stilted at times (and they call themselves on it) and is sponsored by Pragmatic Studio (which is frequently mentioned in the podcast).

Good stuff, Maynard!

My First Rails Plugin: GenerallyUseful

2007-10-26T12:45:13-07:00

I’m having a Fisher-Price moment today as I publish my first Rails plugin. It’s nothing spectacular, just some rake tasks I like to have around in my apps plus a little module with functions to programmatically list the available Rails environments and provide some (shorter or useful) aliases for the usual suspects. It’ll grow as I think up and/or trip over more interesting rake tasks.

Comments, critiques, and cookies always welcome.

Cornplastic!

2007-10-25T13:15:26-07:00

A year and a half ago I tripped over Ecoproducts’ line of biodegradable “plastic” disposable food containers made out of corn. Today I’ve had my first such container (made by Fabri-Kal, a competitor) handed to me at Comforts, a deli and cafe in downtown San Anselmo (a lovely, busy place, btw). Were it not labeled I wouldn’t have noticed the difference.

Nice.

A(nother) Blast From The Past

2007-10-24T22:14:32-07:00

A very smart friend of mine named Andy posted this video on his Facebook profile:

Besides being engaging, it showed a google search which leads to the w3.org’s archive of the first website ever. Bonus: the first ever categorized list of websites, back when it seemed like we could invent a taxonomy to cover the internet (shades of Yahoo! ;) and … an even earlier list of web servers circa 1992 … thus referential nature of the title of this post.

An API on an API

2007-10-24T20:45:56-07:00

Takin’ it to the next level: RockYou! offers an API to their Super Wall app, allowing your Facebook app to post content on someone’s Super Wall.

Interesting.

Link: [Facebook

Super Wall API](http://apps.facebook.com/superwall/developers.php “Facebook

Super Wall”).

Live Blog: Graphing Social Patterns, Day 3, Opening the Social Graph

2007-10-09T15:24:59-07:00

I took a session off to catch up on some other bits. Back in now.

OpenFaced panel: Opening Up the Social Graph - Tantek Celik (moderator), David Recordon SixApart, Chamath Palihapitiya Facebook, Joseph Smarr Plaxo, Ted Grubb Satisfaction Unlimited

Dave McClure “threw down the gauntlet” to Google and Yahoo: if you want to participate in open standards, show up.

[I’ve followed Joseph a bit as he’s been talking about Pulse, a social network aggregator. While I don’t know Ted, I know several other folks at Satisfaction via Christina. </disclaimer>]

Joseph led with a Pulse demo. Maybe that’s what everyone will do, but I’m not diggin’ it. Damn that boy talks fast.

David Recordin talks a bit about his role as kinda the gluemeister at 6A, standing among and between all the 6A products and looking at smart choices for them all. Also coming to this problem from the vox.com perspective. He also lent the laptop folks are demoing from.

Ted shows how Satisfaction is importing user profiles via microformats. MFs are supported all over the place. [me: we should add them to PS] [CW agrees]. Showed how to get the MF profile data from flickr. It also autosyncs, inheriting changes. [me: this lends an interesting model for connecting disparate graphs and makes the connections valuable]

Chamath got pinged on where he thinks open social graph fits … and almost bit, but stepped back to present FBs take on social graphs. He asserts they believe the world has one social graph. FB has to make sure they make their version of social graph as open as possible (viz they had APIs for social graph data 6 months before FB platform). They intend to get more open over time. Urges to remember that they had to lay a foundation of privacy and respect before people would be comfortable sharing things like name, location, phone, AIM, etc. Scale of this effort is going to be in years, not months.

6A: David Recordon
FB: Chamath Palihapitiya
PX: Joseph Smarr
SU: Ted Grubb

6A: consider any obfuscated object (hashed email) same as the underlying object — if they don’t want to share email, don’t use hashed email as an identifier.

FB: If you can combine a good privacy model with a graph that remains relevant as it grows, you’ll allow sites to remain relevant … unless users decide to choose a different site. In the end, we’re at their mercy. [me: as it should be]

PX: Everybody does better when users are in control of their information. http://opensocialweb.org/ User Bill of Rights [me: shades of Ira Glass’s commentary about Ten Commandments] It’s not a tech problem; it’s more a social, political, privacy problem which is (hopefully) a better problem to have.

MOD: What are the problems for the OSG?

SU: Keeping it as simple and managable for the user
6A: Defining the problem is still the problem. We need to take a step back to see what consumers want. Dave Winer pointed out that they don’t know what OSG means … we need a consistent model, offline and online, which breaks it down in their terms (I have friends, etc.). Embody social conventions of the offline world: if you have the email address of someone in my contacts, you can have their phone number (you’d email me and ask, I’d say yes, etc.)

MOD polls the room for who’s members of multiple nets, decreasing hands as he raises the number of nets.

MOD: How much does this matter?

[missed the answer. was short and felt unsubstantial]

PX: For the people I know on FB (a subset of all) I can get some information. Just like in the real world people want to opt in or out

SU: We want the conversations going inside Satisfaction, but we also want people [companies?] to be able to take those conversations to their own websites. We couldn’t do that as a FB app.

FB: Info expires because we’re custodians of the information. We don’t own it.

6A: 24 hour timeout of data you get via the API for more than 24 hours. [me: be sure to segment and age that data]

FB: If they’re engaged and using you regularly you’ll always have that fresh info. [me: this sort of introduces aging memory to digital info]

Q: What are you all thinking about OpenID?
A: Nearly all do or are getting it running. SU is using MF data but not publishing it?

Q: What about the “Peace and Love” or “Platform Wars” Tim O’Reilly spoke of this morning?
6A: Ask again in a year.
FB: We intend to keep providing what it takes to empower developers to engage and please users.

Q: What does “open” mean?
PX: You’re open when you free your data.
SU: Allowing users to own their data.
6A: Allow users to freely manipulate and remove their info.
FB: Enable users to decide how they interact with people.

Q: FB’s API limits access to some data. That’s at odds with being open.
FB: Our API is evolutionary. We want to keep our users data protected and iterate carefully.
6A: We’re not yet sharing info via microformats as we don’t think we’ve yet set the stage for users to understand that the info could be regarded as their identity and how it could be shared/used.

Q: Mark Zuckerberg said FB would deliver openID and microformats when users want it. Who wants it?
(lots of raised hands and pics)

Q: FB wants one social graph … don’t you have to open it up?
FB: We think there’s one in the world, we’re trying to model it accurately and efficiently.

Q: Bebo and others will likely release APIs which resemble FB. What do you think will be the open app model?
FB: Clearspring (?) already exists and embodies the model of build once publish many, which gets the details out of the way and allows companies to focus on engaging users.
PX: Less true of stitching the OSG across many sites. It’s not a theoretical argument at this point.

MOD: What’s next in terms of supporting OSG?
PX: Looking to help you syndicate your friends list across many sites.
SU: Implementing OpenID and syncing hCard
6A: Build in tools to allow our users to control who they interact with via their content.
FB: We want as much info to flow across the graph and touch everyone in the world.

_[Interesting thought: mark all information with a value of how personal it is to you (think: slider with 3-5 presets, a la “more of this/less of this”)]

Chamath was, I think, the most thoughtful and well-spoken. Joseph obviously has a billion ideas screaming around in his head waiting to get out. No slight to David or Ted implied or intended. And now I’m out, heading to see Joel Spolsky in Emeryville._

TypePad 8 my homewerks.

2007-10-09T12:27:56-07:00

I had a nice live blog going on Jeremiah Robinson’s talk and then I didn’t.

I’ll have to read the slide show and riff on it later.

Grrr.

Live Blog: Graphing Social Patterns, Day 3, Facebook App Design: Elements of Style

2007-10-09T12:14:05-07:00

Time for Jia Shen. He’s a good speaker with dense content delivered well. This should be fun.

Jia Shen of RockYou!

RockYou does many popular FB apps as well as widgets all around; more than 10% penetration in all leading social networks.

FB apps are growing ~7x faster than widgets elsewhere.

Concept and Design

Think mathematically

Gauge target audience: get specific. how big is it? [me: he touches on TAM/SAM/SOM without knowing it]

Model which viral channels you’ll be leaning into: invites? notifications?

Your app name is a viral channel: they add you, it’s in their minifeed, newsfeed, and on left nav.

Why and how will it spread? How will people share it and share within it? Design your callouts from there.

Invites: for each install:

User invites some people
Some pct open invite
Some pct who open invite install it
Some pct who install will uninstall

…must be bigger than 1, more like 2. Over time, this will make growth.

Validate the use case – typical users will use it?
Test call to action – will they click?
Validate viral model – any broken links?

Dev process should by hyper-agile with tons of instrumentation. Measure everything: invites, clickthrus, demos, minifeeds, etc.

Prepare for A/B testing. You’re releasing fast, so you want to be able to change things around radically and quickly.

Site stats (google analytics and quantcast again)

Launch!

Phase it out there: make sure it’s totally working before you make your splash (1-5K users will give good infor for tuning) then go big.

Promote on ad networks for guranteed performance and hitting your target audience. Tune the ad as you see results.

Ad nets on FB: RockYou (of course), SocialMedia, et al

Tuning

Validate the viral model: identify the total multipliers and look for weaknesses

Find new channels: how are users usint it? did you miss something? Integrate with other apps (viz. WereWolves, Vampires, Slayers, etc)

Tune your underlying channels: make sure your notifications aren’t spammy, that you’re targeting well, etc.

Monetizing it

Maintain comfortable growth and keep tuning, seeking more engagement. Encourage people to interact in your apps.

Engagement creates value for you to capture.

Some rehash of what I’ve heard from him before (no surprise) but some new stuff, and interesting to see his talk evolve.

Live Blog: Graphing Social Patterns, Day 3, FB Apps with ASP.NET, Silverlight, and Popfly

2007-10-09T11:50:55-07:00

I’ll be really thin here, splitting attention between session and some code. Also, it’s more demo based, so this is largely my comments and inferences rather than Dan’s talk.

Aiming to enable non-developers with Popfly.

Users are the stars. The “best” users should be treated like rockstars; get to know them and cater to them.

Embrace self-expression. Be personal.

Entertain. Evidently people want to earn points to buy virtual poop (example from a session I missed yesterday)

For devs: Free, client-side file hosting, works with open source AJAX. You can fully control the code.

The general interface is a node-based graph where you connect outputs to inputs; for my Anim/VFX friends, it’s like Shake but with a white background and nodes look like spinny cubes. Ooo. ;p

You connect blocks and can see the results interactively. Good iterative development workflow.

It does dynamic resizing, so making it very small or very big is straightforward.

Building a block for it is pretty straihgtforward; you can clone others as a start, or start from scratch. It’s just javascript underneath.

Also makes it easy for users to skin or mash things. Great engagement.

He shows you can use scriptaculous and, off-hand, says “it’s an open-source library used on Apple.com” … almost like a “see, it’s not part of our empire, really”

All built on free frameworks (silverlight, et al) so cross-platform, cross-browser compatible.

Live Blog: Graphing Social Patterns, Day 3, FB Platform, Apps, Metrics

2007-10-09T10:47:50-07:00

We’re now 50 minutes behind schedule. Woot! Also, Dave McClure “dated himself” by mentioning back when he was programming VB was the rage. Made me chuckle.

Platform, Apps, and Metrics
Dave Morin, Facebook

FB is a tech company.

[I stepped out for a moment]

Concentrated pretty heavily in 18-24, but outside of that bias it’s fairly in line with average demographics.

More than 50% users return daily. They like this; they want to be a utility. 60B pages per day, ~6th most trafficed site. They watch this but try not to pay too much attention to it.

FB doesn’t want to own the social graph, just to map it as efficiently as possible.

They try to focus on allowing people to represent their identity online well.

Then, they try to make it really easy for people to communicate, both 1-to-1 and 1-to-many. It strengthens relationships.

They’ve grown ~3%/week. This has made the social graph stronger. The 43-millionth user gets the same utility from the social graph as the first [me: I’d argue more value, but he’s being modest]

FB is the largest photo sharing site on the net, because it ties into the social graph.

FB events was done in a night. And it’s more than 3x bigger than Evite.

FB Platform: Deep integration, Mass distribution, and New opportunities

What’s it mean to be open when based on user data? [he left this hanging]

They want anyone to be able to build an app as good/bad as theirs.

Three integration points: Profiles, Full Pages (canvas pages), home page

[he walks thru the elements of profiles, etc. stuff you can get better explained in the wiki]

Best practices: (looking at photos)

context is king.

Canvas page: it’s really important to represent the social context around your application. Show relevant, up-to-date content through the lens of your friends.

Content page: If a user adds a comment, they’ve bought in and are more likely to upload content, etc. Make sure you engage users as fractionally as they care to; make the threshold very low.

People aren’t using the minifeed enough. Every time a user acts in some user/group’s direction, put it on the minifeed.

Media in the feed updates (e.g. pix) drive engagement.

FB’s opportunity composed of growth, engagement, and monetization. The platform makes growth and engagement as straightforward as possible.

fbFund was launched to help seed the community with grants without taking equity to help folks “get out from under the man”

How’s it going: over 5K apps, over 100 new apps a day, over 90K developers, 80% of users have already adopted one or more apps.

He made good time; we’re now only 25 minutes behind schedule.

Live Blog: Graphing Social Patterns, Day 3, Keynote

2007-10-09T10:19:03-07:00

A late start, thankfully, as there was horrible traffic this morning.

The Facebook Application Market
Tim O’Reilly (wearing a Zimbra t-shirt)

What they really do at O’Reilly: change the world by spreading information to innovators through books and conferences

Watch the alpha geeks to see where the technology is headed (e.g. pringles can wifi antennas implies ubiquitous network, screen-scraping implies web services, the pedal-powered internet implies alternative energy focus)

Social nets are all hacks working around not *really* knowing who our social network is (my phone knows, but I have to tell the app)

http://radar.oreilly.com/research/reports/

(Tim presents a lot of slide andvery fast … and mostly image based. Good choice)

87% of FB usage is 2% of the apps, which could be very bad news for all the excited app developers (take it with a grain of salt; it’s early in its life … could just be that you’ve got to be wicked viral and really quickly)

FB app uptake is not even a power law distribution (?) nor log-log.

Many apps cluster around the same users, competing for htem.

Top 5 categories of apps ranked by active users: sports, gaming, chat, fashion, fun (~10%)

Top 5 by number of apps in category: fun, messaging, gaming, video, chat

Top 5 with > 100K users: messaging, dating, gaming, video, fun

Top ten are predominantly is pwned by slide, RockYou.

Top apps adoption rates have slowed

A Web 2.0 Refresher

It’s all about openness, sharing, blah, blah, … and peace and love. ;]

Systems that harness network effect to get better as people use them.

Building a collective db: pay folks to do it (early Yahoo), get volunteers (Wikipedia now), architect it into the heart of the (P2P file sharing, Google … and FB)

Harnessing collective intelligence: web2.0 companies are building collective databases whose value follows the number of participants.

Remember: the web has always been about users creating content.

Google’s breakthru was finding meaning in all that original, chaotic UGC and turn it into valuable real-time, user-facing services.

Wasabe (and Mint) can look at how you spend your money as an implicit action aka vote. [me: Interesting thing to aggregate]

What’s the equivalent of intelligence when filtering my feed? example: geni.com knows about family, company directory knows who’s working at O’Reilly, amazon knows who’s written books for O’Reilly.

The phone company knows who your friends are: who’d you call and who called you and how often and such. They won’t do anything with it (c’mon, they’re the phone company). It’d be interesting to get into that data. They’re doing it with email a little; Xobni is doing it a lot.

Web 2.0 is about the future of the Internet Operating System (data subsystems)
Location, id, time, products, media types, relationships, price, tags … what else?

Platforms beat apps every time: Lotus 1-2-3, WordPerfect, Netscape all got beaten by Excel, Word, and IE

Two types of platform: One Ring to Rule Them All (CW: Two keynotes in a row!) or Small Pieces Loosely Joined. The latter seems the robust model; no one will own it all, but we can all join in. 6A is opening the social graph (David Recordon), XFN microformat, etc.

Learn everything you can from your users. If google was a restaurant they’d be weighing every unfinished plate, adjusting portion sizes, etc.

Make your app get better in proportion with the size of your userbase, not just more busy and more crowded. FB is struggling with this a bit.

Make sure you’re getting a steady diet of data. If data is the “intel inside” of web 2.0, what data do you own? Make sure your apps are gathering interesting data.

Offer compelling user-facing services.

Be sure you offer more value than you worry about capturing. Does platform give me and my users control? Users are smart, developers are smart.

What I want from the social graph

Reflect my real social network. 5K address book entries, hundreds of FB friends, mostly folks reaching out to Tim. [me: I bet he says his phone knows who’s really his friend]

Help me manage those contacts (how to reach them, how they like to be reached, how they’re doing) including my friends, acquaintances, and even people I don’t know at all but have heard about and want to follow. The amount of info I need is inversely relational to how close we are.

Help me group people meaningfully.

Relationships are asymmetrical

Fine grained control over what to see vs. not (e.g. show me their photos and blogs, not their tweets, etc)

Discover interesting people

Look at geni.com: a special purpose social network that provides excellent reference material for both people and apps.

On FB’s relationship offerings when acknowledging a friend request with Steve Case, to whom he’d sold a company: “We hooked up? *laughs* Actually, that probably is accurate …”

Freebase/MetaWeb is doing interesting things with a structured taxonomy of info.

What does the person want to display about themselves? What of that do I wish to consume? I’d almost want to rearrange their profile to suit my purpose. [me: this is becoming a theme – profiles are for the consumer/viewer, not the person]

Jaiku was acquired by Google.

Your phone knows where you are. How often have you called someone and found out they’re in a different timezone … or in your timezone [me: gathering intelligence about who’s doing what according to where they are/were, viz. Jaiku founder/developer spending a lot of time in Silicon Valley lately]

“I’m an inventor. I became interested in long term trends because an invention has to make sesne in the world in which it is finished, not he world in which it is started.” — Ray Kurzweil

Think out to the possible future and design for it, not for now.

Q&A

Q: You mentioned you study history, and Microsoft. They dominate a developer community. Are machines the new developer, or is it still people?
A: It’s still the people, though machines are doing a lot more dynamically in the mix. If you look back at the history of MS, … the real test is whether a large number of developers are able to make significant money on the platform and survive. At the end of the day MS absorbed its dev community, and that’s when it went stagnant. … whoever’s the winner of the web 2.0 wars will end up in that same spot.

Q: In regards to the criteria you’d advise for startups, on your own domain vs. on FB, etc.
A: I’m not sure it matters. One of my fundemental beliefs about web 2.0 is that no matter where it is it’s about distribution. Depending on your biz model more distribution may be better. The companies on the top are using all of these well; their fundemental biz is developing relationships with their users thru as many contact points as possible. (Incidentally, people seem to be choosing either web or FB; only about 5% overlap according to someone from iLike). App should be focused on the user and interaction with them. [Be mindful of what data you get vs. waht FB gets]

Q: You seem to be an advocate for the open social graph . […] Isn’t FB going to fight tooth and nail to resist their users moving out into the social graph [difusing their lead]?
A: There’s a change in the dynamics. Google’s a good example: they own some private data, but have the same public website info as any other search engine. You can share an awful lot and still differentiate. If Geni and FB were interoperating, FB could know a lot more useful things. There’s a lot of value in this openness; you’re actually getting more than you give up in most cases. FB has more to gain right now by the openness; it’ll enable them to mine more and more datas. All the trends say that openness is good for you.

Either I’m running slower this morning or Tim talks really fast. Maybe both. Really good keynote, and worth seeing Tim talk. Meanwhile, I find that I miss things in the presentation when I live blog so I’m going to ratchet down the level of detail. Apologies in advance.

A Blast from the Past

2007-10-08T22:36:09-07:00

Tripped over this looking for some other things: A list of all the www.*.com hosts from some indeterminate time long ago January 18, 1996. All 2227 of ‘em.

If I get ‘round to it, I may see which are still up and owned by the same entity.

Prolly not.

One Social Graph To Rule Them All? Not Hardly.

2007-10-08T21:08:12-07:00

On the heels of my corny-but-very-workable galactic identity metaphor, I’ll risk another post on identity to follow up on something said by Reid Hoffman in his keynote today at the Graphing Social Patterns conference. It’s being picked up all over the blogosphere: One Social Graph To Rule Them All?

Reid’s conclusion was no, there won’t be. As he themed his talk, it all comes down to use cases and, at the end of the day, those use cases inform what problems you’re solving and how you go about solving them. Each solution you implement closes some (hopefully less interesting) doors and opens some (hopefully more interesting) ones. Just like with people, the choices you make along the way define who you are. Mercator, Lambert, they served different purposes.

He’s right, and there’s another reason he didn’t really touch on: sometimes folks don’t want the graphs to overlap, or want to actively hide any overlap. One example I’ve been quoting in conversation lately is the S&M Grandma*. She really, really wants those graphs to be separate. Other sides of the world kinda separate. She wants you to think that Greenland is Iceland. Believe it.

(I love blogging; it makes me do research and find great sites like Radical Cartography and read cool things in Wikipedia)

* Rule 1, remember?

Graphing Social Patterns Geek Day Tomorrow!

2007-10-08T19:46:00-07:00

Tomorrow’s the big payoff for me at Graphing Social Patterns — it’s “Apps and Developers Day,” yay! The four talks on either side of lunch look golden (I’m becoming a big fan of Jia Shen of RockYou). I got a quick demo of Microsoft’s Popfly today from Dan Fernandez, so I’m looking forward to hearing him talk about it in more depth tomorrow.

I’ll pick up the live blogging again then, but from what Christina says, it’s really hard so lower your expectations.

The Galaxy of Identity

2007-10-08T18:14:51-07:00

I’ve been learning about, thinking about, and talking about identity and reputation for some time now and only recently hit upon a comfortable metaphor whose analogy seems to extend far enough in all the needed directions. And it’s geeky, so you know I’m pleased.

Consider yourself a planet at the center of your own universe. It is, after all, all about you. You’re located at some unique, identifiable place such that anyone given that location will find you and know it’s you (e.g. OpenID). Arrayed about you are many other celestial bodies, some of which are in tight or loose orbit of you (family, friends, colleagues, etc.), many of which have occasionally tangential orbits (acquaintances); mapping your galaxy is the job of the social graph.

There are also stars of various (possibly variable) brightness, as well as comets which look a lot like a variable star but occasionally pass very close, increasing their brilliance. Not to mention novas, supernovas, black holes, etc. Yeah, I like this metaphor.

But let’s ignore the galaxy for a moment and focus on the planet-that-is-you (I said it’s all about you, right?). We can never see all of you at once; at best, we get half. You’ve got terrain — places you’re higher, deeper, broader, narrower than others. You’ve got mysteries, regions no one’s ever seen or documented*. Weather frequently distorts or obscures visibility. There’s some periodic cycle you observe.

Okay enough metaphorical fun. What can we infer from this model and knowledge of our own globe? Here’s a couple; I’m sure that more will emerge.

1. No perspective is ever complete ; it’s only by observing over time we can begin to get a complete picture. Even then, there’s no map which documents the terrain completely and accurately. That’s usually due to cartographic (e.g. Mercator vs. Polar projection) or political biases (e.g. Greenland is icy, Iceland is green)

2. Certain perspectives are valuable in certain contexts and valueless in others. The WOUB Weather Man is very interested in one particular perspective; his interest wanes inversely to how congruent any other view is.

I’m going to have fun with this metaphor. I hope you’ll enjoy the ride.

* Rule 1: Don’t Visualize. Rule 2: No, Really, DON’T VISUALIZE. If you break the rules, you suffer the consequences.

Live Blog: Graphing Social Patterns, Day 2, Keynote

2007-10-08T10:11:38-07:00

The Business of Facebook Apps
Reid Hoffman, Founder of LinkedIn

Overview

Social Nets and Platforms
* Friendster kicked it off in 2003, conversation around it was very feature oriented
* Now we have FB, MySpace, Ning

Social Nets v. Professional Networks
* Difference in context

One graph to rule them all?

Evolution

Social nets started as a feature

MySpace ecosystem grew out of oversight: didn’t turn off javascript includes

Ning came along with a set of tinkertoys to make your own flavor of social net

FB is the first platform based on a real, broad social graph

Social net takes patterns of social interaction from real life to the web in ways that enable or enrich the experience … can enable applications which can really change people’s lives.

FB Newsfeed == many-to-many messaging

Key differentiator: FB allows you to acquire customers about whom you can know interesting things and message them (and through them, their friends )

FB and LI embody different use cases.
* Search by name on either, you get something that suits the basic context of the app
* Search for “open source expert” on either; FB renders not much, LI is more focused on finding someone of that expertise
* Finding companies for partnerships: not really an FB thing.

The features and general behavior of users in FB vs. LI tends to highlight the different contexts.

SEO: what do you want people to find when they google you?

Trivia: Reid rereads Lord of the Rings every few years.

Q’s: is there only one social graph? is there one which contains all different sorts of relationships?

Reid: there will be multiple social graphs based on “the semantics of the connection”

Geek/Blogger dream: one graph and tag everyone in it.

(is it important to see all graphs integrate?)

A sufficiently large graph can provide the foundation of a successful platform; one big graph not required

Possibilities

Current apps: communications, games, music/pics/movies

Future likely holds iterations of today’s successes

Interested in seeing where friending apps (Top Friends) and honesty apps (Questions, etc) go.

Enabling platform and luring heaps of developers results in getting to that 0.5% of interesting/useful ideas.

With no limits on how many apps get installed and nothing urging users to uninstall apps, you have to work to rise above the noise. Number of installs becomes uninteresting; number of daily users (engagement) is interesting.

FB apps that don’t work so far: biz, politics, money

What’s the second act? How do you know it when you see it? [me: you instrument, iterate, play, and learn]

Economics of FB apps parallels the internet gold rush – cpi installs and advertising inventories/ad network aggregation; these lead to interruptive experiences (crappy ads, etc) and unsupportable models like incented installation or “installations financed by future hope”

Low cost apps with sufficient appeal will survive, as will apps which fit the FB use case which will further define FB’s use case.

Still up in the air: any new use case (anything you try to sell others will give away to acquire customers), major apps

Stickiness and engagement will always matter.

FB is redefining communication patterns based around sharing. Their photo sharing is particular genius with their version of tagging.

What’s the future of discovery/search on the web? Your social network will certainly be a piece of this.

Ecosystems have microcosms; it’s part of the diversity that akes it interesting and sustainable.

Summary

FB will be the entry-point for college-age interesting entrepreneurs

Interesting ties between existing sites and FB app (iLike, Flixter)

Economics will be an issue: keep it on the cheap

Constant newness required for entertainment apps: that’s the current model for engagement

Q&A

Q: Why isn’t LI providing a FB app to [help us merge the two graphs]? (noted that others, CraigsList, et al also don’t)
A: LI will likely provide some FB apps. Biz apps have almost no traction. Bumperstickers, an entertainment app, is one they wrote that has fair traction. Just because you build it doesn’t mean people will jump in (viz profusion of search bars yet most still search from Google or Yahoo). Economics of opportunity development haven’t yet made it possible.

Q: You mentioned a few things that hadn’t worked on FB platform thus far. Do you believe that ecommerce will someday come, or do you believe that social networks will exclude commercial interests?
A: There will be social commerce on the web, but it might be a year or two off. It’ll likely come through sharing (media, etc). Monthly, the most interesting apps on FB are based on communication; the trend will likely continue. There’s always the chance of something new.

Q: Curious with your position as angel and developer in the space, [how would you recommend someone try to build a social media business]? (references to FB-only, rich experience elsewhere, both … ?)
A: Right now, the FB+rich website model seems most interesting. There will be angel things funded on FB, but VCs are looking towards returns which FB apps haven’t yet proven out.

Live Blog: Graphing Social Patterns, Day 1, Geek Track, Late Afternoon

2007-10-07T17:06:12-07:00

Despite being individually wonderful, fresh-popped popcorn and hot coffee don’t go together at all. Bye bye, popcorn.

Jia Shen and Raymond (?) from RockYou! talking about launching and distributing your FB app. RockYou are the folks who brought Werewolves, Vampires, and SuperWall, as well as GlitterText.

Widgets for MySpace, Bebo, Friendster, are all like clothing – self-expressive, no user interaction.

Facebook supports this, but also offers user engagement. It’s like a MMORPG vs. the single-player games of the others.

Networks with engagement gives you ~7x more distribution.

(He showed quantcast rockyou.com web statistics)

RockYou is helping monetize now by offering an ad network on FB, both for use by developers to monetize their apps as well as seeking good matches between their apps and advertisers (e.g. Zombies & Resident Evil movie)

New application setup page walkthru:

Add FB Dev app, create a new app, *make sure it’s checked on to allow folks to add it*. You can limit that to developers only further down the config.

Picking a good name: expressive, short, and catchy.

Callback URL is your server.

Interaction is: FB App page -> your callback URL -> returns text (FBML) -> FB puts it on the page

You can do iframe within FBML, so there’s little advantage to doing iframe instead of FBML.

Developers of your app will be able to see debug messages (as comments in rendered HTML).

Post-Add URL: where you go when someone adds the app. Default is profile page; this is a great opportunity to move them into your app’s design flow – help them take the next step

Post-Remove URL: Isn’t called in user space, but at backend.

Adding left vs. right: right side’s wider, left will likely place you higher.

*BE SURE TO SET DEVELOPER MODE* They have a dev app with 300K users because they forgot. You don’t want to deal with this.

Side Nav URL: set this! users like to nav to the app from there; if you leave it empty, you don’t show up.

Private installation is good for dev mode – don’t announce you’re adding your test/dev apps.

Make sure to have a banner/splash/screenshot for the app when you’re ready to submit it. It shows on the profile page.

insert much configurational fun as we all make dirs on the rockyou dev server. If you want to see what my “app” is up to, it’s here.

Each request sends the following pieces in the session:

fb_sig_session_key: session key (hopefully infinite)
fb_sig_user: uid
fb_sig_added: boolean 1/0, is it added
fb_sig_time: current time, float sec since epoch
fb_sig_friends: comma sep’d list of uids

The wiki is a treasure trove. Keep it open all the time.

Tracking

You need to know you’re tuning your app in the right direction. Here’s some good stats to track:

Infinite Sessions
Page Views
Installs/Uninstalls
Invites Sent/Opened

If you have infinite session key, you can look at their info for as long as you’re installed. Allows you to regen their profile regularly, etc.

Use Google Analytics, Quantcast to capture stats.

Not sure if FBJS allows Google Urchin yet.

Google Analytics doesn’t manage to capture good stats (gets bad total uniques/page views) sometimes if you’re doing “fancy things” where Quantcast is very straightforward, brute-force and rock-solid reliable.

Tracking via post-add URL: how are you getting installed? day-to-day growth is important for sensing how your user community is reacting. Set “next” parameter (&next=…) and send to something that reacts dynamically to that param.

Tracking via post-remove URL: gets called at uninstall; just be sure you’re being pinged by FB and catch the user id. You don’t have to ditch their info, but note that they’ve uninstalled.

Debugging

You must have a dev app and a production app.

FB Dev blog explains how to make test accounts which won’t be removed.

Use cases: They’re looking at your app but haven’t added; they’ve added; they’re looking at someone’s profile and have/don’t have your app, …. Be sure you consider them all.

Look at HTML for debug info; it shows you everything FB set and sent in a comment.

Firebug: use the console to see the AJAX calls to be sure that the request went out right and got a good response.

Test on all browsers, including IE 6 & 7. Use Fiddler (Windows only HTTP debugging proxy) to see all the calls in IE.

Be sure your browser isn’t caching content you expect to be updated.

MS Script Debugger for debugging JS. Allows you to break into JS and step thru.

File logging, email notifications in your app: you’ll see invisible errors and track production issues.

Invites

FB gives multi-selector for picking your friends.

fb:request_form tag gives you an easy

After the user clicks the action in invites (either to do it or skip it) you can take care of them (preview, thank, etc). FB tells you who’s been sent an invite (friend_ids)

Profile FBML is cached at FB, as are images. You can use API to invalidate cache data.

FB:Refs are also cached.

FB carries the bulk of the load. Usually scaling is a DB issue on your side.

Amazon EC2 and S3 are very useful for doing big FB apps.

And then we ran out of time. Jia and Raymond ran a tight session. It kept moving, they actively made sure everyone was moving along well, and I actually got an app running. They kept it really simple, from Hello World to some basic essential concepts, no DB, nothing unnecessary. An ideally structured hands-on workshop.

Live Blog: Graphing Social Patterns, Day 1, Geek Track, Post-Lunch

2007-10-07T14:07:51-07:00

Ahhh, there’s nothing like a conference box lunch. Thankfully so.

The example app Tyler wrote to share: Why Are You Awesome? (and source and db schema ql) (will also be posted at Tyler’s blog)

Store session keys in db and you can use them as unique user ID for stats, being sure the user’s got the latest update of your app, etc.

All the internal FB API calls are built on top of FQL, so writing your app in FQL where possible will give better performance. Ari: “The goal of FQL is not to hide any of the complexity of what we’re going to have to do […] Just to be clear: these queries aren’t just run on a DB server; they’re processed first.”

FQL supports nested selects; they’re resolved as individual ops, innermost to outermost.

Template stuff in a separate file as just FBML. (fbml.inc.php)

Interesting files:

index.php: the main landing for users
logic.inc.php:
fbml.inc.php: template stuff including mock AJAX

Mock AJAX can be done from a link with three properties:
<a clickrewriteid=”<div_to_rewrite>” clickrewriteurl=”<query_url>” cickrewriteform=”<form_from_dom_to_send_data_from>” …>FOO!</a>

Don’t call require_login for mock AJAX calls. Duh.

Mock AJAX returns are quite literally pushed into that div, errors included. Catch exceptions aggressively.

Security-wise, don’t access the fb_sig_{user,profile} . You can use Firebug to check how secure your calls are … if you could use Firebug to change the internals of the call and post as someone else, etc, that’d be bad, now, wouldn’t it? (Tyler did this once to Superwall a bunch of people as Mark Zuckerberg, inviting them to go shopping for NorthFace jackets. Go figger.)

Make a legit FB test account each for your dev, staging, and live apps. Make your dev, staging apps visible only to you or only to a network of folks who’re helping you. You really don’t want your dev server buried by viral adoption.

Tyler pushes with rsync and –exclude fu to keep things configurationally close.

Integrating with your existing app:

Add a column to track their FB id with your user id
Notice cookies (FB, yours, etc) and let that influence your app’s behavior

Profile Action Links (the links beneath profile pix) is more a function of the viewer than the person whose profile is being viewed.

Remember that co-developers can remove you as a developer. Choose wisely. You’re limited to 5 other than you.

All in all, was less of a “Let’s make an app!” and more “Here’s a tour of an app.” That’s fine; I wasn’t looking forward to reliving PHP4. Next: Launching and distribution

Live Blog: Graphing Social Patterns, Day 1, Geek Track, Pre-Lunch

2007-10-07T12:38:57-07:00

I’m at Graphing Social Patterns for the next three days; it’s a conference revolving around the Facebook platform. I’ve watched Giles live-blog and he always does it so well (and I always appreciate it when he does) so here goes:

Presented by R. Tyler Ballance of Slide, Ari Steinberg of Facebook

Useful URLs:

API console and FBML console, which returns to you exactly what your app will see: http://developers.facebook.com/tools.php

They just added bugzilla: http://bugs.developers.facebook.com/
Developer FB App: http://www.facebook.com/developers/
Developer docs and wiki.developers.facebook.com

Subscribe to the platform status feed to keep up with what’s working/broken at the moment: http://www.facebook.com/feeds/api_messages.php

IRC: #facebook @ irc.freenode.net

Use JSON. It’s much more compact compared to XML (~3x in some common cases), keeps bandwidth costs low and interactive response high.

FB is mostly PHP.

Pros & Cons of FB app langs:

PHP:
Pro: super easy to write, officially supported, wide variety of hosting
Con: Only does XML with FB; PHP makes it easy to write crap

Python: (PyFacebook API)
Pro: supports JSON or XML; integrates with Django easily
Con: hosting folks don’t often host mod_python; uses simplejson by default (cjson is 70-80% faster); Python has a steeper learning curve

Ruby: (rFacebook API/lib)
Pro: RoR scales well; RoR encourages clean, readable code; RoR encourages testing; “Very Web 2.0”
Con: Ruby has an even higher learning curve; rFacebook isn’t as RESTful as it could be; “really easy to write bad code in Ruby”, “rFacebook doesn’t quite conform to Ruby coding standards”

.NET: (Mono)
Pro: A variety of existing UserControls; usable from C# or VB.NET; tightly integrated with Visual Studio; supports LINQ (new feature in .NET 3/IIS); *very* well documented (70pp of very detailed docs)
Con: IIS or mod_mono/Apache2 (latter not fully supported); some aspects require .NET 3.5; very “enterprisey” and complicated (separates biz logic from display layer, etc); tightly bound to XML.

Other libs: ColdFusion, Perl, Java, C++

FB Platform is moving so quickly be sure that your chosen client lib can keep up.

Ari: if you know what you’re doing, you can crank out your own FB API lib in The Language Of Your Choice in ~1-2 hrs.

Example app: “Why are you awesome?” written in PHP4, FBML-based, with only one request for info from FB.

The user needs to be called to action – they need to add info, do stuff, manage something, or something else in order to be engaged. Find intriguing social contexts that engage folks.

Work hard to stay within FB’s UI look and feel to increase user comfort. Compare Friends is an excellent example of going far, far away from it.

Make sure to add notifications to increase awareness of your app. Make sure they’re not spammy, or people will mark them as spam and you’ll be restricted to being seen only by your app’s users.

Brevity is the soul of wit.

Use Mock AJAX to provide some cool interactivity.

There’s a limit of 40 notifications per day, 10 user-to-user invites per day. Ari: “Ideally, we’d like to have the limits not exist [and everyone to behave sensibly] … we just have the limits there to keep things from getting out of control.”

FBML-based apps:

Don’t have to fetch tons of info (reduces your load). Free gifts is installed ~5M users, 500-900K daily users, and one server.
Fewer FB API calls == less exposure to bandwidth latency, etc.
Loads about as fast as the rest of FB, very responsive feeling.
FBBYB: FB bugs become your bugs (and users don’t make any distinction)
FB has had issues with unicode
Keep FB’s bugzilla handy all-the-time

iFrame/HTML-based apps:

You’ve got nearly absolute control (some browsers may not allow iFrames to set cookies, so do a page reload)
Easier to port existing web apps
Only constrained to 696 pixel width and terms of service (which you really need to read, particularly about what you can do with data or persistence).
You have to call the FB API (lots), which can increase load on servers
You don’t have access to FBML
You can totally hose the user experience (viz. scrolling iframes, bad self-written controls)

General Tips:

Pick what you’re most comfy with – what you can ship fastest
Subscribe to dev feeds; monitor FB’s changes often.
Use JSON.
You can get an infinite session key using require install (vs. require login).
require add will force the user to add the app before interacting, but they may not opt in just to test things

Let there be lunch and app-hacking afters.

Giles Channels RubyEast

2007-10-03T14:53:26-07:00

If you can’t tell, I’m catching up a bit after a long period of being buried in other things. Found Giles Bowkett’s notes from RubyEast waiting for me in my RSS reader. It’s his usual incredi-dense summary of a day in the life of a RubyEast attendee. Fun to read and inspires many questions.

Link: Giles Bowkett’s Live Notes From RubyEast. [in pastie]

DSLs Are Our Future

2007-10-03T09:10:51-07:00

I have to admit that I’ve been puzzled about the rise of domain-specific languages (DSLs) of late. Not resistant, just curious whether or not it was a fad due to geekish interest or a trend motivated by some unseen force at work. Joel Spolsky’s Strategy Letter VI made it click by relating it to a pattern I’d lived through (and yes, I’m that old):

The C programming language was invented with the explicit goal of making it easy to port applications from one instruction set to another. And it did a fine job, but wasn’t really 100% portable, so we got Java, which was even more portable than C. Mmmhmm.

Right now the big hole in the portability story is — tada! — client-side JavaScript, and especially the DOM in web browsers. […]

What’s going to happen? The winners are going to do what worked at Bell Labs in 1978: build a programming language, like C, that’s portable and efficient. It should compile down to “native” code (native code being JavaScript and DOMs) with different backends for different target platforms, where the compiler writers obsess about performance so you don’t have to. […]

If you don’t read Joel on Software, you should.

Test Helper: assert_not_valid

2007-10-02T11:32:58-07:00

I was a little surprised that Rails offers assert_valid but not its implied converse, assert_not_valid … so I wrote it:

def assert_not_valid(record, message='expected not to be valid, but was valid')
    clean_backtrace do
      assert !record.valid?, message
    end
  end

Test Helper: assert_equal_multiline_text

2007-10-02T11:26:21-07:00

I got tired of poring over the crappy output when comparing two large text blobs for equality fails; the more subtle the difference, the harder it is to spot it by eye. So I wrote this:

  def assert_equal_multiline_text(a, b, label = nil)
    return true if a == b
    label = label ? label + ', line' : 'Line'
    a_lines = a.split(/\n/)
    b_lines = b.split(/\n/)
    (0..([a_lines.length, b_lines.length].min - 1)).each do |i|
      assert_equal a_lines[i], b_lines[i], "#{label} #{i}:"
    end
  end

… where label is just a nice human-readable name for the things being compares. I call it like this:

assert_equal_multiline_text @expected.encoded,   Mailer.create_invoice(payment).encoded,   "Invoice mail"

My eyes are much happier.

Setting Constants While Testing

2007-09-30T20:59:05-07:00

Let’s suppose you’ve got behavior in your app which is dependent on how RAILS_ENV is set and — of course — you want to test that it’s behaving properly. To do this, you need to change the value of RAILS_ENV just before you call that function (and change it back just after). This puzzled me for a while, but I finally figured it out:

class FooTest << Test::Unit::TestCase
  def test_doing_the_thing_with_the_stuff_in_production
    foo = foos(:the_one_to_test)
    orig_rails_env = RAILS_ENV
    suspend_warnings {Foo.const_set(:RAILS_ENV, 'production')}
    assert foo.do_the_thing(:with => the_stuff)
    suspend_warnings {Foo.const_set(:RAILS_ENV, orig_rails_env)}
  end
end

( const_set comes from Ruby’s Module class, which also provides const_get, const_defined?, and const_missing. Nifty!)

That was all well and good for my unit tests, but what about functionals tests? That turned out to be a slight variation on the theme:

class FooControllerTest << Test::Unit::TestCase
  def test_getting_the_thing_with_the_stuff_in_production
    foo = foos(:the_one_to_test)
    orig_rails_env = RAILS_ENV
    suspend_warnings {FooController.const_set(:RAILS_ENV, 'production')}
    get :the_thing_with_the_stuff
    assert :success
    suspend_warnings {FooController.const_set(:RAILS_ENV, orig_rails_env)}
  end
end

You’d think that was enough, but noooo, I’ve even got a function in my ApplicationController which not only behaves differently per RAILS_ENV but also honors a constant (call it FOOISH) which may be set differently (or not set) in each environment. Holy crap! Dealing with that’s a bit painful since I don’t want to hardcode FOOISH’s current per-environment settings in my tests. Then I tripped over a neat bit of code in config/boot.rb and mutated it into this test helper:

def get_var_from_rails_environment(varname, env = RAILS_ENV)
  global_env_file = File.join(RAILS_ROOT, 'config', 'environment.rb')
  specific_env_file = File.join(RAILS_ROOT, 'config', 'environments', "#{env}.rb")
  [specific_env_file, global_env_file].each do |env_file|
    IO.readlines(env_file).grep(/^\s*#{varname}\s*=\s*(['"]?)(.*)\1/)
    break if $2
  end
  $2
end

This lets me make it clean and easy:

def test_getting_the_thing_with_the_stuff_fooishly
  orig_fooish = ApplicationController.const_get(:FOOISH)
  orig_rails_env = ApplicationController.const_get(:RAILS_ENV)
  %w(development staging test production).each do |rails_env|
    fooish = get_var_from_rails_environment('FOOISH', rails_env)
    next unless fooish
    silence_warnings {
      ApplicationController.const_set(:FOOISH, fooish)
      ApplicationController.const_set(:RAILS_ENV, rails_env)
    }

    get :the_thing_with_the_stuff
      
    silence_warnings {
      ApplicationController.const_set(:FOOISH, orig_fooish)
      ApplicationController.const_set(:RAILS_ENV, orig_rails_env)
    }
  end
end

That’s as elegant as I can make it. Not a bad day’s work.

Update: Ryan Bates rightly advises caution in changing RAILS_ENV as you might trigger subtle side effects. Proceed with caution.

Obakucinbidwardston for President

2007-09-29T23:04:51-07:00

Tonight I finally, reluctantly, watched the Democratic Presidential Candidate Debate held a couple of days ago at Dartmouth College. I’m one of those undecided liberals who’s convinced that we’re doing ourselves harm by moving state primaries earlier, thus extending the presidential campaign. I’ve watched it now, and I’m still undecided, though I’ll say that Sen. Dodd, Gov. Richardson, and former Sen. Gravel are off my list. Dodd weaseled, Gravel was outright bizarre, and Richardson came off a bit too idealistic for even me.

While I was at it, I checked out the SelectSmart 2008 Presidential Candidate Selector to see if their results agreed with my gut feelings. Their ratings are expressed as a percentage match between your theoretical ideal candidate’s positions and the stated positions of the potential candidates. Ironically, it does – I’m still largely undecided. All the Republicans were below a 50% match, all the Dems above it, and the five I’m still pondering (Biden, Clinton, Edwards, Kucinich, Obama) within a 10% band.

And I’m still undecided.

CDBaby.com goes back to PHP

2007-09-25T00:59:59-07:00

A bit over two days ago, Derek Sivers posted in the O’Reilly Ruby blog on his experience rewriting CDBaby.com in Ruby on Rails and why he chose to go back to PHP. Without going point-by-point, Derek said that after a hefty time investment, he abandoned reimplementing the website in Rails and took the lessons learned back to create a fresh, lighter-weight PHP and SQL implementation.

As you’d expect, this generated a comment storm filled with at least four types of logical fallacies, often degenerating to name calling and language specification waving. The vast majority missed the point, many of them after Derek made it unequivocally clear: he felt was trying to bend Rails in a way it wasn’t meant to bend, so he stopped. I was going to categorize the comments and poke some fun, but Jamie Flournoy beat me to it (and with a better overall message, to boot). Add to that some earlier thoughts from Chad Fowler and Joel Spolsky on why and how to evolve code rather than rewrite it wholesale, you’ve got a balanced point of view, imho.

Predictably, most commentators got caught up in the X-is-a-better-language-than-not-X argument for nearly any possible value of X. I’m kinda down with Paul Graham on this one: there are better and worse languages. We part ways, however, in that PG feels the continuum is absolute where I believe it’s a matter of perspective, turning on a complex set of factors such as purpose, expertise of the available team, timeline, expected lifecycle, hiring pool, &c.

It’s arguable, though, that since I don’t know Lisp, I lack the proper perspective. I’ll share three interesting quotes from Paul Graham’s quotes page:

“SQL, Lisp, and Haskell are the only programming languages that I’ve seen where one spends more time thinking than typing.” —Philip Greenspun

“We were not out to win over the Lisp programmers; we were after the C++ programmers. We managed to drag a lot of them about halfway to Lisp.” —Guy Steele, Java spec co-author

“Some may say Ruby is a bad rip-off of Lisp or Smalltalk, and I admit that. But it is nicer to ordinary people.” —Matz, LL2

I guess it’s time to learn Lisp …

RailsRumble: Post Mortem Ruminations

2007-09-11T01:01:00-07:00

I’ve been reflecting on the RailsRumble and our less-than-successful entry and thought I’d share some observations while they’re fresh.

Jen’s Long Drive

First, let me frame this up for you. Jenifer Hanen and I jumped into this at the last minute (we were team #140, registered roughly 12 hours before registration closed) in the midst of busy lives and on the basis of a recently-formed friendship but no prior collaborative experience or sense of each others depth, breadth, or style. We live about 400 miles apart, so there aren’t any opportunities for casual communication. I have a wonderful wife and the two best kids on the planet who were very indulgent of what was later termed “my two-day playdate,” (which nearly made me shoot tea out my nose =) but can’t ever be put on full ignore.

We went in with a low-pressure, let’s-have-fun-while-trying-to-make-something attitude. We knew that busy schedules before and during as well as the vagaries of busy lives could mean any number of interruptions and possibly just end the contest early for us.

I got to know Jen better. New collaborations are hard; as much as you may think you’ll just get together and throw down, you’ll invest a lot of (fun!) time telling stories, swapping philosophies, and getting a sense of what the other one means when they say something like, “don’t sweat that other stuff.” What’s the other stuff and why am I not sweating it? Should I have been sweating it earlier? We could have conquered this by getting together sooner, but we were both swamped right up to contest start time and beyond.

I learned more abut CSS and Javascript in a weekend than at the last three conferences I’ve attended. Hanging out with practitioners of an art is, for me, the best way to learn. But then, I like people and conversations.

I have a much deeper (and clearer) understanding of Capistrano. I like the namespaces in Cap 2.0; I also like the big pile of implicit tasks you get.

You never think you’re over-complicating things, even when you are. I was. Ironically, it’s one of the things I most watch for (and kill quick) when managing software teams … and I got a very intimate view of the other side of the coin. Probably the single biggest lesson I learned; I’ll blog about it more later.

Constraints are good. The usual cliches, from Parkinson’s Law to “art is never finished, just abandoned” to “what gets measured gets managed,” are near and dear to my heart. In retrospect, I wish we’d mapped out the time and attached loose deadlines to stages.

It takes a village. There were many of you who helped with advice and well wishes before during and after. Thank you for that.

I also learned a bunch of stuff that I’d already know if I read Agile Web Development with Rails from end to end. I’m going to make time to skim it once through and try to take notes rather than dive in and play with new bits. I’ll post some of the bits here.

Economics & Fishing

2007-09-10T21:20:57-07:00

Anyone ever notice that nearly every reading in economics involves a fishing story?

I think it’s so economists can deduct fishing vacations as research.

RailsRumble: It's Over.

2007-09-09T21:14:06-07:00

The bad: we don’t have a complete app. It’s unfortunate, but true.

The good: I’ve learned an incredible amount in two days, and we’ve got a solid start on what it will be.

I’ll post-mortem the effort in several posts; for now, I’ll observe three things:

Ms. Jen’s a goddess CSS and Javascript.
We worked hard and had fun (and good food and good drink =).
My wife’s the most wonderful, generous woman in the world.

Next year …

RailsRumble: Halfway ...

2007-09-08T20:57:23-07:00

…and we’re chugging along. A slow start – we both decided that next time we’ll get together the day before to do all the brainstorming, model mapping, etc., but we’re well on our way and things will start being visible soon. =]

RailsRumble: Phase 1 Complete

2007-09-08T02:38:23-07:00

Nothing to see yet, really; there are some sketched-in models and a basic controller underneath it all but no views (yet). If you’re looking for visuals, check the RailsRumble Flickr pool.

The server’s up, running, with all the gems we need (so far), including RMagick (which went quietly if really long). Monit is keeping an eye on things, and munin’s running for the fun of it. I know a lot more about the underpinnings of Rails than I did before.

Time for sleep … it’s gonna be a long day tomorrow. =]

RailsRumble.start!

2007-09-07T21:00:00-07:00

Here we go:

ArtLog is available at http://artlog.railsrumble.com/ . We’re going to deploy early and often, so if it doesn’t seem to be up, try again in a minute.
Our development mailing list is up; join via the web or email.
Please look at our app as often as you can/care to and send us comments!

We’ve got until 9p PDT Sunday to do the deed. Wish us luck!

Wheeee!

RailsRumble: Who's Rumbling?

2007-09-07T15:45:47-07:00

I’ve found two teams blogging with the RailsRumble tag:

If you spot any others, let me know. It’s gonna be fun to see what folks are thinking and doing in the next couple of days …

RailsRumble: Choosing the OS for VPS

2007-09-07T13:33:01-07:00

The work begins before the contest: they’ve asked that we set up our VPS before the contest starts to avoid an enormous pileup at 12p EDT. Having watched the excellent smarticast re: setting up a Linode VPS, it’s easy to see that Linode’s got things pretty sweetly automated to handle that part. However, I’ve never been a serious Debian user or admin, so as much as I’d love to just follow the instructions, I have to balance that against my ability to debug esoteric system issues like the ones you can encounter installing RMagick … so I started asking around for advice.

This is where I should note that Linode has a wide array of choices with all your favorites, from CentOS to Ubuntu, as well as some less common (I think) choices like Arch Linux and ActiveGrid. Most of my Linux history splits between Slackware (early days) and RedHat (later and enterprise days), but have heard really good things about Gentoo for its flexibility, stability, and compact footprint. Talking with several smart people, I ended up choosing CentOS 5 as the Rails-production-proven OS which will require the least configuration work and provide the most comfort for me should things head south.

Many, many thanks to Matt of EastMedia, Ezra of EngineYard, and Bradley of RailsMachine for their excellent advice about distros, configuration, and well wishes.

RubyConf Registration is Open

2007-09-07T10:59:40-07:00

Chad’s announced the opening of RubyConf registration (at regonline.com), though the official website hasn’t yet been updated. Register soon as it’s likely to sell out (again). Meanwhile, the Omni Charlotte has a block of rooms available for RubyConf at $109/night. I’d book fast as that’s a stellar rate in the neighborhood and the Omni has zero availability otherwise.

Four Ruby conferences in 6 months feels like an addiction; my wife’s certainly questioning my sanity, and rightly so. Still, the immense amount of quick learning and the fantastic folks (and werewolves) at these conferences make it incredibly worth it.

RailsRumble: Strategy, Phase 1

2007-09-05T23:59:27-07:00

Our first steps:

Configure linode VPS (and huge thanks to Bryan at Smartic.us for the screencast!)
Install and configure monit
Create the app
Commit to svn
Deploy! (and watch for problems)
Add (broken) tests for the gems we’ll need
Install the gems
Run (now successful) tests
Commit to svn repo
Deploy!
Cause some intentional failures to be sure they recover or warn appropriately

We’re only allowed to touch things during the 48 hours from 9p PDT Friday to 9p PDT Sunday and we want to spend most of that time working on the app (or cooking, drinking, playing with kids, sleeping, etc.) so it makes sense to us to get all the infrastructure in place and working first thing.

Meanwhile, we’re looking for folks who can poke, prod, and provide feedback on the app starting ~10a Saturday. We’re figuring that more feedback as we iterate is better. If you volunteer, we’ll ping you each time we deploy; look when you can, offer any feedback you have. We figure the more feedback we get during the development period, the better, so please tell all your friends.

Ready to Rumble!

2007-09-05T22:39:29-07:00

So, this weekend, Ms. Jen Hanen and I are going to participate in the RailsRumble. In the space of 48 hours, starting at 9p PDT, we’re going to build, test – okay: test, build, deploy, and repeat on an app called ArtLog. And we’ll be blogging from now to the end (and beyond) about the experience.

Lucky you. ;]

0-30% Test Coverage? That's Nothing.

2007-09-04T00:47:31-07:00

I’m finding with some amusement that most cases where rcov says something is 30% covered or less, this means all the function definitions (e.g. def foo(bar)) are covered … and nothing else.

Time to put all the excellent knowledge gained at Ruby Hoedown and RailsEdge to good use …

More cool tech from SIGGRAPH

2007-09-01T09:45:47-07:00

A full 360° display using synced projection onto a rapidly spinning mirror:

Seems practical for small displays. I’ll be curious to see if anyone commercializes this; seems like it has obvious medical imaging applications as well as biotech/nanotech.

Thoughts on Software Testing and Rails (a little)

2007-08-31T15:51:05-07:00

Three good reasons to make sure your code has full, solid test coverage:

Tests are a transfer of confidence from you to the next developer down the line. It’s a joy to walk into a codebase that’s fully covered by solid tests. It means I can safely seek to understand only the part of system I need to alter rather than worry over side effects of my changes. It doesn’t guarantee there won’t be any side effects, but if there are, they’re either (a) found immediately due to the tests, (b) new, or (c) harmless.
Tests are distributed ownership. The level of confidence I mentioned above means new developers can hit the ground running and start to take ownership of some portion of the code nearly immediately. In turn, earlier code owners can move on to the next interesting problem rather than taking a new job just to get their hands on a new challenge. Rapid onboarding, continued growth of mature team members – did you hear that, you managers of technologists? It’s in your best interest to demand your codebase is fully covered by solid tests.
Time spent testing saves time. Time is your most valuable asset. Nuff sed.

You knew all this, though. You can’t have been a software engineer for any length of time without the test-early-test-often meme coming into your consciousness. What you did with it depends on the culture you were in at the time. For me, it was 1999 when I got my first SIGSOFT proceedings talking about this “coverage testing” thing which sounds really cool … but the studio where I was working didn’t much value testing and I was the juniormost engineer in a team of nearly 60 developers. They must know better than me, right? It took me a few years to get over that and start writing tests wherever I could. But I’m not bitter. Really.

One of the most incredibly great things about the Rails community is its passionate pursuit of testing. I don’t think I’ve ever seen a development community which values testing so highly, discusses it so frequently, and even goes so far as to make testing cool.

It’s cool to do the right thing? Excellent.

Alice is Lost!

2007-08-31T09:38:48-07:00

Have you seen her?

Ms. Teen South Carolina

2007-08-31T09:16:28-07:00

I’d taken peripheral notice of the Ms.-Teen-South-Carolina-Answers-A-Question meme but hadn’t checked it out because, frankly, I knew what I’d find: we’d all be laughing at a teenage girl who, despite spending an inordinate amount of her young life seeking beauty, poise and grace, blew it when presented with one of the slow-pitched softball questions pageants gently lob to their beauty queens in waiting.

I just watched the video this morning and it was all I expected and more. Frustrating and saddening, seeing how many people are reveling in a teenager’s moment of anguish using it as the basis of fallacious arguments against Southerners (I’m one), Americans (me again), and nearly any other way they can slice this to get at their favorite love-to-hate-‘em group.

Then I saw this video response, courtesy The Real News Junkies:

Right on. STFU, y’all.

The Topology of Social Networks

2007-08-30T22:08:46-07:00

Shel Israel makes some astute observations about the nature of one’s social network. He’s far too social (read: non-geeky) to have discussed it in terms of star vs. mesh topology but his observation made me consider that these are the two ends of the social network spectrum. More star-like and you’re either a focal point or a nexus; more mesh-like and you’re involved in one or more vibrant, potentially overlapping social communities. I’d also posit that full connection (all members are connected to all other members) is the most vibrant a community can possibly be.

This got me wondering if anyone’s thinking about topology of social networks; searching a bit says yes, but in communities I didn’t expect: computer and human virus vector studies, automated agents, and counter-terrorism. In and amongst all that, I tripped over this gem which ties in one of my favorite topics, reputation, into the mix. Cool!

DIY Upper Atmosphere Photography

2007-08-28T18:35:15-07:00

I can’t wait until Sam’s old enough to think this is cool so we can do it together.

I might not wait.

Link: SABLE-3 Balloon Launch.

The Difficulty of Non-Technical Users

2007-08-28T17:41:20-07:00

Two things struck me about Kurt’s post:

It’s an incredible pleasure to watch a domain expert at work.
Every technologist has to deal with the dilemma of non-technical technology users.

Every technologist should have a measure of their market’s median technical ability and target it carefully. And I’d have to say that differently if both of my readers weren’t technical.

Link: Kurt’s Weblog: External Sensors to Verify AIS Ship Data.

Live! Nude! Code!

2007-08-27T15:50:22-07:00

I love a good code review. Looking at real, honest-to-goodness deployed code is one of the best ways to learn the idiom of a language as well as what gets done through the exigencies of circumstance (a nice way to say, “things I done ‘cause I’m ignernt and inna hurry”). Speaking of dumb and rushed, you should attribute any goofs to my misunderstanding or poor expression of what was actually said; I am but an egg.

Watching this session with Chad Fowler and Marcel Molina, Jr. at the helm, I discovered the meta-enjoyment of watching two experts in the idiom of a given language who also have both a frankness and a generosity of spirit review code. They were funny, explicit about their methodology, candid about their ignorance, and inviting of comment from the room. This is how code reviews should be done.

Marcel likes to start with rake stats. (Note: this caused me to trip over flog, an ABC He generally is looking for as much of the code as possible to be driven down into the models (and quoted Jamis Buck’s “Skinny Controller, Fat Model” blog post). I seem to recall he liked at least a 2-to-1 ratio of model-to-controller code, but I could easily be mistaken. We were looking at the code for a real app written by someone in the room that was about 5K lines of code, which Marcel referred to as “pretty big” … which makes me think PublicSquare rates as “really big” with its 10K lines, but it hits the 2:1 ratio well. (I could wish it had 100% coverage, but we’ll get there …)

They started on the biggest model with the thought that it would offer the most opportunity for refactoring. And then the rapid-fire observations started:

Cache sweepers are much like an ActiveRecord Observer but they have access to the controller, too. I seem to recall Chad or Marcel frowning a bit about their use in this case, but I didn’t note why.

TextMate’s TODO bundle allows you to collect the traditional “# TODO”, “# FIXME”, and any others you’d like to define into a printable, saveable list. Color coding included. Nice.

Using composed_of is an excellent way to create objects which can do smart things without requiring a full ActiveRecord class (and db table) behind them. A made-up example; rather than:

student.full_namestudent.reversed_full_name

… do something like this:

student.full_namestudent.full_name.reversed

… and you can then create a method wrapper or delegation on student if you’re a strict interpreter of the Law of Demeter (more below).

Naked finds in a controller (think: @foo = Foo.find_by_bar(…)) are a red flag that some model is missing a needed method. Additionally (and it may be elementary but bears repeating), if you’ll generally follow the CRUD model in your controllers , carving out what you don’t need, you’ll end up with cleaner, more readable apps. This was in line with Chad’s observation that he actually likes to start with the scaffolding and trim back.

At one point a debate on the proper observation of the Law of Demeter broke out, largely expressed in terms of “dot tolerance” (e.g. “I don’t like to see more than one dot – foo.bar(); foo.bar.baz() should b refactored to something like foo.baz_the_bar()” vs. “I’ll usually allow two dots – foo.bar.baz(); anything past that needs refactoring.”) During the discussion, Dan Manges’ post on the Law of Demeter and the response from Alex at Pivotal Labs were mentioned; the comments on the latter are entertaining and informative. Dave Thomas ended the debate an amusing observation: “[The ‘Law of Demeter’ is] the worst named thing in computer science. There is no law. It’s just ‘The Generally Accepted Good Idea of Demeter’.”

Hey Technorati! Get this: therailsedge railsedge railsedge2007

Digging It Up with Dave Thomas

2007-08-27T13:06:06-07:00

mental note: I should liveblog like Giles did. His terse notes are incredibly readable. I don’t know that my additional pass does anything but delay getting it out there.

One of the most, if not the most, information-dense presentation was Dave Thomas’ “Buried Treasures: Hidden Rails Tips.” It’s a bit of a misnomer; some of the best learning I had was about TextMate

PLEASE NOTE: the code below is pseudocode only. It will not run. If you try to run this code, it will likely cause all of your pet’s hair to fall out.

with_scope

Apply scoping conditions, etc., to ActiveRecord finds. Two interesting ways to call this:

with_scope.(:find => {:readonly => true}, :create => {…}, :conditions, :order, :limit )

… or …

LimitNotExceeded = {:conditions => '...', ...}NotExpired = {:conditions => '...', ...}with_scope(LimitNotExceeded) do  with_scope(NotExpired) do    find(:all)  endend

with_options

Great for DRYing up your routes file ( Jamis Buck agrees), but as it’s available on all Ruby objects you could use this to factor out repeated options in any block of code.

returning

This is awesome. Change this crappy construction:

def badly_coded(foo)  status = foo.do_that_thing(:with_that_stuff)  unless status    unless status = foo.do_some_recovery_thing      logger.warning("That didn't go right ...")    end  end  statusend

… to this nifty, readable one:

def shiny_shiny(foo)  returning foo.do_that_thing(:with_that_stuff) do |status|    unless status = foo.do_some_recovery_thing      logger.warning("That didn't go right ...")    end  endend

each_with_index

Yeah, everyone but me knew about this one, but it’s one I’ve wished for in Perl and Python so it’s nice to have tripped over it in passing. [As Drew points out in the comments, Python has a form of this.]

Lookup Constants

I have a cryptic note “Use them for compile time caching” and I don’t have the slides handy so I can’t decrypt right now.

Don’t Call Models in Migrations

IIRC, this came from a quick side discussion: to avoid the race condition where a model column you need to manipulate might not exist in the database quite yet, you should recreate the model object in the migration rather than call it through ActiveRecord. This bit me a while back and I hacked around it with far less grace.

Date Formats

There are some built-ins:

Time.now.to_s(:db)Time.now.to_s(:short)Time.now.to_s(:long)Time.now.to_s(:rfc822)

You can add your own:

Time::DATE_FORMATS.merge!(:timestamp => '%Y%m%d.%H%M%S')Time.now.to_s(:timestamp)=> "20070827.122334"

You should never need strftime again. =]

BTW, (now..then).to_s(:db) makes a proper BETWEEN clause. Sweet!

Partitioning Groups: group_by

>> (1..20).group_by {|n| n % 3}=> {0=>[3, 6, 9, 12, 15, 18], 1=>[1, 4, 7, 10, 13, 16, 19], 2=>[2, 5, 8, 11, 14, 17, 20]}

Related are index_by and in_groups_of.

Groovy String Tricks

str = "Ruby on Rails"str.at(3)str.from()str.to()str.first()str.last()str.each_char {...}%w{Foo Bar Baz}.to_sentence(:connector => 'and also')

generate + svn

Don’t forget to add/remove your generated bits again.

script/generate model foo --svnscript/destroy model foo --svn

Groovy console Tricks

You can create a custom .irbrc file to preserve command history, etc. Dave said he’d post his; until then, here’s Blaine Cook’s, an exploded view of Dr. Nic’s, and a clever method finder to pop into your .irbrc [irbrc file]

Good to remember that you have an instance of the integration testing ApplicationController as “app”. Mike Clark blogged a bit about this last year. Handy.

A headslapper, in retrospect: you can call irb SomeClass or irb my_object to invoke a subsession inside the scope of that class or instantiated object. I think I’ve got that right.

And reinforcing Giles’ observations, reload! is handy to for getting your code changes into your current console, but it doesn’t reload instantiated objects … so it may not net out. Still, good tool to have handy.

_ I’m going to leave out the TextMate shortcuts as I’ve spent too long on this post already. I mentioned this was info-dense, right? ;] _

Hey Technorati! Get this: therailsedge railsedge railsedge2007

Context Aware Image Re-Sizing

2007-08-27T00:29:52-07:00

I was sorry to miss SIGGRAPH this year; now I’m doubly sorry as it seems there was a paper on how to resize and rescale images non-uniformly with the usual artifacts. This is amazingly cool and, as the video demonstrates, has some pretty exceptional uses.

Link: Context Aware Image Re-Sizing [via O’Reilly Radar]

Rails and TDD/BDD

2007-08-24T21:48:44-07:00

I’m going to limit this post to my notes on the excellent learnings from the day of many testing talks at Rails Edge Chicago and follow up with another post for my addle-pated musings thereupon.

First I’d like to observe that Rails benefits heavily by the fact that Railers care passionately about BDD , coverage testing, seeking complete coverage, and beautiful, elegant code. In the past month at various Rails and non-Rails conferences, I’ve heard it hammered over and over again in many ways:

You’ve got to have 100% coverage; anything less is uncertainty.
It’s better to submit a test with no patch than a patch with no test.
I pretty much don’t accept patches without a test.

…and many more. This shared mindset results in a community of developers who not only bring an ethic of constant improvement but also who seek to make code ownership irrelevant. More on that in the Conclusions.

Watching Jim Weirich do BDD was a religious experience. His talk was titled, “Red, Green, Refactor” which is the concise expression of his process: make a failing test, make the test pass, then do it again. He just sat down and started making tests, then writing just enough code to satisfy the test. It was design on the fly, iteratively building up the model in small increments, testing, and again; part of my joy from this was to watch as an excellent programmer exposed his process, even to the point of incorporating feedback from the room. His rake task for testing tracks the number of times it’s been called. If his talk is to be believed, he does it like this every day. And I believe.

Incidentally, Jim writes monster-long test names like “test_portfolio_with_one_stock_has_value_same_as_stock_value” to make his intent instantly readable. His point is that you’re only going to type this name once, so don’t let a desire for “manageable” method names get in the way of expressive, semantic naming.

And he made me realize how cool flexmock is: flex_mock_obj.should_receive(:quote).times(3).and_return(4,30,100)

Justin Gehtland and Stuart Halloway are awesome presenters and it’s easy to see they enjoy working together. They’re BDD practitioners and advocates of it and it was a pleasure to hear them talk.

David Chelimsky is the lead developer of RSpec, a very thoughtful person (as in “he thinks often and well”, though I’m sure he’s also kind =), and a vicious Werewolf waiting to happen. He did a half-hour speed talk about RSpec, which I’d only just heard about three weeks ago when talking to Matt Pelletier and again a couple of weeks ago at Ruby Hoedown. The general idea is you define your spec in a domain-specific language (DSL) which can then be executed to demonstrate that you meet spec. This was already on my list, but David’s moved it up quite a lot. He left me with some research:

RBehave: story level behavior testing
assert_select: find CSS items in page and assert about them.
CrossCheck : java tool that emulates popular browsers to test your Javascript “in browser”
ZenTest/AutoTest : test like mad
Test Notifications via Growl : combined with AutoTest, you can just watch things start working as you fix tests. Cool!

He also gave me reading: Domain Driven Design , by Eric Evans.

Mike Mangino’s talk, “Testing for the Real World”, made some excellent points:

Fast tests get run.
Slow tests don’t.
The only tests that help are the ones that get run.

Actually, his talk seemed like a barrage of excellent points in retrospect; I ended up with a bunch of concise snippets in my notes which I’ll try to reconstitute into full explanations here. Apologies to Mike if I screw it up and libel his abilities; you should assume any errors are in my understanding or ability to explain the concept.

Fixtures don’t scale well for large quantities of data, so if you’re testing against a lot of data you’re best not using them. However, they’re alway useful as templates for test objects; for example, if you’re going to do a bunch of tests which all use the same data with only slight variations (testing each of many validators, &c) it can be handy to get the original object from a fixture and tweak it to suit your test.

Mocking can reveal the ugly spots when testing a controller. For example, if you find you need to mock a phonebook object to return an entry, an entry object to return a name, and a name object to return a last name … you’ve got issues in the code. In fact, it’s a good generalization that hard-to-test code very likely needs refactoring.

Meanwhile, it’s much easier to generate tests for views and controllers if you do them separately. Particularly, Mike recommends mocking helpers in view tests and testing the helpers separately.

Mike finally said the right words to make the difference between stubbing and mocking stick for me (yeah, I may be dumb but I’m slow). If the return value is important to the bit you’re testing, mock it; otherwise, stub it. He also mentioned that filters are hard to stub, but I didn’t catch an explanation so I’ll have to learn why on my own.

One of the best snippets: Tests should be informative first, DRY second. You want people who read your tests to instantly get it, not have to slog through various helper functions to figure out what’s going on.

Oh, and more research: CruiseControl, a continuous build/integration tool based on the work of Martin Fowler and Matthew Foemmel. When I tried to have a look at the Ruby/Rails version is seemed the server was not responding.

Hey, Technorati! Get this: therailsedge railsedge railsedge2007

MIT's 'clutter detector' could cut confusion - MIT News Office

2007-08-22T14:37:32-07:00

This looks promising: some folks at MIT have developed metrics for visual clutter as well as a tool to measure it. Haven’t read the whole paper yet, but

Link: MIT’s ‘clutter detector’ could cut confusion - MIT News Office [via Lifehacker].

Scripting News for 8/22/07 « Scripting News Annex

2007-08-22T13:07:38-07:00

From Dave Winer’s blog Scripting News, How Things Get Better:

[…] Before he had fully settled in I knew it wasn’t going to work. I was able to play out, in my mind, what was about to happen. The software would say nothing to him, so how could he know what to do. I waited and what I predicted did happen. He looked at me and asked “What do I do now?”

That’s where the conversation between product and user begins. […]

Uncommon common sense. Engage your customers in a conversation and listen carefully – they’ll tell you everything you need to know.

Consistency Pays

2007-08-21T16:29:42-07:00

A brief observation: consider well the names you choose for models and their methods. Try very hard to make them intuitive, and not just to a programmer. They’ll haunt you for a long time, especially if you have to expose them via Liquid.

Rails, Oral Tradition, and Getting Started

2007-08-21T15:45:20-07:00

Just occurred to me in the aftermath of last night’s post that capturing some thoughts about the process of getting started with Ruby on Rails. Just a quick sketch as I’m pressed for time, but here it is:

1. Read and Learn

Buy “ Agile Web Development with Rails” (by Dave Thomas, David Heinemeier Hansson, et al) from Pragmatic Programmers and do the whole tutorial.
Buy “ Programming Ruby” (by from Pragmatic Programmers and read at least the “Facets of Ruby” section
Buy “ Rails Recipes” (by Chad Fowler) from Pragmatic Programmers (sensing a theme?) as a great way to see known patterns mapped into Rails
Download and read “ Why’s (Poignant) Guide to Ruby” if only because it’s not only informative but beautiful and a delightful departure from the traditional programming language guide.

Also, Huw Collingbourne has some other book recommendations. As an aside, I like PragProg books because you can get them as as PDFs (though only if you buy direct from them). It’s nice to be able to haul around my entire reference library without breaking my back.

2. Get Connected

Join your local Ruby or Rails meetup.
Attend any Ruby or Rails conferences within easy reach and budget.
Find a mentor, someone you can IM with wonky questions
Hang out in #rubyonrails @ irc.freenode.net
Join the Ruby Talk group on Google Groups (note: this is just an echo of the ruby-talk mailing list)

3. Read and Learn, Redux

Get the code for any open-source Rails project and fix two bugs.
Join the rails-core google group and make a vague attempt to keep up with what they’re talking about.
Start reading good Rails blogs ( Peter Szinek took a novel approach: let Technorati and Alexa tell you which ones are good)

That’s all for now.

Beloit Mindset List for 2011 is out

2007-08-21T10:18:27-07:00

Every year for the last 10, Beloit College publishes their Mindset List. It originated as a way to help their faculty better understand the world view of their incoming freshman class. In the last decade, it’s release has become a national event, with nearly every media outlet sharing some form of the list with its audience.

I could push a lot of spoilers, but I’ll save that for later this week. I’ll just say that as they years march onward, there are fewer and fewer references I don’t immediately get … and more and more which startle me with a reminder of my age.

(non-spoiler: #4 … really? #9 … really?!)

Link: Beloit College Mindset List For The Class of 2011.

Finding Rails {Docs,Plugins,Info,Stuff}

2007-08-20T21:32:59-07:00

I’ve now taken to explaining Ruby on Rails as “the fastest way to get from idea to implementation in a form you can iterate on.” Alliterative, I know, but I’ll blame Mom* and we’ll move on.

One persistent annoyance, though, is understanding where to find certain plugins, how to install them, etc. This is part of the larger issue that Rails is still an oral tradition – that is, if you don’t have someone helping you, you’ll read a lot of source and spend a lot more time puzzling things out than you really wanted to. Or maybe it’s just me. Prolly just me.

There is much (rightful) admiration for the distributed nature of Rails’ plugin tool. Unfortunately, the distributed nature makes it challenging to find some of the plugins which might be mentioned in a chunk of code you’re admiring. Even more confusing, some plugin which existed in one place just a few months ago can move with little fanfare and less explanation.

The one that just bit me is browser_filters. It’s in an existing app I inherited and, when I wanted to install it in a new app I’m working on, the plugin script couldn’t find it. After assuming that I was doing something wrong, a little searching showed it had been moved to legacy with the ambiguous comment “Moved acts_as_taggable, browser_filters, and simply_helpful into legacy – all have either been deprecated or rolled into Rails core already [DHH]”.

Don’t get me wrong. I’m sure that longtime Rails practitioners likely all know which have been deprecated and which promoted into Rails core. I’m also sure that those folks living in Edge Rails (and I hope to be one of them someday) are enjoying the benefits of the non-deprecated plugins right now. I’m growing accustomed to being a half-step behind. I’m catching up.

One could make a full time job of discovering these details and documenting them. It wouldn’t pay well, though, so until it does we’ll make do with blog posts (like this one), hope that enough of them are out there to help the next generation (like me) along.

Okay, I’m over it now. ;]

*Mom == Sally Joan Harmon Meyer, English Composition Teacher and Playful Linguist. Miss you, Mom. World’s (quite) a bit smaller without you.

Changing Feeds

2007-08-17T21:39:50-07:00

I’m twiddling my feeds, so if you’re subscribed to the old one, you’ll want to change over to:

http://feeds.geekdaily.org/geekdailyblog

That’ll go active sometime this weekend (18-19 August).

ThereWolf!

2007-08-11T21:43:56-07:00

We played Werewolf! at the Velvet Cloak Inn where Beth and I had stayed the night before. They gave us The Charter Room for the evening as it’s currently closed for renovations. They were awesome to us, incredibly hospitable; one of the owners, David Smoot, personally worked on the air conditioning issues in the room (it was a toasty 86 when we arrived) and was about to rip the protective cover off the wall when the guy with the key called and got it sorted.

Werewolf Rules and more Pictures of the Villagers.

BTW, staying at the Velvet Cloak was incredibly great. Super hospitable, great rooms poolside, strong-fast wifi … and the Charter Room opens again in three weeks! Sorry I’ll miss it this time around, but I promised Mr. Smoot that we’d drop in next time we’re in town. And we will.

Names as best I remember them (send me more and I’ll fix this; I’m teh suck with last names it seems): Carl Youngblood, Chris Redinger, Joe Martinez, Tony Devlin, Coby Randquist, Rick DiNatale, Ted Behling, Evan Light, Marcel Molina Jr., me, Chad Fowler, Lyle Johnson, Devin Mullins. Not pictured: Ryan Daigle.

Update: So now I’m home, have slept (a little), and am rapidly correcting names as I learn them have added everyone’s names, and see that I left out the part about how astonishingly fun the games were. They were. I’m really looking forward to Rails Edge now. =]

Decadence is ...

2007-07-31T11:37:23-07:00

… Ditch Plains restaurant at 2:30p on a sunny Tuesday, having walked from 2nd & Houston. Half-bottle of El Coto Crianza. Enormous fruit salad. Gunfighter bench seat in the back corner complete with power, wifi, and a view of the folks wandering by. James at the helm of the place, takin’ fabulous care of yours truly.

Truly decadent.

Link: ditch plains [via eleganthack]

Programmers == Stinky Bums

2007-07-30T22:32:51-07:00

From the latest “Python URL!”:

“Why would you want to become a programmer? Programmers smell bad, they have
no social life, they get treated like crap by everyone. They can get paid
pretty well but then they spend all the money on useless electronic junk so
they still live like bums. There is only one reason to be a programmer,
which is that the drive to program burns in you like a fire. But in that
case don’t ask how to become a programmer, because you are already one, so
welcome to the ranks ;-).” - Paul Rubin

Twue. So twue.

Profiling and Optimizing Rails

2007-07-28T16:12:43-07:00

Picture me doing a happy dance. A very happy dance. I’ve only been playing with Rails since April and I’ve been looking for this since the beginning.

Link: How to Profile Your Rails Application and Make Rails Go Vroom!.

CoRaid ATA-over-Ethernet

2007-07-27T10:49:50-07:00

Nice tight mini-SANs. This plus GFS is pretty sweet.

Link: Coraid :: The Linux Storage People.

Sparky: The Alexa Toolbar for Firefox

2007-07-19T14:58:10-07:00

Invaluable tool for quick info about any website you visit.

Link: Alexa - Alexa Toolbar.

LOLCODE ... OMG!!1

2007-07-18T16:07:46-07:00

I can’t stop laughing. Particularly at this example. And this feed.

KTHXBYE.

Link: LOLCODE.

Three Observations on Reverse Engineering

2007-07-17T14:17:49-07:00

I’m working on the tech slides for our presentation deck today. The purpose is to demonstrate that not only do we have something (and we do =), but that it’s easier to buy us than to rebuild us. This got me thinking about what influences how easy or hard it is to reverse engineer a thing. Here’s what struck me as true.

1. It’s straightforward to reverse engineer an existing product feature. The cost of doing so varies in direct proportion to the opacity and complexity of the feature.

2. It’s more difficult to reverse engineer existing workflow as embodied in code (e.g. a pipeline) as it requires knowledge of the pipeline’s foundations as well as an understanding of the nuances of its implementation. Lacking the nuances, you’ll repeat the mistakes of others; lacking the foundations, you’ll go horribly astray. The cost of doing so varies in direct proportion to your knowledge of the process, its basis and nuances, and your experience in working with it and its variations.

3. Most difficult is to reverse engineer strategy, which can be considered as the implicit fundamental knowledge and intent in which a product’s features and workflow are grounded. The cost of doing this varies in direct proportion to your insight into the mind(s) formulating and executing that strategy.

You are not your product. You are not your workflow. You are your ideas.

"Beta" is the Web 2.0 "In Construction Guy"

2007-07-14T13:49:43-07:00

I’m sitting in this year’s Community Next Viral Marketing Conference where Jonathan Abrams (of Socializr) just noted that his web app “is gamma … which is just like beta, but sillier.” He’s right; the beta badge has become the “in construction” of Web 2.0.

The Next Big Thing

2007-07-09T17:47:13-07:00

If you’ve ever talked with Shel Israel (co-author of Naked Conversations), you’ve likely watched him strew the conversation with priceless observations pithily dressed so they’ll cling to your grey matter like Paris Hilton somehow clings to her celebrity. I don’t know that he’s read “Made To Stick,” but he’s certainly tuned in to its message.

The one he dropped on the table today which made my brain explode, quoted as closely as I can manage:

“Computers lack common sense. […] Integrating common sense with computing power [is the next big thing].”

This immediately rings true. The rise of social media, of crowdsourcing, of online communities are all based on finding ways to integrate common sense into the system. It’s the thing people continually bring to the party that no amount of engineering will ever conquer (with apologies to Jeff Hawkins, many of whose aims I share).

All the better that PublicSquare fully embraces this; it’s how I know we’re on the right track.

Models < Drops

2007-07-06T14:51:51-07:00

A member of the Liquid Templates group contributes an 18-line extension to Liquid that lets you specify which attributes, methods, and associations to expose to liquid templates, like so:

class SomeClass     # could be an AR model   liquid_methods :attrA, :methB, :associationC, ...  ...end

Just for ducks, here’s the entirety of his extension code, sans comments:

class Module  def liquid_methods(*allowed_methods)    drop_class = eval "class #{self.to_s}::LiquidDropClass < Liquid::Drop; self; end"    define_method :to_liquid do      drop_class.new(self)    end    drop_class.class_eval do      allowed_methods.each do |sym|        define_method sym do          @object.send sym        end      end      def initialize(object)        @object = object      end    end  endend

Veddy nize.

Link: [extension for drops - Liquid Templates

Google Groups](http://groups.google.com/group/liquid-templates/browse_thread/thread/bf48cfebee9fafd9?hl=en “extension for drops - Liquid Templates

Google Groups”).

Confessions of a RambleGeek

2007-07-06T01:16:52-07:00

Random thoughts after a month of being a caffeinated technomad:

An iPod at appropriate volume and a ballcap at the right down-angle work as a limited sensory deprivation chamber. That is, until the baby shows up.

The Alameda Public Library is heaven to a mobile office person. The study rooms are particularly lovely.

I used to like StumbleUpon a lot. I still do. But I’m way deeper into del.icio.us than I ever was before.

An EVDO rev A card, while it sucks battery, lets you choose the less crowded cafe with no wifi but good coffee/art/treats.

Restaurants that close between lunch and dinner yet still let you camp out in their shaded outdoor seating are things of rare beauty. Treasure them by not abusing their kindness.

A decent computer microphone/headset combo and Skype put together are pretty dang awesome. Even better if you pony up for SkypePro, SkypeIn, and SkypeUltimate – $78/year for all the incoming, outgoing, and portability you need.

Not Living In A Cave ...

2007-07-05T12:47:33-07:00

… just refuse to post about the tech of the moment.

You know the one.

From Apple.

Yeah, that.

Not gonna give it more press. It’s got enough.

But I desperately want one. =]

Boobs, Injuries, & Dr. Pepper

2007-06-27T11:37:24-07:00

Okay, this is a geek blog. And a lame one as I so rarely link to someone else’s blog. Come to think of it, this is pretty much masturbation to techporn in prose for all that I care about readership. I’m no linkwhore.

So you know it’s serious if I say, YOU MUST READ THIS BLOG. Particularly, THIS ENTRY.

You’ll blow milk out your nose. And she ain’t gonna clean it up for you.

Link: Boobs, Injuries, & Dr. Pepper .

The Circle of (typographic) Life

2007-06-27T10:07:25-07:00

I find it amusing that, in the early-to-mid-eighties we put a lot of typesetters out of work as desktop publishing eliminated the need to memorize cryptic typographic encodings.

http://hobix.com/textile/

We’ve come full circle.

SW Dev: Back to the Mac

2007-06-15T23:32:18-07:00

For the last decade, my software development has been a UNIXy thing, filled with xemacs, gcc, xxdiff, and gdb. In my new role, I’m becoming MacOS-centric again and discovering amazing new tools with all the wonderful Macish je ne sais quoi as well as all the bits I miss from my former environment.

For me, the job of software development is all about four things:

Design
Code
Test & Debug
Release

Or, to break them down from a tools perspective:

Scribble (& Erase & Scribble Again): whiteboards and notepads
Type (poorly ;p): text editor
Execute, Tweak, Repeat: debugger, diff/merge tool
Copy (with a flourish): deployment tools

Whiteboards and notepads are universal and ubiquitous (though as a member of the cafe mobile workforce, I’m still in search of a good whiteboard analog). Debugger choice is mostly guided by the language you’re using. Deployment tools are largely a function of the platform upon which your tools run. That leaves text editor and diff/merge tool as the highly personal choices which shape your development experience.

Text editor is obvious; every experienced coder knows with certainty exactly which is the best text editor that should be used by everyone has a (usually strong) preference. For me, it was XEmacs with a carefully crafted set of customizations painstakingly preserved from one job to the next — familiarity bred, in this case, great speed. Luckily, the Rails crowd made it clear from the start that the only choice was TextMate; this was validated by a few weeks familiarization. Awesome tool. Almost as good as XEmacs. =]

Diff/merge tools were less obvious. It was only this week I found myself in need of one and was delighted to see my old friend xxdiff available as a MacOS app; that is, until I discovered that it consumes CPU cycles while idle and has a focus battle with Virtue Desktop (that’s likely Virtue’s fault; I wish it weren’t moribund, but Leopard’s Spaces killed it without even being delivered).

A bit of poking around, though, and I discovered DiffMerge. It lacks a bunch of the command line flags I’m used to, but it makes up for it (mostly) with excellent polish and good customization twiddles. I’m falling in like with it.

But I’ll throw it over for xxdiff in a heartbeat once the CPU issue is fixed and Leopard’s out. =]

Link: TextMate — The Missing Editor for Mac OS X.

Link: Free Diff Tool: SourceGear DiffMerge.

More Tech, Less Smarts

2007-06-14T23:33:02-07:00

A little over a year ago, we bought a new family car (minivan, really, but we’re still in denial =) which includes a navigation system. It’s pretty spiffy, but it’s led me to discover two things:

1. Technology is Dumb
I use the nav system when I’m heading somewhere I don’t already know how to find. Since I’m usually starting from home and since I know the surrounding area pretty well, there’s a good chunk of the route I already know. With only five ways off this island, the nav system gets it about half right by knowing that two of them are fairly unavoidable and trying to fit all routes through those two. Of course, I ignore it when I know better; I wish it would notice my strong preference as expressed by my choice of routes.

2. Technology Makes You Dumb
Used to be that I’d spend a fair bit of time figuring out how to get from A to B when I hadn’t been to B before; at the very least, I’d scribble down the address and phone number, then pull out the Thomas Guide and work it out. That involved noticing which intersections I’d have to pass, which ones would mean I’d gone too far, and all of that. I’d learn just a bit more about the area. Not so any longer.

The phone numbers in your speed dial; your pocket calculator … those dumb devices are making you dumber.

A Change of Pace

2007-06-04T21:11:21-07:00

One or both of you who read this blog may already know this, but today was my first full-time day with Cucina Media, we the makers of Public Square.

Today I:

Fed my kids breakfast
Saw them off to school (Beth took them)
Talked with my partners about what the week holds in store
Talked with our development partners about what we need to do in the next few weeks
Met with a couple of prospective customers
Talked at length with Christina about the focus of our next week, month, and quarter
Chatted with Rob, a long-time friend and colleague who’s just returned to the US after ~9 months in Asia
Picked up Sam after pre-school
Went grocery shopping with Sam in tow
Cooked dinner for the family
Played Elmo Uno with Beth and Sam
Posted to my blog

It was an intense business day, with five meetings squeezed into six hours. We discussed everything from what needs to be done in the next week to the different opportunities out there for us and what it would mean – to our company and to our values – to take this or that one. Values. Objectives. What we want to do and why. Good stuff like that.

It was a very happy day. =]

Radio Open Source: David Weinberger

2007-05-29T21:45:46-07:00

Sent to me by Christina, and leapt upon because (a) I really enjoyed his Library of Congress lecture, and (b) I’m feeling guilty for low blog entry count. From the summary of the April 26, 2007 edition of “ Open Source with Christopher Lydon”:

It’s hard to summarize his theory of everything in one sentence, but this is pretty close: “To get as good at browsing as we are at finding — and to take full advantage of the digital opportunity — we have to get rid of the idea that there’s a best way of organizing the world.”

I’m excited to see the buzz around David’s latest book, “ Everything is Miscellaneous,” as I think he’s very right. I’m also coming to the conclusion that taxonomies are merely folksonomies with a contextual hierarchy and which context might be shared. But I still have a lot to learn.

Link: Open Source » Blog Archive » Weinberger’s Miscellany.

The Tagline Graveyard

2007-05-19T12:48:48-07:00

Sometimes, when I change my tagline, I like to keep the previous one. Simple as that.

?? - 2007.05.19: _ Be careful with that axe, Eugene! _
2007.05.19 - 2007.05.28: _ If lazy == efficient then this is the most efficient blog on the internet. _
2007.05.28 - 2007.08.16: _ teh future is n0w _
2007.08.16 - 2007.09.11: _ now with 100% post-consumer recycled bit content! _
2007.09.11 -: _ … it is by will alone i set my mind in motion … _

Sir Tony Hoare The Elegant

2007-05-17T21:35:25-07:00

I’m in Portland for RailsConf where I met Matt Smith with Semantra. Some of the things we talked about put me in mind of some papers Prof. C.A.R. “Tony” Hoare did a while back and, while attempting to find them online, I ran across his 1980 Turing Award acceptance speech, “The Emperor’s Old Clothes” [pdf] in which he says,

“I have regarded it as the highest goal of programming language design to enable good ideas to be elegantly expressed.”

That ties in nicely with the belief I heard expressed years ago that complexity and beauty are intertwined such that over-complex things are generally not perceived as beautiful.

Incidentally, the search wandered me in and through Wikipedia’s List of Important Publications in Computer Science. A feast for the thinking geek.

Moleskine USB Drive

2007-04-26T10:39:02-07:00

If you’re a Moleskine fan, here’s a fun twist: a guy built a USB drive into one.

note@20070426.1040: The link’s getting Boing’d pretty hard right now.

Link: 80GB Moleskine hard drive [via Boing Boing]

Running the Numbers

2007-04-20T13:01:11-07:00

Chris Jordan is a photographer with an eye for visualizing real-world datasets. I suspect that he’s a fan of Edward Tufte (and likely vice versa).

Link: Running the Numbers.

We Are What We Know

2007-04-20T10:02:41-07:00

In 1900, Ladies’ Home Journal compiled 29 predictions of life in the year 2000. It’s fascinating to note where they hit (prevalent autos, gyms in all schools, etc.) and where they missed (“Storekeepers who expose food to air breathed out by patrons or to the atmosphere of the busy streets will be arrested with those who sell stale or adulterated produce.”) More fun for me, though, are the underlying theme of electricity and the absolute absence of computing.

I hope they did it again in 2000.

Link: 1900 Predictions.

Exquisite Book Desecration

2007-04-19T12:07:42-07:00

These are absolutely captivating.

Link: Fun Forever » Don’t try this at home!.

Olde Skool Web Syte

2007-04-12T18:30:49-07:00

I can’t stop looking at this site.

Genius.

Link: No one belongs here more than you. Stories by Miranda July. [via]

How To Get My Attention

2007-02-08T00:48:08-08:00

Wrote this for work to explain and make easier the process of grabbing my focus when needed. Thought it’d be fun to preserve it here, too. All the contact info has been X’d out; you’ve got it if you need it. ;]

How To Get My Attention

I’ve been asked several times in the last week for the best ways to find me and/or get my attention. Seemed worthy of a wiki page (and a blog entry).

Summary

For very urgent issues, call my cell phone at XXX/XXX-XXXX.
To give me quick short info, by SMS at xxxxxxx@xxxxxxx.xxx. I can also send short responses.
If you want my soonest attention to an email, send it to me
directly (not through an alias) and put the word “URGENT” in the title.
Know that I don’t often get to read email until very late in the day.
If you want my immediate attention to an email, call or text
me after you send it. I’ll do what I can if I’m in a place where I can
read your email.
If you want me to act on an email, tell me what action you want me to take.

Why Are You So Hard To Find?

Generally, I’m either walking around talking to folks or in a meeting. This means I’m not usually at my desk to answer my phone or read my email, which in turn means it can be hard to find me. The busier the day, the harder it is to find me.

But I All-Paged You!

I mostly don’t answer all-pages. Usually it’s because I’m in the middle of a meeting or conversation that is moving something along and don’t want to stop to find out whether or not the reason for the all-page is urgency or convenience.

The problem with all-paging is that it’s so cheap and painless for the person paging that we casually all-page because it would be more convenient to talk to them right now than to send them email or leave them voicemail¹ without regard to whether the matter at hand is really urgent, not to mention whether or not we’ll inconvenience the person being paged. And, since every time there’s a page you’ve got to listen to see if it’s for you, we’re interrupting every person in the studio multiple times per hour in order for two people to connect. If it takes each person just 10 seconds to listen, process, ignore the page, and get their head back into their work, can you imagine how much time we waste every day² … and how much we fragment people’s ability to concentrate?³

I’ve been in a conversation at my desk when my phone rang twice and, as I reached for it, stopped ringing … and then I was immediately all-paged. When I called back, it was to see if I had lunch plans. Well intentioned, but not urgent in any way, and not worth interrupting the business conversation I was having. I used to get all-paged for similarly non-urgent-but-convenient-to-the-paging-party things three or four times daily.

So now I mostly don’t answer all-pages. I’ll answer if I’m not in a meeting or working against a deadline, but those two cover most of my time.

¹ Don’t even get me started on the fact that our voicemail doesn’t pick up after 4 rings. If I can ever get mildly caught up, that’s gonna change.

² Here’s my guess: we’re 186 people, so that’s roughly 30 minutes per all page (180 * 10 secs = 1800 secs = 30 min). I’d conservatively estimate we average 5 pages per hour during the 8.5 working hours we have. That’s roughly 21.25 person hours we sacrifice daily to the all-paging system. Ugh.

³ I could quote endless literature and scientific studies but it turns out it’s easier just to google it

But how can I get you if I need you immediately?

My cell phone is XXX-XXX-XXXX. Use it for “I need an answer right now” kinds of things. If I think I’m not interruptable, I won’t answer. If you think I need to interrupt whatever I’m doing, call back immediately and I’ll answer. Bonus points for calling me from your cell phone so I have an opportunity to know it’s you (yes, I’ll add your name to my phone after the first time =).

You can push text (~120 chars) to my phone at xxxxxxx@xxxxxxx.xxx. Use that to give me important FYIs and to get very short responses. I have unlimited text messaging; please use this as often as you need.

Do you ever read email?

As of this writing (7 Feb 2007), I get roughly 1100 emails per day, of which roughly two-thirds are machine-generated and only about a quarter (~300) are safe for me to ignore. The rest I must process⁴ if I’m to stay current. Most days I manage to do okay; others I fall behind.

I process email up to five times daily:

Between 6-7a (sometimes) for about 30-40 minutes to deal with all the automated messages from the night and know what I’m in for when I arrive. This is mostly read-only time; replying takes too long.
Between 9-9:30a (almost always) for 15-20 minutes to reply to anything I put off the first time as well as to send out queries and action items to others.
Between 12:30-2p (always) for about 5 minutes to skim the morning’s mail for anything that seems urgent.
Between 6-7p (usually) for about 30-40 minutes to handle anything that needs an answer by the end of the day.
Between 10p-midnight (always) to read anything else that needs reading and respond to anything that needed a longer response than I could give earlier.l

Depending on the day, I’ll drop some of these; on the worst of days,
I’ll do the 9-9:30a, 12:30-2p, and 10p-midnight. On average, I seem to
get about an hour during the workday to process, read, and respond to
email.

⁴ I say “process” because I don’t actually read all ~1100. Many I discard out of hand as uninteresting spam due to membership on the “edit” or “film” aliases, which get a notice for every published asset ever made. Some I skim titles and decide if they need reading. I do read anything sent to me specifically.

How can I get your attention in that vast sea of email?

In order to make my time effective, I have a number of tools and tricks:

Email me at xx@xxxxxxx.xxx and you fall into a special folder which is the first thing I process. It doesn’t guarantee I’ll read it right away, but it does mean it’ll get read that day, mostly.
Put the word URGENT at the front of the subject and
you’ll get read even if I have only five minutes; you’ll usually get a
response, though it may be terse if I’m pressed for time. Please don’t
take this personally; I’m trying to be sure you know I saw your mail
and, if you need action from me, that you know what to expect.
**If you only have one line of info, put it in the subject and end with “"**. EOM means "end of message" and it tells me I don't need to read anything but the title.
Tell me what to do. If you need me to act, I need you
to tell me what you want me to do; e.g. buy something, tell you when
the something will arrive, look into why it hasn’t arrived, send it
back because it’s not what we thought it was, etc. If you don’t ask me
to do something, I’m going to think you CC’d me to make sure I’m aware
of what’s going on.
If it must be long, summarize at the top. You’ll see me do this for most any email I write that’s more than one screen; it usually looks like this:

Subj: Last Weekend’s Foo CrisisThe Short Story:The foo broke last weekend. We fixed it, but it took 10 hours during whichthe farm was completely down. We’re taking steps to be sure this can’t happen again.The Epic Saga:At 8:47p last Saturday, the foo began spewing errors; by 9:14p it had failedcompletely, taking batcho with it. We began investigating …

…that way the reader can decide quickly if they want more details now or later (or never).

Bobby McFerrin + 11

2007-01-25T21:23:57-08:00

Beth and I left Sam and Cassie with a friend and went to see Bobby McFerrin and Voicestra at Zellerbach last night. Was a great show; he’s as silly, reverent, and amazing and with 11 talented improvisational vocalists to work with, it was a decadently visceral experience.

Seeing the show brought back memories of the only other time I’ve seen him in concert, way back in 1985 at the River Oaks Theatre in Houston. He was the second act of a double bill that night, with Stanley Jordan (!). I remember going with Rich Simental (aka Roadrunner, aka RoadRich from the pre-Internet BBS days where we answered to our handles more often than our names) and Matt Tye Whose-Surname-And-Handle-Escape-Me.

It was pre-Voicestra; in fact, at the end of the show, Bobby invited “a few people from the audience” to come up on stage and sing with him. I’m pretty sure the three of us just teleported from the excellent seats Matt’s Tye’s mom got for us (she was with Rice or U of H, who were putting on the show) straight onto the stage, along with 21 other folks. And we followed it up with my first visit to Marfreless. A night I’ll never forget. Blow-you-away incredible.

Was in the middle of a particularly spiritual piece of last night’s concert when I remembered that Matt Tye died not long after that night years ago. I couldn’t keep my eyes dry for a while after that. He was a smart, gentle, generous soul and the world’s a poorer place for his absence. It’s got me remembering Toast Terrace, Penny Lane, BBS parties across from the old Zilker shooting range, and a host of handles: RoadRich, SSR, Rael, Wiggly Toes, +Rich, JTK, ref, jas, Johnny Mnemonic, Big Boy, SysLep, Reaper, sine, cq, Wall, Gryph, JoeD, HowardTheDuck, Phant, Trav, Info …

Hope the world’s working out for all of you, if you happen to look in.

–purp

Walgreens != Privacy

2007-01-23T12:28:56-08:00

Was renewing a Walgreens prescription by phone the other day when suddenly it quoted my phone number to me and asked me to confirm or deny it. This with no authentication other than a prescription number.

Don’t leave a prescription bottle where some black hat might see it …

Procrastination is ...

2007-01-23T11:33:25-08:00

If you read up on economics, you know that finding the value of utilityis one of the core, elusive bits folks would like to be able to calculate; if they could, they’d be able to predict with fair accuracy how markets will react to a given phenomenon. Piers Steel, a professor from Univ. of Calgary, has factored in procrastination. Neat!

Link: [A formula for procrastination

CNET News.com](http://news.com.com/A+formula+for+procrastination/2100-1008_3-6149636.html?tag=nefd.lede “A formula for procrastination

CNET News.com”). [via]

Spöka-y!

2007-01-01T12:16:57-08:00

When Ambient Devices first came out with their Ambient Orb, a wireless softly glowing globe which you can buy for a silly price and pay a monthly fee to have it track something online, I knew I wanted one but not at that price point. Fortunately, many other geeks found this interesting, too, and chose to hack things to make it so. The folks at Ambient even make it easy by supplying DIY notes and sources for parts.

An aside: I used some of those pages to make a Secret Santa gift from a lightbulb. Nifty fun, that. =]

Then I ran across Spöka (pronounced SPOO-kah) at Ikea. Little red, blue, and green ghost nightlights for $13. Soft diffuse light, just like an Orb. Cute shapes with faces. Already got all the power and stuff inside; also turns out to be fairly easy to disassemble, contains a rechargeable battery pack that will run it for 4-6 hours, and has tons of space inside for a builder to tuck in some custom circuits. Have a look at pics from the brief autopsy I did.

I’m pretty sure I can tuck an Arduino board and USB port neatly into one of these; I like the idea of it doing bluetooth and/or wifi, too. That plus some RGB LEDs and this is gonna be way fun. (If you do something fun with one of these, please toss me a comment and a pointer to your project if it’s online)

Update: the Ikea website doesn’t seem to have Spöka anymore. The Emeryville Ikea has scads of them, though; I may have to go buy a few extra just in case. =]

Track My Baggage Closely, Please!

2006-10-15T13:01:01-07:00

Fun lifehack via Bruce Schneier: if you want your checked bags to be tracked very carefully (and never get lost), buy a starter pistol. They’re considered a weapon; once you place it in your bag (unloaded, of course) with your very valuable camera/computer equipment and disclose it, the TSA will keep a very close eye on it for you. Handy.

Link: Matt Brandon Weblog: Tighter Security, what does this mean to the Photographer?.

Weak security in our daily lives@Everything2.com

2006-10-15T10:55:36-07:00

Reading Bruce Schneier’s Crypto-Gram today where he points out an Everything2 entry which calculates the DeBruijn sequence required to hack a car with the standard 5-button keyless entry pad, which they believe would take ~20 minutes to key in.

Securing expensive assets with what amounts to a cheap combination lock is a pretty dorky idea.

Link: Weak security in our daily lives@Everything2.com.

Chaos Theory In Practice, Part 1

2006-09-05T07:26:00-07:00

For us, Labor Day weekend was all about bailing ourselves out of the disorganized, cluttery chaos we’d allowed to take root since the birth of our daughter, Cassie, in mid-April. We tried a number of times to stem the tide with limited success; usually we’d reclaim some section of the house only to lose another. Not a satisfying victory.

We’d both read Getting Things Done by David Allen, so we decided to base our approach on that. David might be less than pleased with us, though, as we divided things into 12 categories:

Reference/Aging: things we’ll get rid of when some known period of time has passed; for example, tax records, term contracts, etc. At first we tried to differentiate inside this category, but we ended up with so little in other piles that we decided to just lump it by year and expire it when after 7 years, when we expire our tax records.
Reference/Permanent aka “permaref”: things we’ll keep for the conceivable future or which are attached to some significant item. For example, car maintenance records, a healthcare provider directory, extended warranty information.
Projects/Now: either can be dealt with in < 2 minutes or must be dealt with in the next seven days
Projects/Later: anything that doesn’t belong in “Now”
Things Which Have A Home In Our House
Things Whice Have A Home Outside Our House
Things Which Need A Home In Our House
Things Which Need A Home Outside Our House
Trash/Shred
Trash/Recycle
Trash/Garbage
Other: The category for “I don’t know what to do with this” aka “You figure it out.”

Note that categories 5-8 are really just a special subset of the two projects categories which we wanted to handle slightly differently; category 12 is just another way of saying “someone else’s inbox.”

What We Did

First, we wandered the house with boxes and grabbed anything that wasn’t where it should be. No stopping to rearrange; you could only put something where it belonged if it took zero steps to get it there (a long reach was okay). We ended up with approximately a cubic yard of stuff; think of that as 27 cubic feet, which is 8 medium moving cubes (18x18x18” or 3.375 cubic feet each), or roughly 18 bankers boxes (12x12x18” or 1.5 cubic feet each). Clutter accumulates quickly.

Next, we sorted it all to categories. Objects sorted quickly; paperwork (the bulk of what had accumulated) sorted slowly. Still, it turns out that you can do a rough sort incredibly fast if you arrange things so you can easily reach a container for each category and resolve to spend no more than a few seconds per item. We did ours on our living room floor with a paper bag labelled for each non-trash category; as we emptied boxes, we’d shift the contents of the most-full bag into that box and keep going. We’d sort for an hour or so, then take a break to play with kids, eat a meal, or just stretch and do something different for a while. At the end of each sorting session, we’d dump the trash and deal with the “Things Which Have A Home” piles as quickly as we could.

We finished our sort in about 8-9 hours split over two days. The net remaining material, by category:

Reference/Aging: 3.75 cubic feet, approx. 2.5 banker boxes
Reference/Permanent: .38 cubic feet, approx. .25 banker boxes
Projects/Now: .75 cubic feet, approx. 0.5 banker boxes
Projects/Later: 1.5 cubic feet, approx. 1 banker box
Things Which Need A Home: 1.5 cubic feet, approx. 1 banker box
Other: 1.5 cubic feet, approx. 1 banker box

So we managed to go from 27 cubic feet to a bit over 9 cubic feet. It’s amazing how much more orderly things feel for it.

Next week: Project Sorting.

Meyer's Conjecture on Philosophy

2006-07-15T15:33:13-07:00

All fundamental tenets of a philosophical system must be able to fit in a single SMS. Anything longer isn’t basic enough.

Parallel: Brevity is the soul of wisdom.

One Red Paperclip. One House. One Year.

2006-07-09T22:09:15-07:00

You probably already know about the One Red Paperclip guy. He’s managed to make his final trade, which he’ll officially complete on July 12, one year after he started. Pretty cool what you can do with the internet. (via BoingBoing)

Hey Apple, Let Me iWander!

2006-07-09T15:16:09-07:00

[note: this is a repost from a long-neglected blog I started last summer while at SIGGRAPH]

I’m currently doing a work-from-hotel morning since there’s nothing at the conference that lured me away this morning, listening to my rather eclectic ~4K song playlist on my iPod on shuffle. This leads to some interesting, occasionally jarring, boundaries between songs; it also got me thinking more about something I’ve wanted for a while.

If you’re like me, you have moods. Sometimes my moods do an energetic sprint through hours of Pete Tong’s Essential Mixes. Other times, they dance delightful undulations down the sandy beaches of Angelique Kidjo’s Black Ivory Soul. Occasionally they strut, oozing an attitude of down-home Texas hip as only Stevie Ray Vaughan or Lyle Lovett can manage.

(We’ll ignore the immature, pissed-off moods that stomp into Evanescence, shoot you the finger, and slam the door in your face. They just want attention. Don’t give in.)

For me, the analogy is apt: music makes a path for me and, when I find I’m on the right one for the moment, I want to follow it until I come to a crossroad where another path seems more interesting.

I want to iWander.

If you think about it, it should be easy. Every song starts with some implicit metadata attached: title, artist, album, year, genre – though the last is often argued over, especially out on the electronica/dance/techno/alternative fringe. Apple adds some more: your rating, grouping, playlists, last played, artwork, etc. I want to be able to add more, defining relationships between artists, songs, albums, etc. that exist only for me. I want to be able to change paths – say, from playlist to album, then perhaps genre, etc. – to follow my mood.

An example might go something like this:

Texas Flood, Stevie Ray Vaughan, Greatest Hits
Man, I loved living in Austin while he was there. More of him.
Pride and Joy, Stevie Ray Vaughan, MTV Unplugged, Vol 1
I remember when he did this at Austin City Limits. That reminds me of the time we saw Indigo Girls there …
Romeo and Juliet, Indigo Girls, Rites of Passage
That was a good night; seeing them, Nancy Griffith, and Mary Chapin Carpenter on the stage was phenomenal
This Shirt, Mary Chapin Carpenter, Hitchhiker Exampler
Wow. For a cheap sampler of music from a mediocre cable show, this is amazing. Let’s hear it all again.
[…]
Make It Easier, Indigo Girls, Hitchhiker Exampler
Oh, yeah, they have a tune on this one, too. I must be in a mood.
[…]
Talkin’ at the Texaco, James McMurtry, Hitchhiker Exampler
He knows small-town like nobody’s business, that’s sure. Didn’t David Garza do something fun like this?
Big Stick, Twang-Twang-Shock-a-Boom, Me So Twangy
Bummer they couldn’t get along … they were awesome.

…and so forth. Stevie Ray Vaughan could have just as easily branched to Robert Cray or Eric Clapton (great guitarists all and SRV died in a plane crash while on tour with both of them).

There’s even a commercial angle: it’s not legal for me to share my music with others, but give me a way to share my metadata and I can find other people who have similar but diverging tastes and find new music I’ll probably like … and, as a bonus, I can judge that likelihood by evaluating the nature of the association between the two (“Austin City Limits? That’s a hick show … ewww.” or “David Garza was in a band? Wow!”) and follow the paths that seem interesting through the iTunes Music Store and on down the sidewalk.

Hey, Apple, you listening?

Cool Blog: Creating Passionate Users

2006-07-09T15:08:38-07:00

Wandering through some blogs and ran across Kathy Sierra’s “Creating Passionate Users” blog, about which I can’t say enough nice things. She hits a number of my favorite issues, such as:

Link: Creating Passionate Users: Ultra-fast release cycles and the new plane.
Link: Creating Passionate Users: Death by risk-aversion.
Link: Creating Passionate Users: The hi-res user experience.
Link: Creating Passionate Users: Conversational writing kicks formal writing’s ass.

And one that made me smile, too:

Link: Creating Passionate Users: Code like a girl.

I’ll be spelunking her archives for quite some time, I suspect.

Cleaning up old mail

2006-06-23T14:11:29-07:00

Just a random thought as I start posting again: I want an easy way to clean up thousands of old emails quickly. Sure, I could leave them around, but frankly it just bugs me to know that I’m maintaining a heap of digital compost some of which I will never, not in a million years, find useful again.

The surest way would be to quickly tag any email I read and choose not to delete with some indicators about why I didn’t just ditch it immediately. Evolution, the Linux Outlook-a-like, gives me a little of this via its Followup feature; Thunderbird didn’t have a plugin to help with this when last I looked.

Still, there should be some helpful methods, such as a list of all the old mail’s senders along with a count of how many they sent and expandable to see the titles. Let me easily select all or some of any sender’s emails for deletion (for UI, think packages/subpackages in an installer). Others might be grading the importance of the old email according to length of thread, number of times I replied in the thread (indicating I found it important) and so forth.

And yes, I’ve got a lot of these to go through. =\

Bring Your Own Laptop To Work?

2006-04-26T08:53:20-07:00

Don Marti blogged an idea that’s been floating around for a while: what if, instead of corporate IT supplying you a desktop which is largely underutilized, the company paid you a “laptop allowance” of some sort and you brought/brought your own? While it’s less feasible in the CG industry (those desktops are makin’ pixels, so we need ‘em around and all ;), it makes great sense in all sorts of other contexts – and was something I was begging for ca. 2000 when I was a “pre-sales technical consultant.”

There’s heated discussion in the comments of Don’s post which largely divides into “Right On!” and “IT’s Worst Nightmare.” Most of the latter seem to miss Don’s point: the user owns the laptop; IT supplies and supports only the services (network connection, DHCP for IP address and pointers to directory services such as DNS and LDAP, etc.) Users get some amount of free tech support per year (Don suggests “two major incidents” where I’d suggest one hour per week which accumulates with a 16 hour cap); after that, some sort of billing kicks in, either to the user or to their department.

In the end you have positive motivation (“I get to choose!”) and negative motivation (“That’s gonna cost me …”). Sounds like a good closed system to me.

Link: Don Marti business/byol.html.

Slowly Crawling Towards Daylight

2006-04-25T11:50:53-07:00

Expect a flurry of postings as I start to catch up post-baby (did I mention? We had our baby girl. She’s great. =) and rejoin the land of the living. I’m taking off work until early June, so lots of time to immerse myself in cool tech, maybe even to get some things done.

Maybe. Baby. Maybe. =]

SHDH9 This Weekend!

2006-04-25T11:43:26-07:00

Woohoo! I hope I can go, but with a new baby, that’s a lot of hoping. =]

Link: Upcoming.org: SuperHappyDevHouseNine at SuperHappyFunHouse (Saturday, April 29, 2006).

Panasonic's Vision of the Future: All Pixels, All The Time

2006-03-29T12:43:20-08:00

From Akihabara News, deck the walls with screens of interaction. Two side-by-side 110” touch screens with user-manipulable windows of video and audio from any/many sources. This makes me shudder. I can’t believe it’d ever be interesting to monopolize so much space to artificially deal with the world – much less to facilitate the emerging ADD trend.

But then, I prefer my 1024x768 12” PowerBook to a WXVGA monster display.

Ajaxy Homepages

2006-03-27T22:25:23-08:00

A nifty find via TechCrunch: NetVibes and ProtoPage, your personalizable ajaxy homepage. Toss in nearly any kind of content and neatly arrange or slop it around (respectively). Look for this on Google’s personalized pages sometime later this evening or early tomorrow. ;]

Meshing an Airport Extreme and a Linksys WRT54G

2006-03-25T23:19:45-08:00

I’ve spent the day working on home tech projects; we’re paring down, getting rid of excess tech accumulation. Part of that is to eliminate one Airport Extreme by meshing the other one with a new Linksys WRT54G we’d bought a few weeks ago. According to this post at rgbdream.com, we should be able to just mesh them, no muss, no fuss. No dice, either. I spent hours sniffing networks and trying to understand why, but it just didn’t work out.

Then I found out about dd-wrt.org and their firmware for the WRT54G; they’re based on openwrt.org’s work and are one of many implementations. (I chose to avoid sveasoft’s implementation as there seem to be questions about their adherence to the GPL. Icky.) Bummer, though – the new WRT54G is v5, which is no longer an embedded Linux device. This makes it an inhospitable environment for the various open source firmware projects.

The old WRT54G is v1 — yay! — and is now reflashed according to the advice of Swimmer on testmy.net and, with very little problem, there’s now just one wireless net in the house (yes, there were two before – WRT and the Extremes ;p). It’s amazing what the dd-wrt firmware enables. I can ssh into the router now. I can set up a hotspot, complete with http redirect to a login page (and easy ChillSpot setup, whatever that is). I can set up fixed MAC address to IP associations for DHCP, which means that I can actually have decent internal DNS without needing a separate DHCP and DNS server. Most of all, it’s nice streaming Twang-Twang-Shock-A-Boom through iTunes to the stereo knowing there’s far less hardware in the mix.

Anyone need an extra Airport Extreme? I’ll trade for a pre-v5 WRT54G …

LinkStations: NAS for the Hacking

2006-03-22T15:29:57-08:00

Think network-attached storage (NAS) that talks to whatever flavor bx you’ve got. Windows? No problem. Mac? Sure thing. Linux? Happy to. Pull it out of the box and in a few minutes you’ve got hot-and-cold running fileservice.

Then I found out there’s a community of folks hacking them. Putting Debian on ‘em. Augmenting them with all sorts of goodies.

These little hummers are too much fun.

Link: [LinkStationWiki

Main / Home Page](http://www.linkstationwiki.org/ “LinkStationWiki

Main / Home Page”).

LiteBrite on your Fridge

2006-03-22T14:36:39-08:00

By way of MakeBlog comes these Magnetic Refrigerator Lights … they’re phenomenal! Mother of All That’s Decent and Good, I must have this.

p.s. the site this is hosted on, Instructables, looks fun, too. Gotta find time to browse that.

Biodegradable Food Containers

2006-03-20T17:38:33-08:00

Most excellent: disposable “plastic” food containers made from corn resin. They biodegrade in ~45-60 days in your compost bin. You can buy them at Ecoproducts.

Link: Popgadget: Personal Tech for Women

Tiny MP3 Streaming Server

2006-03-18T20:41:02-08:00

From hack a day: a DIY MP3 streaming server … BSD licensed so you can download the plans and source for it or buy the kit, assembled or not. Maybe when I’ve got a couple hundred to spare …

Link: Network attached MP3 streamer - hack a day - www.hackaday.com.

Leaftag: Tag files and stuff in Linux

2006-03-17T09:20:20-08:00

Drew turned me on to Leaftag this morning. Looks wicked awesome for doing TagEverything (I need a a better name for that).

Tagging for the Linux desktop

Leaftag is a library and set of utilities for tagging files on the Linux desktop. It’s a convenient way of organizing files in a non-hierarchical manner. Local files, websites, or anything with a URI can be tagged with one or more names and easily referenced by anything that supports Leaftag.

Okay, Okay!

2006-03-14T23:24:59-08:00

Yes, Drew, I’ll extend semfile to for my Tag Everything project!

There, I’ve said it in public. Or as public as the internet is.

Libraries as Playlists

2006-03-14T10:42:11-08:00

I love reading Dave Weinberger’s blog; his excitement at the ways technology can augment the world mirrors my own. I love this post for the reduction of where Web 2.0 and ubiquitous metadata are heading us: “When all our works are digitized, a local library will be nothing but a playlist.”

Think about that … mash up Google Books with a video iPod and BAM – it’s a library in your pocket. Toss in Audible.com and it’s even better.

I’m with Dave … I can’t wait!

How To Tie Your Shoes

2006-03-14T10:31:19-08:00

Found this on Dave Weinberger’s blog. I’ve been tying my shoes wrong for 40 (okay, 36; before that, Mom was doing it wrong) years. Who knew?

OLPC: You Know, For Kids!

2006-03-09T17:58:23-08:00

A quick link to my initial thoughts on the One Laptop Per Child project from my other blog where I think more about social implications (and which doesn’t accept trackbacks. I’ll have to figure out how to fix that …) The geek in me wants desparately to play with one of these. But then I want to make solar mobile wifi hot spots, do cool things with tiny lcd displays, and play with stomp boxes.

I guess I want to go off the network without leaving the net behind.

Good Engineering!

2006-03-09T16:38:29-08:00

Good Engineering!
Originally uploaded by jimmm.

Someone was thinking when they made this fridge; instead of making some odd-shaped plastic water holding tank that would be a pain to install and replace, they just coiled up a bunch of the standard water supply tubing they were already using to route water through the fridge to the icemaker and door.

The coil is roughly 12 inches in diameter and 3 inches thick; the tubing is 0.33 inch outer diameter and 0.25 inch inner diameter. This means there’s ~60 feet of tubing holding ~2.5 quarts of water.

Sweet!

TypePad == Geek Haven

2006-03-08T12:11:40-08:00

I love that I can post to my blog from my mobile. Thanks, TypePad!

Okay, I could have just said, “First Post!” and had done with it, but this
was much more fun. =]