A few weeks ago, I was wondering about how to keep my fledgling start-up from falling prey to unplanned maintenance or downtime. Because of the nature of the site (managing design mockups), the element that most concerned me was in-Passenger handling of thumbnailing. We use paperclip. Paper relies on ImageMagick, a notorious resource hog, especially when it comes to RAM. I didn’t want to be able to let a single user hamstring our entire site for a thumbnail.

Fortunately, at exactly the same time, Jesse Storimer created delayed_paperclip to defer thumbnailing (or any other paperclip processing) using delayed_job. Very cool! I started watching it.

I knew that I wanted A/B testing and metrics to play a big role in Mocksup and, right now, that means Redis and Vanity. After a very successful trial run with them both, I knew that Redis would play a part in the application’s future. Having read a history of Resque from my friend defunkt, that bit of software was on my list to try out anyway. Resque was easy to get up and running. Why not try to port delayed_paperclip to Resque? After some hacking, it was working fine in my application. Even cooler! So I blabbed on Twitter about it to see if anybody else liked the idea. Some people did, including the man himself. But I couldn’t release with broken tests, that’s not me!

I got the tests passing, and let Chris know. He tweeted about it and suddenly my little hack is trending on GitHub. If Mom could see me now! Well, I mean, she can see me, but— if… I could explain it to her without her being bored! Yeah, that’s it!

Seeing my “name in lights” was definitely gratifying, but I started to wonder what the future would be for this little piece of code. The author is using Delayed::Job; he works at Shopify for god’s sakes. He may not be interested in this functionality. And what of the integration with paperclip over time— am I really going to be able to keep this supported? There’s a lot of sophisticated Ruby work happening in that plugin, to balance Rails, Paperclip, and Delayed::Job. So I sent a pull request and figured “Let’s just see.”

Jesse could not have been cooler, and integrated my changes immediately! He even added on an additional bit of coolness that made the plugin even better for the end user. After one last little change pointing that additional nicety out to the plugin’s potential users, I was done! I updated my fork and pointers to it with the information that the functionality I added was merged into the project.

Let’s recap the winners:

• Thoughtbot, for getting an even more robust and featureful plugin allowing delayed post-processing
• Jesse Storimer, who gets a plugin with more exposure.
• Chris and resque, for getting a new plugin for resque
• The potential users of delayed_paperclip, for being able to choose which work queue system fits them better.
• Me, for scratching an itch and getting involved with all these great people and projects.

To me, much of the beauty of contributing to open source is being a member of this kind of community. I enjoyed it so much, I had to write a little bit about it.

The case of the 500-mile email

Read the FAQ about the story.

The following is the 500-mile email story in the form it originally appeared, in a post to sage-members on Sun, 24 Nov 2002.:

From trey@sage.org Fri Nov 29 18:00:49 2002
Date: Sun, 24 Nov 2002 21:03:02 -0500 (EST)
From: Trey Harris
To: sage-members@sage.org
Subject: The case of the 500-mile email (was RE: [SAGE] Favorite impossible
task?)

Here’s a problem that *sounded* impossible… I almost regret posting the
story to a wide audience, because it makes a great tale over drinks at a
conference. The story is slightly altered in order to protect the
guilty, elide over irrelevant and boring details, and generally make the
whole thing more entertaining.

I was working in a job running the campus email system some years ago when
I got a call from the chairman of the statistics department.

“We’re having a problem sending email out of the department.”

“What’s the problem?” I asked.

“We can’t send mail more than 500 miles,” the chairman explained.

I choked on my latte. “Come again?”

“We can’t send mail farther than 500 miles from here,” he repeated. “A
little bit more, actually. Call it 520 miles. But no farther.”

“Um… Email really doesn’t work that way, generally,” I said, trying to
keep panic out of my voice. One doesn’t display panic when speaking to a
department chairman, even of a relatively impoverished department like
statistics. “What makes you think you can’t send mail more than 500
miles?”

“It’s not what I *think*,” the chairman replied testily. “You see, when
we first noticed this happening, a few days ago–”

“You waited a few DAYS?” I interrupted, a tremor tinging my voice. “And
you couldn’t send email this whole time?”

“We could send email. Just not more than–”

“–500 miles, yes,” I finished for him, “I got that. But why didn’t you
call earlier?”

“Well, we hadn’t collected enough data to be sure of what was going on
until just now.” Right. This is the chairman of *statistics*. “Anyway, I
asked one of the geostatisticians to look into it–”

“Geostatisticians…”

“–yes, and she’s produced a map showing the radius within which we can
send email to be slightly more than 500 miles. There are a number of
destinations within that radius that we can’t reach, either, or reach
sporadically, but we can never email farther than this radius.”

“I see,” I said, and put my head in my hands. “When did this start? A
few days ago, you said, but did anything change in your systems at that
time?”

“Well, the consultant came in and patched our server and rebooted it.
But I called him, and he said he didn’t touch the mail system.”

“Okay, let me take a look, and I’ll call you back,” I said, scarcely
believing that I was playing along. It wasn’t April Fool’s Day. I tried
to remember if someone owed me a practical joke.

I logged into their department’s server, and sent a few test mails. This
was in the Research Triangle of North Carolina, and a test mail to my own
account was delivered without a hitch. Ditto for one sent to Richmond,
and Atlanta, and Washington. Another to Princeton (400 miles) worked.

But then I tried to send an email to Memphis (600 miles). It failed.
Boston, failed. Detroit, failed. I got out my address book and started
trying to narrow this down. New York (420 miles) worked, but Providence
(580 miles) failed.

I was beginning to wonder if I had lost my sanity. I tried emailing a
friend who lived in North Carolina, but whose ISP was in Seattle.
Thankfully, it failed. If the problem had had to do with the geography of
the human recipient and not his mail server, I think I would have broken
down in tears.

Having established that–unbelievably–the problem as reported was true,
and repeatable, I took a look at the sendmail.cf file. It looked fairly
normal. In fact, it looked familiar.

I diffed it against the sendmail.cf in my home directory. It hadn’t been
altered–it was a sendmail.cf I had written. And I was fairly certain I
hadn’t enabled the “FAIL_MAIL_OVER_500_MILES” option. At a loss, I
telnetted into the SMTP port. The server happily responded with a SunOS
sendmail banner.

Wait a minute… a SunOS sendmail banner? At the time, Sun was still
shipping Sendmail 5 with its operating system, even though Sendmail 8 was
fairly mature. Being a good system administrator, I had standardized on
Sendmail 8. And also being a good system administrator, I had written a
sendmail.cf that used the nice long self-documenting option and variable
names available in Sendmail 8 rather than the cryptic punctuation-mark
codes that had been used in Sendmail 5.

The pieces fell into place, all at once, and I again choked on the dregs
of my now-cold latte. When the consultant had “patched the server,” he
had apparently upgraded the version of SunOS, and in so doing
*downgraded* Sendmail. The upgrade helpfully left the sendmail.cf
alone, even though it was now the wrong version.

It so happens that Sendmail 5–at least, the version that Sun shipped,
which had some tweaks–could deal with the Sendmail 8 sendmail.cf, as most
of the rules had at that point remained unaltered. But the new long
configuration options–those it saw as junk, and skipped. And the
sendmail binary had no defaults compiled in for most of these, so, finding
no suitable settings in the sendmail.cf file, they were set to zero.

One of the settings that was set to zero was the timeout to connect to the
remote SMTP server. Some experimentation established that on this
particular machine with its typical load, a zero timeout would abort a
connect call in slightly over three milliseconds.

An odd feature of our campus network at the time was that it was 100%
switched. An outgoing packet wouldn’t incur a router delay until hitting
the POP and reaching a router on the far side. So time to connect to a
lightly-loaded remote host on a nearby network would actually largely be
governed by the speed of light distance to the destination rather than by
incidental router delays.

Feeling slightly giddy, I typed into my shell:

$ units
1311 units, 63 prefixes

You have: 3 millilightseconds
You want: miles
* 558.84719
/ 0.0017893979

“500 miles, or a little bit more.”

Trey Harris
–
I’m looking for work. If you need a SAGE Level IV with 10 years Perl,
tool development, training, and architecture experience, please email me
at trey@sage.org. I’m willing to relocate for the right opportunity.

Dedicated to Happy Bear’s biggest fan, Ashley-Rose!

“

(Via Savage Chickens.)

More Useless Mutants.

“

(Via Savage Chickens.)

Sure, I could go back and tell my teenage self not to give up playing the piano, but would that just leave me in my 30’s dreading another gig in the bar of some hotel waiting for the inevitable tooth-grinding moment some drunk tourist requests ‘Piano Man’ for the sixty thousandth time? Would I hunch over the keys, bang out those first chords and wish to my core I could go back in time and advise my teenage self to teach himself Perl?

So I pondered. And I thought. And I finally came up with the one important, nay vital bit of advice I would give myself if I had the chance….

Charles,

This is your future self. I’m 33 now and I’ve been given the opportunity to reach back in time and give you one piece of advice. You won’t understand it today but if you hold it close to your heart you will one day thank me for it.

Whatever you do, prefer composition over inheritance.

See you in 17 years,

Charles.

“

(Via The Fishbowl.)

As you may have heard, our organizer and sponsor, Jesse Schoch, has left the Queen City for Northern Virginia early this month. That leaves us with some pretty big shoes to fill if we’d like the group to continue on!

I’d like to suggest an organizational meeting for Amelie’s Bakery on Tuesday, Aug. 25th, starting at 6PM. The goal for the meeting, in addition to socializing with fellow Rubyists, would be for us all to talk about the role we could play in charlotte.rb, if any.

In my experience, the less likely you are to come to meetings in general, the more important and helpful it would for you to attend this one. The single most important role in these groups over time is usually “meeting attendee” so there’s no need to be able to spend a lot of time volunteering or coding or anything like that to have fun and help the group at this meeting. The more idea the group can get about what we’d all like to see, the better.

I basically picked that day out of a hat, so if that’s problematic we can talk about picking a better one. Please leave a comment if you’d be able to come, or have any other questions or comments.

Amelie’s Bakery is located in NoDa. Its location is pictured below. Click on the map to go to Google Maps and get directions.

View Larger Map

Links:

They are also getting very excited about the Joyent Smart Platform and its full-stack JavaScript approach. This certainly fits right in with using web standards all the way to the client.

I’m interested to see where they go with this, for sure.

The “big idea” of the talk was doing open-source software development on an open-source hardware platform that does something cool! Like flying a freaking blimp to The Flight of the Valkyries!

Arduino development (and in this case, BlimpDuino development seems like the truest form of hackery possible to me, and I found it a thing of beauty. As they don’t appear on the official schedule, I may have gotten their names wrong, but I believe it’s the Evans brothers put on quite the show. It’s clear they put a lot of time and love into the project. And the end result is incredibly cool.

When you are doing this kind of hacking, you’ll run into problems that you might never expect. For example, their blimp’s engines both rotate in the same direction. When the blimp is at full-forward, the torque itself actually steers the ship in the opposite direction to the rotation, and they’ve not been able to correct (or get an engine that rotates the opposite direction yet.)

I loved the feeling of having two brothers in their garage, working on different parts of the problem, crashing things, setting things on fire, having to chase down their own invention (the tether was added in later models, LOL). Plus, the style of the hardware hacker with his handlebar mustache and muttonchops was a real plus.

It will be great to watch this field closely, as the Arduino boards are still very limited in terms of capacity and throughput. They mentioned needing to be able to fit their entire blimp control scheme in about 14k, which is not a lot of resources to bring to bear. The Ruby code that instructs the hardware compiles to C++ code, then to machine bytecode, and finally uploaded wirelessly to the blimp. It’s cool to watch. But didn’t you get that message already?

Blimp development: p3wn@ge