Zetetic - Home

Changes to Tags in Tempo

2008-08-20T16:55:00Z

We need to make an adjustment to how Tempo handles tags in order to squash a bug that keeps popping up on some users. Basically, we’re only going to allow tags to contain letters, numbers, hypen and underscore. Any other characters (like parentheses, +, and *) will be converted to underscores automatically.

If you enter sys+admin as one of your tags you will get sys_admin instead, which we think isn’t so bad, really. This change will be going into production tonight.

SQL Developer

2008-08-14T14:22:00Z

While I’m waiting for my Windows VM to reboot, just figured I’d mention that Oracle’s free tool SQL Developer totally rocks. And it supports MySQL for you geeks out there (POSTGRES 4EVA!!!!!)

It makes coding and testing complex PL/SQL fairly painless. It’s built on Java and runs on Mac OS X beautifully.

Yesterday

2008-07-31T14:37:00Z

We made a couple of maintenance updates to Tempo last night, and added in a couple of features that numerous users have been asking us for:

“Yesterday” date range selection, which we now use all the time.
Inclusion of locked status on time export, a flag that some use to track whether or not they’ve billed for an entry already.

2D Barcodes and Semapedia.org

2008-07-28T18:21:00Z

Recently I met Alexis Rondeau, one of the two clever fellows that created the site Semapedia.org. The site allows you to create 2D barcodes, called Semapedia Tags, that link to information on Wikipedia. The idea is that you print a tag that links to information about a place or thing, then you stick that tag on the place or thing. Anyone with a 2D scanner program on their phone can then lookup the information at the site when they see the tag. Pretty cool! (I’m still trying to get the program for my Treo installed correctly.)

Anyway, I saw this on their blog and I had to share:

so you are on a bus stop and there is a barcode to download the bus schedule. Great Idea, not. The poster with the barcode takes a whole side of the bus stop, why not just print the time table, how often does that change? Why pay for anything like that. If the service behind the barcode would tell you exactly in realtime where the bus currently is located or tell you if any of your friends are on that bus, then we have something a printed time table cannot provide and is clearly more attractive. Haven’t seen any of the other ideas, but for starters, detect needs, find out what current medias don’t provide and so on.

Indeed! Always go for the simplest solution.

On why not videomail

2008-07-25T19:25:00Z

Warren Ellis wonders aloud why “videomail, in these broadband days of ours, has never made a bigger dent. Why I don’t get videomail in my inbox along with email.”

I’d say it’s because most of us see a camera pointed at us and we feel the need to act. It’s rather difficult for a lot of people to “be natural” when they are being filmed.

When most folks seem to be looking for ever faster, ever more seamless (and often literally asynchronous) communication, the last thing they want to do is “have to act” for a two minute videomail.

Coding for Failure

2008-07-25T19:04:00Z

We all love mash-ups, right? Especially us developers, builders of fine web tools. When we build useful web applications I think we all tend to want to provide integration hooks to other services because our users will get more functionality (in many cases they get more bang for their buck, so to speak) and because it’s kinda cool! Nothing wrong with that, gives you something to get in touch with your users about, sometimes gets you a bit of a press, too.

But mash-ups aren’t all fun and games, they require some careful planning and hard work, even if your current system is well designed with low-coupling and a good MVC model. I saw this post by Hampton over at Unspace and got to thinking that I ought to do a little musing on coding for failure and discuss some of the techniques we’ve used in our services.

When you run a reminder service like PingMe, where your users trust you to deliver their messages without fail and on time, you have to step up your game when it comes to implementing a robust system. When you then integrate your app with an external service like Twitter to provide your users with a useful and cheap SMS/text messaging interface, you have to consider the reliability of that external service and code for failure.

Now, on some level there’s only so much failure you can prevent. Mail systems and domains can go dark, e-mail to sms gateways can blink out, there’s not much you can do about it beyond picking a good MTA and spending a solid amount of time configuring it properly. (We highly recommend Exim, which is the most flexible one out there with great documentation and a strong user/development community.)

The great thing about serious business mail servers like Exim is that they have been very good at handling failure, retrying, and eventually giving up for a very long time, and negotiate this process with other mail servers over a long-established protocols. So if we send a message to your_phone_number@vtext.com (Verizon Wireless’s email-to-sms gateway), and the vtext.com MTA is temporarily unavailable, Exim will try again. And again. And again. And then give up. And our PingMe messaging dispatchers never have to worry about this. The E-mail and SMS handlers simply turn the messages over to Exim on time and wash their hands of the matter.

While most of PingMe’s outbound messages are delivered via e-mail, a large portion go out over Twitter. Without beating a dead horse, and while acknowledging that their reliability has improved quite a bit, Twitter is not like our local MTA, it’s just not as reliable and as a remote HTTP service, not nearly as fast. On the other hand, once in a while our MTA might be down (perhaps I bork the config file and it doesn’t come back from a restart). More importantly, there is no mechanism in place for handling failure. When you send a message to the Twitter API, it either works out or you get a failure. And if you don’t handle that failure, you fail, too!

We handled this by implementing a retry system for our dispatchers. We caused a number of exceptions to bubble up in our test environment, everything from inability to connect to twitter to no network at all, and began catching the exceptions and wrapping them as DeliveryExceptions. If Twitter (or our MTA) is down, the message instance is delayed by a few minutes and marked for retry. We’ll retry numerous times before giving up (there comes a point at which a time-based message loses its relevance…).

Just a little peaking into our messaging code:

rescue DeliveryException => e
@log.error "Caught delivery exception, marking event for retry." 
retry_event(event)
...
def retry_event(event)
      event.status = Event::STATUS_RETRY
      event.retry_count += 1 # up the retry count
      event.retry_at = event.dt_when + (5.minutes * event.retry_count)
...
  def lock_a_block(type_name)
    before = (Time.now.utc).to_s(:db)

    ActiveRecord::Base.connection.execute(
    <<-END_OF_SQL
      UPDATE events SET dispatcher = '#{@name}'
      WHERE id IN (
        SELECT e.id FROM 
          (( events e INNER JOIN targets t ON e.target_id = t.id )
          INNER JOIN pings p ON e.ping_id = p.id)
          INNER JOIN target_types tt ON t.target_type_id = tt.id
        WHERE 
          tt.const = '#{type_name}'
          AND 
          (
            (e.dt_when < '#{before}' AND e.status = '#{Event::STATUS_PENDING}')
            OR
            (e.retry_at < '#{before}' AND e.status = '#{Event::STATUS_RETRY}')
          )
          ...

The code actually gets quite a bit more complicated than that, and I don’t really want to go fully dissecting the polymorphic message handlers we’ve written, but it shows you how handling failure isn’t really an outlier problem, it becomes core to your system. It’s just as important as returning those nice model validation errors that Rails makes so convenient for you.

Another technique we use in PingMe is pipeline prevention. Well, that’s what I call it. But basically you can’t have one Twitter-bound ping holding up every other outbound ping at 5pm EST! We spent a lot of time implementing a system that allows for many concurrent dispatcher daemons, and all Twitter-bound pings go through only two of them, preventing the others from being affected by the high latency when connecting to Twitter. We ended up using the mutex pattern with Postgres:

  def acquire_mutex
    ActiveRecord::Base.connection.execute(
    <<-END_OF_SQL
      LOCK mutex IN ACCESS EXCLUSIVE MODE;
    END_OF_SQL
    )
  end

In our time-tracking app Tempo, we allow users to send time entries and start timers by sending messages to our Twitter account (twitter.com/keeptempo), and we have a daemon checking the API for new direct messages every couple of minutes.

Two things have to happen for that to work over direct messaging – both accounts have to be “following” each other. So the user follows us on Twitter, then enters their Twitter ID on their Tempo profile. Tempo does a quick check to make sure you’re following ‘keeptempo’, and then attempts to follow you. Either of those connections to the Twitter API can and often do fail.

So what do we do? We put together a rake task that generates a list of twitter ids on our user’s profiles that we aren’t following, and sends a follow request for each of them. We run it as a periodically and it catches quite a few. Not perfect, but just about the best we can do. It’s better than letting users walk away thinking that it doesn’t work at all! In that case you just look bad, and it’s not even your fault!

But it is your fault, actually, because you have to code for failure, or you look pretty bad when the exceptions bubble up to the surface, literally. Or, worse, you present the user with inaccurate information based on an exception state you didn’t plan for, which can really put you in a bad light.

I stay positive, but I code for failure ;-)

Earthlink Spam Blocker

2008-07-25T15:49:00Z

Listen, we need to talk. It’s about your Earthlink spam blocker. The one that does this:

I apologize for this automatic reply to your email.
To control spam, I now allow incoming messages only from senders I have approved beforehand.

If you would like to be added to my list of approved senders, please fill out the short request form (see link below). Once I approve you, I will receive your original message in my inbox. You do not need to resend your message. I apologize for this one-time inconvenience.

Click the link below to fill out the request:

I realize that this has its uses and I’m sure it cuts down on the amount of spam you get.

But, there are better ways. Much, much better ways. And if you’ve been wondering why you never get your activation e-mails for things that you sign up for, that’s pretty much why. You wouldn’t believe how many of these responses we get every day for PingMe and Tempo. We always try to do you the favor and click the links when we have time, but lately there are so many (now requiring us to fill out a form), that we can’t really keep up.

Do everyone a favor, turn that thing off. Cut down on the amount of mail flying around the internet.

@rubyfringe - Giles for the win

2008-07-20T16:42:00Z

Giles is up there doing the Archeopteryx Ruby MIDI generator talk. Best quote so far: “most people don’t use the freedom they have.”

I can’t describe to you how rapid, funny, inspiring and informative his talk is. Maybe I can: we’re all starving for dim sum lunch and are willingly sticking it out to the end of his talk. It’s that awesome.

@rubyfringe - when programmers don't like it

2008-07-20T15:29:00Z

Hampton Catlin’s on, he’s talking about replacing Javascript, basically. Well, there’s more to it than that. But he says, “I find when programmers think an idea is a really bad idea and they can’t give you a good **ing reason for it, then it’s a good idea!”

Guilty as charged.

@rubyfringe - \m/

2008-07-19T21:45:00Z

Zed Shaw literally just rocked the house by playing into his own sequencing software and winging some lyrics, guitar and harmonica.

@rubyfringe - Tiger in the cage

2008-07-19T20:55:00Z

Zed Shaw is up next. He is pacing. He told me he’s especially looking forward to angering the musicians in the audience. And he has a harmonica. Lord help us.

On Identity

2008-07-19T20:41:00Z

I’m going to break up the Rubyfringe blogging for a minute to stump on a topic that’s close to our hearts here at Zetetic.

When it comes to building a new web application our designing decisions related to user accounts should be driven by a common paradigm: a user object should represent a real life human being. That user object can then be associated with any number of other entities. Fun! Flexible!

You can see this at play in our time-tracking app Tempo: every user has only one account, and can be associated with any number of other people’s projects. It’s a simple many-to-many relationship, a social network relationship really, and it seems like an obvious and straight-forward design decision—but that’s not how a lot of web applications work.

Many really good systems out there have made the decision to build their web applications around companies or organization and not people. From there, people are users in the company (as in “assets” or “resources”), and are walled off from other people in other companies using the same web application. In some cases they even log into different sites! This can require a real flesh-and-blood human being to have multiple accounts, one for each company they work with that uses the system.

This paradigm can provide some conveniences (although none of them unique to it other than the feature of company-as-corral), but it can cause issues in the real world when flexibility is severely limited by the “company” wall. If Tempo followed this paradigm, our users, many of whom work with many different companies, client, and subcontractors, would be at a real disadvantage having to maintain numerous accounts and log into different sites just to summarize their full monthly billing!

Many folks who are independent subcontractors or consultants love to use Tempo because it allows them to work with many customers and partners without having separate logins. They can log in and report on all their time from one place.

Social networking applications, however cliché it is to laud their innovation nowadays, are really paving the way when it comes to recognizing that organizations and companies are just groups of people. They are defined by relationships between people, not a fixed hierarchical structure where the organization is the center of the universe and people are bolted on ad hoc (or grown like leaves on the company tree)!

Consider how useful it would be to add groupings or private channels to an asynchronous messaging system like Twitter. Everyone on Twitter has their own account. Allowing people to join groups would be easily implemented (scalability issues aside). Now imagine Twitter being based on organizations primarily, and having to have one account for each organization. Ugly, right? And unnecessary.

@rubyfringe - Jazzers and Programmers

2008-07-19T20:08:00Z

So far the overwhelming blowout talk here was Nick Sieger’s of Sun Microsystems. He’s got the whole thing over here on his blog, with a complete discography of the great music he played.

While listening to this great primer on the history of Jazz and discussion of core concepts, some ideas collided in the back of my brain. I’m pretty familiar with a lot of it, but it’s been a while and I’ve been stuck in some of the same genre/idea ruts. I started thinking about some things I’ve been working on for my band, things I want to try, some of the concepts Nick was discussing and illustrating by playing us music and I suddenly had an explosion in my head and a near-complete song idea! I actually drew out a few bass clef staves and wrote some sheet music notes real quick to try when I get back home (or in a music shop on Queen St. on the way to the after party.).

Nick, in half an hour you just made up every single JavaOne conference I attended ;-)

@rubyfringe - Owning what you do

2008-07-19T20:02:00Z

A major theme, concept, or understanding here at “Ruby Fringe”http://www.rubyfringe.com amongst folks seems to be owning what you do. Not that it’s expressed this way – but I think a lot of what we’re discussing here comes down to owning your work, owning how you spend your time, owning your mistakes and growing from there. From Fail Camp to discussion on sales and business relationships to networking and how you spend your time, to the slogan on the back of the conference tags, “Ask me about my startup!”, this is about owning what you do and consciously running with it.

Quite frankly, it’s exhilarating. This is not a normal programming conference.

@rubyfringe - Go For It, Dude

2008-07-19T15:13:00Z

First two talks were great (Jay Philips on Adhearsion, Dan Grigsby on hacking the market, basically). While on different topics altogether, both were plugging a common theme here: keep coming up with new ideas, set out on your own, and if you look for the niches and cracks you can fill in the market, you can exploit/hack those efficiencies to your advantage and really make it as an indie with a few friends and not work for the man, so to speak.

Which we already knew (or at least hope) is the case, but it’s awesome having these folks sharing their experiences, ideas, and methods of doing it themselves. It’s basically a lot of people who have that kind of “hack everything” mentality that is the focus of the HOPE/2600 folks and apply it to the question, “how can I set out for myself as a hacker and make a living?”

Also: the memcached talk was a totally excellent geek-out on caching.

Next talk is called “Living on the Edge,” and it’s about what’s going on in Merb edge development, but you see how it fits the broader theme by title.

There’s a great community here in Toronto, looking forward to meeting some New Yorkers and seeing if we have this kind of community back in NYC and Jerz.

More to come.