subWindow Blog

Ruby Module Reference

erik@subwindow.com (Erik Peterson) — Mon, 17 Nov 2008 20:46:33 GMT

In Ruby, I always get confused as to the difference between include and extend with modules, and the effect of class << self inside those. So here's a reference script as to what gets included where:

module ExtendedMod
  def a
    "a"
  end

  class << self
    def b
      "b"
    end
  end
end

module IncludedMod
  def c
    "c"
  end

  class << self
    def d
      "d"
    end
  end
end

class Klass
  def e
    "e"
  end

  class << self
    def f
      "f"
    end
  end
end

class SubKlass < Klass
  def g
    "g"
  end

  # Skip h (ERB::Util.h)
  class << self
    def i
      "i"
    end
  end
end

Klass.extend ExtendedMod
Klass.send(:include, IncludedMod)

[:a, :b, :c, :d, :e, :f, :g, :i].each do |m|
  puts "Klass responds to #{m}? #{Klass.respond_to?(m)}"
  puts "Klass.new responds to #{m}? #{Klass.new.respond_to?(m)}"
  puts "SubKlass responds to #{m}? #{SubKlass.respond_to?(m)}"
  puts "SubKlass.new responds to #{m}? #{SubKlass.new.respond_to?(m)}"
  puts "\n"
end

And the output:

Klass responds to a? true
Klass.new responds to a? false
SubKlass responds to a? true
SubKlass.new responds to a? false

Klass responds to b? false
Klass.new responds to b? false
SubKlass responds to b? false
SubKlass.new responds to b? false

Klass responds to c? false
Klass.new responds to c? true
SubKlass responds to c? false
SubKlass.new responds to c? true

Klass responds to d? false
Klass.new responds to d? false
SubKlass responds to d? false
SubKlass.new responds to d? false

Klass responds to e? false
Klass.new responds to e? true
SubKlass responds to e? false
SubKlass.new responds to e? true

Klass responds to f? true
Klass.new responds to f? false
SubKlass responds to f? true
SubKlass.new responds to f? false

Klass responds to g? false
Klass.new responds to g? false
SubKlass responds to g? false
SubKlass.new responds to g? true

Klass responds to i? false
Klass.new responds to i? false
SubKlass responds to i? true
SubKlass.new responds to i? false

The takeaways? Don't use class << self in modules. Always put class methods and instance methods in different modules. Use extend for class methods and include for instance methods. I had already "learned" this several times, but for some reason I couldn't remember it until I put this thing together.

Places where Rails is not DRY

erik@subwindow.com (Erik Peterson) — Fri, 05 Sep 2008 19:23:37 GMT

As I get more and more involved in Rails, I keep finding places where it is definitely not DRY. When adding, removing or changing something, I never have complete confidence that I have checked every place where it could be referenced. This is always a sure sign that something is amiss with the framework. The following is kind of a running list of un-DRY thorns that I keep running into while developing Rails applications:

Migrations: Specify both up AND down.
Migrations->Validations: Specify column creation in migration and validations.
Associations: Specify column creation in migration and association in model.
Validations->View: Specify validations in model and specify which ones are required in the view (red-asterisk problem)
Migrations/Model->Show view: Specify columns in migration and in the "Show" view.
Migrations/Model->Index view: Specify columns in migration and in the "Index" view.
Migrations/Model->Form view: Specify columns in migration and in the "Form" view partial (edit/new).
Migrations/Model Associations->Show view(s): Specify an association in the migration, in the model, and in the views for BOTH the child AND the parent.
Model "Human Identifiers": Human-recognizable triggers for a specific record ("Name" for Users, "Title" for books, etc). Specify column in migration and potentially dozens of places in the view and/or helpers.
Controllers: The vast majority of my controllers are cookie-cutter. Update is almost always find->update_attributes->if no error, redirect to show/if error render edit.

Can anyone think of any others?

The particularly sticky part of this is that these all stack. With an important column you end up having its specification in potentially dozens of places, which is completely unmaintainable. By "Specification" I mean its name, its data type, its associations, its validations, how it is displayed in a show/index, how it is displayed in a form, and search characteristics. Is it possible to keep this all in one place? I'd be satisfied with two places, which is what I'm targeting (migration and "Presenter").

Although there are some existing (and hacky) workarounds to a few of these problems. I look forward to them being solved in a cohesive manner. Specifically, I'm working on #4-10, and my coworker Jordi is working on 1-3. Is there anyone else working on these problems? DataMapper and Merb come to mind, but so far I'm not satisfied that they solve the problems correctly. That doesn't mean we can't learn from them, though.

Summer Rails Plugin Series #3: acts_as_referenced

erik@subwindow.com (Erik Peterson) — Wed, 23 Jul 2008 05:31:45 GMT

Next up in my summer series of plugins is acts_as_referenced. This one is a bit small, but I think it can be extremely useful. This is a versatile and reasonably well-tested plugin that I'm using every day. On my main project, I have about 8 models that use reference numbers- five of them use the same numbering scheme, and the other three use different ones. I needed a way to DRY up the assignment of these reference numbers while creating a central search index (GREAT for barcoded reference numbers), allowing for legacy reference numbers AND having them be valuable to a human reader by including the date and object type in the reference number.

Installation

If you are on edge rails:

script/plugin install git://github.com/subwindow/acts_as_referenced.git

If you are not on edge rails:

git clone git://github.com/subwindow/acts_as_referenced.git vendor/plugins/acts_as_referenced

Usage

To make use of acts_as_referenced, you first need to create the Reference model & its table:

script/generate reference reference
rake db:migrate

You also need a column 'reference' in all of the tables you plan on using acts_as_referenced with. (You can change this with the :referenced_column option)

Enable the functionality by declaring acts_as_referenced on your model

class Order < ActiveRecord::Base
  acts_as_referenced
end

Advanced Usage

Advanced usage is enabled through a set of options. Here they are:

# Separator
acts_as_referenced # Defaults to '-', e.g.: '080722-1'
acts_as_referenced :separator => "" #e.g.: '0807221'
acts_as_referenced :separator => "_" #e.g.: '080722_1'

# Prefix.  A string that is appended at the front of the reference number.  Does not affect incrementing.  Accepts a proc.
acts_as_referenced # Defaults to '', e.g.: '080722-1'
acts_as_referenced :prefix => "OR" # e.g.: "OR-080722-1"
acts_as_referenced :prefix => Proc.new {|o| o.category.first.upcase } # e.g.: "M-080722-1"

# Increment base.  A string that is used as the base for incrementing- all reference numbers with the same base will be incremented together.  Accepts strftime arguments.  
acts_as_referenced # Defaults to "%y%m%d", e.g.: "080722-1"
acts_as_referenced :increment_base => "#{(65+(Time.now.year-2003)).chr}%m%d" # e.g.: "F0722-1"

# Increment size.  An integer that is used to pad incrementers.  Useful if you require reference numbers with a consistent legnth.
acts_as_referenced # Defaults to 0, e.g.: "080722-1", "080722-9999"
acts_as_referenced :increment_size => 4 #e.g.: "080722-0001", "080722-9999"

# Referenced Column.  Change the name of the "reference" column that is stored on the model.
acts_as_referenced # Defaults to 'reference'
acts_as_referenced :referenced_column => "order_number"

A Note on Racing

There's a racing condition inherent in the plugin as it is shipped. If there are two processes creating references at approximately the same time, it is entirely possible that they will collide and create reference numbers at the same time. Rails may or may not catch this, so the possibility for duplicate reference numbers exists, and the possibility for confusing end-user errors is even greater. To solve this, you have to lock the table from reads while creating a reference number. It should only take < 50ms, so in most cases it is not that big of a deal. The locking procedures are different for different DBMSs. For PostgreSQL, uncomment lib/acts_as_referenced.rb line #91. For MySQL, uncomment lib/acts_as_referenced.rb lines #93 and #99. For other systems, simply write your own table locking/unlocking SQL and place it in the same spots.

Conclusion

As always, I hope someone can find this plugin useful. Please let me know if there are any bugs or you are confused by something. My normal use-case is a bit esoteric, so if you have questions on how this might be useful, let me know in a comment and I can try to explain it better. For any other questions or comments, email me at erik [at] subwindow (dot} com. To contribute code, fork it at github.

Summer Rails Plugin Series #2: needs_approval

erik@subwindow.com (Erik Peterson) — Wed, 09 Jul 2008 06:54:26 GMT

Ok, so I know I promised a Rails plugin every week. About a week after I made that promise I learned something: making a Rails plugin every week is fucking impossible. Even if you've already written the code and all you need to do is extract it. It is impossible. I'd be lucky to fit in one plugin a month. Still, one plugin a month for 6 months is still a pretty decent clip, so I'd be satisfied if I can hold myself to that (relaxed) schedule.

Anyways, next up is a plugin called needs_appoval. This is a pretty simple plugin designed to help you manage approval flows. If you have records that need to be approved by users, this plugin might help you out. It also has the following features:

Ladder-based approvals (where one user can only approve after the users listed previous to him have approved as well)
Approval requirement conditions (where a user only needs to approve on certain conditions)
Optional, built-in authentication (where a user must enter in their password whenever they approve something- often required by 21 CFR Part 11 and ISO 9001)

Instructions

Install using the command

script/plugin install git://github.com/subwindow/needs_approval.git

Generate a scaffolded approval model/controller using the command

script/generate approval APPROVALMODEL USERMODEL [--include-authentication] [--skip-migration]

Example

script/generate approval approval user

Put the approval structure definition in your desired models

class Order < ActiveRecord::Base
    needs_approval do
      of User.find_by_login("boss")
      of User.find_by_login("bigboss") if total_price > 1000
    end
  end

For ladder-based approvals, use the following. The function next_approvers goes in order of approval definition.

needs_approval(:ladder => true)

Include approval information in the show page of an approved object

<%= approvals_for(@order) %>

Other useful functions:

# Boolean, returns true if all approvals have been satisfied
@order.has_all_approvals?

# Returns an array of all users that are required to approve this object
@order.approvals_needed

# Returns an array of users that have approved this object
@order.approvals

# Returns an array of users that need to approve this object, but have not yet
@order.failed_approvals

# Boolean, returns true if this object requires and does not yet have the approval of this user
@order.needs_approval_of(user)

# Boolean, returns true if this user has approved this object
@order.passes_approval(user)

# Returns an array of users that are set to approve this next (for ladder-based approvals)
@order.next_approver

Note

This plugin kind of depends on restful_authentication or something similar. A "current_user" method must be accessible which returns the currently logged in user.

If the current_user function is named differently, either:

define an alias, or
change the references in the approvals controller and pass in the current user as a second argument to the approvals_for helper [eg: "approvals_for(@order, active_user)"]

An example app is located in the directory example_app. You can delete this if you don't want it (especially if you use TextMate-style "Go To File"). To run, execute the following commands:

cd vendor/plugins/needs_approval/example_app
script/plugin install git://github.com/subwindow/needs_approval.git
rake db:create
rake db:migrate
rake db:fixtures:load
script/server

Conclusion

I hope somebody can find this helpful. This was originally a much larger plugin (>100 lines; currently ~50), but in the process of extracting it I refactored the code and made it much smaller. This means it doesn't seem as useful, but at the same time it was very useful to myself because I end up with a much nicer codebase. As usual, If you have any questions, concerns or have found any bugs, please email me at erik [at] subwindow dotcom. Or, reply here. Or, go to the github page and fork the code.

Next up: acts_as_filtered. A plugin to assist in dynamically filtered models. The code is very mature and in production use, but it might need to be refactored and generalized a bit. Hopefully I'll get it out before the end of the month.

What You're Doing is Not New, Not Revolutionary

erik@subwindow.com (Erik Peterson) — Sat, 05 Jul 2008 18:44:08 GMT

In 1971, Mark Zuckerberg was negative 13 years old. The phrase "social network" would have returned awkward silences and weird stares, even amongst the savviest tech entrepreneurs. If you had said the word "Blog" people would have thought you were coughing something up. Yet, in their own weird little ways, thousands of people were social networking and blogging.

In my post Wednesday, I said:

We tend to live in our own little bubbles and think that somehow, our problems are brand new, and that they've never been solved before. Well, guess what: almost all of them have been solved before, have been solved better, and are included in almost all Unix systems out there.

I've found that this maxim is almost universally applicable to movements in technology. I've there's some big new thing, it has probably been done before, and it has probably been done better, and it was probably included in Unix.

finger is to Friendster as Friendster is to Facebook

The finger Unix program was developed in 1971 by Les Earnest. It provided a way for you to figure out what other people were up to. You'd execute the command "finger user@domain" and you'd get all sorts of useful information back: The user's phone number, how long they've been idle, how long it's been since they've checked their mail, and the contents of their .project and .plan files. Here's what it looks like if you execute "finger erik@subwindow.com":

[subwindow.com]
Login: erik           			Name: 
Directory: /home/erik               	Shell: /bin/bash
On since Sat Jul  5 14:11 (UTC) on pts/0 from c-98-242-74-4.hsd1.ga.comcast.net
   3 minutes 9 seconds idle
No mail.
Project:
Current Projects:
  -Super Secret project.  Relates to http://subwindow.com/articles/14
  -Rails plugin for managing approvals.  Was called acts_as_approved, but I think I'm renaming it.

Projects in Stasis:
  -Aloe: A long-term work in progress. Aloe (or "A-l=o.e": Assets - Liabilities = Owner's Equity) will
    be an easy-to-use online accounting system. It is designed for the small business owner that does
    not specifically have any skills in accounting. It is currently in pre-alpha.  http://aloe-acct.com/
  -Skribit: Formed as part of Startup Weekend Atlanta, I'm an ongoing contributor to this social suggestion
    engine for bloggers.  Contributions are erratic, based on the activity of those in the "Current Projects"
    section.  Others are more active than me, so the entire project is not in stasis, mind you.

Projects in Maintenance:
  -Forecaster.ws A simple and uncluttered weather information service, tailored specifically for mobile
    devices. Currently in late public beta.
  -PropertyBuilder A proposed alternative to Object#andand and the ilk. Available through the
    'gem install propertybuilder' command.
  -Zsff A parser/validator for the ZSFF feed format. Version 1.0 released as a gem and is available on
    Rubyforge or through the 'gem install zsff' command
  -Rhobbler A Last.fm / AudioScrobbler track submission tool for Rhapsody. Currently in public beta.

Plan:
Today, I'm going to have a 4th of July party/Kelly's Birtday party.  But it's on the 5th.  Because even parties
    can be fashionably late.

This week: Super-secret project continues.  On Tuesday I hope to release my approval Rails plugin.

Hmm. That looks awfully similar to some other services I use heavily. It's pretty much the amalgamation of my Facebook page, my blog, and my tweets. And this was available in 1971!

The Trouble With finger

You may be wondering "If finger was so great, how come I've never heard of it, and how come we aren't using it now?" Well, finger had a lot of problems, unfortunately. Here's my understanding of what caused the downfall of finger (Also note that I was negative 12 years old when finger was invented, and about 10 when most people stopped using it):

It depended on a doomed computing style. The only way finger really made sense was if everyone at an organization regularly logged into the same physical Unix machine. This made a ton of sense in the mainframe era, but finger's popularity was directly proportional to the popularity of the mainframe.
There were no privacy protections. finger was notorious for being used as a tool for crackers and other nefarious characters to gain information. If you fingered the root of a domain (ie: 'finger @gatech.edu'), you'd get a listing of every user in the system- their phone numbers, and what they were up to. This information is kind of sensitive to just be putting out into the ether. Subsequently, fingering was seen has kind of a rude thing to do to someone unless you knew them and had a justification for seeing what they were up to. This didn't sit well with lots of people and contributed greatly to its demise. finger had no privacy features like Facebook now has. If it did (ie: Here's who my friends/coworkers are, let them see this information. Only display this subset to other users), I think it might still be in heavy use today.
finger was a protocol, and not a platform. There was no party responsible for carrying it forward or for providing necessary upgrades. When Facebook users complained about too much information being available to the public, they instituted incredibly granular privacy controls. When finger users complained about the exact same thing, there was either nobody to complain to, or they said "Well, turn it off." So they did.

Lessons are for Hares

Of course, there's some valuable lessons to be learned here. The main one is that you should always make more of an effort to see what's been done before you. The makers of Facebook shouldn't have just learned about Friendster, but they should have made more of an effort to learn about the earlier ancestors of social networking. If they had looked for and learned about the reasons for finger's demise, they might have avoided about half of the major confrontations that they've had with their users. With a full understanding of history, you can better realize that you are not a unique butterfly, and that your idea is not really that new. You can better gain perspective and learn the lessons of your forebears. This is something that's not unique to technology. You should apply this maxim everywhere in life- especially politics.

Background Processing in Rails Using 'at'

erik@subwindow.com (Erik Peterson) — Thu, 03 Jul 2008 02:13:10 GMT

So here's a fairly typical problem that happens in the Rails world: you have a long-running process that you need to offload/queue up, and you don't have time to fuck around with some kind of elegant solution. Before, either I'd just make the user eat it and wait for the long request (at the risk of timing out the request and/or pissing off the user), or figure out some way to procrastinate on implementing the feature in the first place.

But alas, I came to a situation that absolutely had to be offloaded: a 50MB file import that takes about 3-5 minutes to process. The request is guaranteed to time out, and it is something that will be done fairly regularly. I'd love to tell my users "Hey, this shit is going to break, but it will be the best and most awesome breaking you've ever seen," but somehow I don't think that would fly. Damn. I don't want to screw around for two days trying to cook up some kind of "scalable" solution involving worker processes and messaging queues. I don't need scalability (this will be run 2 or 3 times a week, max), I just need it to work.

When faced with a problem like this, I think web developers in general underestimate the massive amounts of thought and effort that have gone into the systems that we use every day and generally take for granted. We tend to live in our own little bubbles and think that somehow, our problems are brand new, and that they've never been solved before. Well, guess what: almost all of them have been solved before, have been solved better, and are included in almost all Unix systems out there.

Enter `at`

If you have a command, and want to have it wait a tick before it starts working, and want the work of several jobs to be handled in some kind of sane fashion, at is the perfect solution. Like most things Unix, at is pretty damned simple. You have a set of commands in some kind of a file. You want to execute them at some date in the future. You send those to at:

at -t 07021824 -f /path/to/my/commands

. That's pretty damned beautiful.

Now, putting it in Rails is actually damned easy. I wrote a 3-line method in my ApplicationController so that I can offload any arbitrary command. Take a look at this awesome sauce:

def offload(command)
  job_id = MD5.hexdigest("#{command}+#{Time.now.to_i}+#{$$}")
  `echo "#{RAILS_ROOT}/script/runner -e #{RAILS_ENV} \\"#{command}\\"" > /tmp/#{job_id} &&
   at -t #{1.minute.from_now.strftime("%m%d%H%M")} -f /tmp/#{job_id}`
end

Every time I need to offload some arbitrary command, all I do is pass it to offload:

offload("Part.import_bom('#{massive_file}')")

. Simple as pie.

Scaling? You say you want scaling? You have 3 application servers that you want to dish these offloaded processes to as equitably as possible? You are also as lazy as I am? Awesome:

APP_SERVERS = ["172.16.200.50", "172.16.200.51", "172.16.200.52"]
def offload(command)
  job_id = MD5.hexdigest("#{command}+#{Time.now.to_i}+#{$$}")
  `ssh user@#{APP_SERVERS[rand(APP_SERVERS.size)]} "echo \\"#{RAILS_ROOT}/script/runner -e #{RAILS_ENV} \\\\"#{command}\\\\"\\" > /tmp/#{job_id} &&
   at -t #{1.minute.from_now.strftime("%m%d%H%M")} -f /tmp/#{job_id}"`
end

Now you've got your distributed job queue implemented. Three lines of code. Win.

Sidenote: If your dev machine is running OSX like mine is, and you want this to work, you're going to have to run the command sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.atrun.plist and restart before this stuff will work locally.

English as Code as Pseudocode

erik@subwindow.com (Erik Peterson) — Sat, 28 Jun 2008 21:02:54 GMT

The Tradition of Pseudocode

Traditionally, when developers got together to talk about code, they would invariably end up writing pseudocode on the nearest white board, piece of paper or dinner napkin. The reasons for this are myriad. First, developers favor different languages, and might not be able to read their friend's favorite language. Especially 15 or 20 years ago, languages used to be so stylistically different that a developer who hadn't already had exposure to, say, Algol or Fortran, might not be able to easily parse even basic code statements. The second main reason for the use of pseudo code is that 20 years ago, most languages didn't read like English. If you wanted to really describe how a program worked, you would have to use pseudocode, because all of the programming languages you knew read like shit.

That's not such a big deal these days, also for a couple of reasons. First, the languages currently in favor are much more high level and all have much more sophisticated Object models than the ones of yore. You generally don't have to explicitly allocate memory, declare variables, or do other kinds of tedious dreck. With more refined OOP concepts, you (generally) don't have to write as much boilerplate or worry about dumb gotchas as you had to with antiquated object models like Java or Perl's sad excuse for OO. Secondly, most newer languages have gone a long way towards actually reading like English. Ruby and Python, in particular, have both eschewed the use of unnecessary tokens (braces in particular) and made their calling syntax much more natural.

As a result, you don't see much pseudocode being written these days. It used to be that anytime you would talk about algorithms, you would use pseudocode. Now? not so much.

Taking it a bit Further

So, since modern languages read so well that they have essentially supplanted pseudocode, should we stop there? Is it good enough that a programming language can be readily understood by any novice? I say no. I think we shouldn't stop pressing the development of languages until the concepts that the code is portraying cannot be expressed any clearer or more succinctly than they already are. That seems like a pretty hefty goal- to have a programming language that expresses things so clearly and succinctly that there is literally no room for improvement. We've generally seen the succinct side maximized- I would argue that Perl takes the cake on that one, and it is unlikely that there will ever be a language as succinct as it is. But Perl focused on succinctness at the expense of clarity. That's not really necessary.

So our goal is maximum succinctness and clarity. How far away are we? Surprisingly, not as far away as I had thought. Certain things have already hit this "wall of awesomeness"- domain-specific languages in particular. This following code snippet of the Rails plugin acts_as_state_machine does a great job of explaining both why I love DSLs so much, and why they're our best hope for accomplishing our goals:

acts_as_state_machine :initial => :pending
state :pending
state :submitted

event :submit do
  transitions :from => :pending, :to => :submitted
end

Try reading that code. Literally read it. It should read something like: "This acts as a state machine, with an initial state of 'Pending'. It has two states: 'Pending' and 'Submitted'. It has an event called 'submit' which transitions from pending to submitted." You'll notice something about reading that- the code is actually more clear and more succinct than the English representation of the code. Drawing a diagram of this small state machine would take longer and possibly be less clear than the code sample itself. When code is both clearer and shorter than either English or a drawing, you know you've won.

Are We There Yet? Are We There Yet? No!

That one example explains why I'm such a big fan of Ruby. It encourages code to both read like English and to read better than English. Whenever I come across a common pattern in my code, I find it extremely easy to abstract out that pattern into a DSL and bask in the clarity. This usually achievable, but isn't always easy, as in this example:

needs_approval do
  of department.manager
  of User.cfo if total_price > 5000
end

This code comes from a plugin that I was working on when I originally published this article. This code specifically deals with an approval structure for a Purchase Order. Each purchase order needs the approval of the Department Manager and, if the total price of the PO is over $5000, the CFO. This is actually pretty good.

I updated this article on July 17th, after I released the plugin. I was able to figure out a way to clean up the syntax quite a bit, and remove all of the brackets. I thought that I needed Ruby to have more power (specifically, being able to pass blocks around willy-nilly), but really I just needed to work hardt to create some nice syntax. So are we there yet? Maybe. I might be.

Summer Rails Plugin Series

erik@subwindow.com (Erik Peterson) — Wed, 11 Jun 2008 05:44:10 GMT

I apologize for my extremely lengthy blogging break. But I'm back! I promise! I've got quite a backup of code that I want to release, and as part of that, I promise to release a Rails plugin each week for the rest of the summer. That's quite a hefty promise, but I've got June mapped out, at the very least. We'll see where it goes from there. First up in my series of Summer Rails Plugins is negative_captcha:

What is a Negative Captcha?

A negative captcha has the exact same purpose as your run-of-the-mill image captcha: To keep bots from submitting forms. Image ("positive") captchas do this by implementing a step which only humans can do, but bots cannot: read jumbled characters from an image. But this is bad. It creates usability problems, it hurts conversion rates, and it confuses the shit out of lots of people. Why not do it the other way around? Negative captchas create a form that has tasks that only bots can perform, but humans cannot. This has the exact same effect, with (anecdotally) a much lower false positive identification rate when compared with positive captchas. All of this comes without making humans go through any extra trouble to submit the form. It really is win-win.

How does a Negative Captcha Work?

In a negative captcha form there are two main parts and three ancillary parts. I'll explain them thusly.

Main Part #1: Honeypots

Honeypots are form fields which look exactly like real form fields. Bots will see them and fill them out. Humans cannot see them and thusly will not fill them out. They are hidden indirectly- usually by positioning them off to the left of the browser. They look kind of like this:

<div style="position: absolute; left: -2000px;"><input type="text" name="name"  value="" /></div>

Main Part #2: Real fields

These fields are the ones humans will see, will subsequently fill out, and that you'll pull your real form data out of. The form name will be hashed so that bots will not know what it is. They look kind of like this:

<input type="text" name="685966bd3a1975667b4777cc56188c7e" />

Ancillary Part #1: Timestamp

This is a field that is used in the hash key to make the hash different on every GET request, and to prevent replayability.

Ancillary Part #2: Spinner

This is a rotating key that is used in the hash method to prevent replayability. I'm not sold on its usefulness.

Ancillary Part #3: Secret key

This is simply some key that is used in the hashing method to prevent bots from backing out the name of the field from the hashed field name.

How does the Negative Captcha Plugin Work?

Install the plugin:

With Subversion

script/plugin install http://code.subwindow.com/negative_captcha

With git

git submodule add git://github.com/subwindow/negative-captcha.git vendor/plugins/negative_captcha

Add in the controller hooks

#At top:
before_filter :setup_negative_captcha, :only => [:new, :create]

#At bottom:
private
  def setup_negative_captcha
    @captcha = NegativeCaptcha.new(
      #A secret key entered in environment.rb.  'rake secret' will give you a good one.
      :secret => ::NEGATIVE_CAPTCHA_SECRET, 
      :spinner => request.remote_ip,
      #Whatever fields are in your form  
      :fields => [:name, email, body],
      :params => params)
  end

Modify your POST action(s) to check for the validity of the negative captcha form

def create
  if @captcha.valid?
    @comment = Comment.create(@captcha.values) #Decrypted params
  end

  if @captcha.valid? &amp;&amp; @comment.valid?
    redirect_to @comment
  else
    flash[:notice] = @captcha.error
    render :action => 'new'
  end
end

Modify your form to include the honeypots and other fields. You can probably leave any select, radio, and check box fields alone. The text field/text area helpers should be sufficient.

<% form_tag comments_path do -%>
  <%= negative_captcha(@captcha) %>
  <ul class=&quot;contact_us&quot;>
    <li>
      <label>Name:</label>
      <%= negative_text_field_tag @captcha, :name %>
    </li>
    <li>
      <label>Email:</label>
      <%= negative_text_field_tag @captcha, :email %>
    </li>
    <li>
      <label>Your Comment:</label>
      <%= negative_text_area_tag @captcha, :body %>
    </li>
    <li>
      <%= submit_tag %>
    </li>
  </ul>
<% end -%>

Test and enjoy!

Possible Gotchas and other concerns

It is still possible for someone to write a bot to exploit a single site by closely examining the DOM. This means that if you are Yahoo, Google or Facebook, negative captchas will not be a complete solution. But if you have a small application, negative captchas will likely be a very, very good solution for you. There are no easy work-arounds to the dom-predictability problem quite yet. Let me know if you have one.
I'm not a genius. It is possible that a bot can figure out the hashed values and determine which fields are which. I don't know how, but I think they might be able to. I welcome people who have thought this out more thoroughly to criticize this method and help me find solutions. I like this idea a lot and want it to succeed.

Credit

The idea for negative captchas is not mine. It originates (I think) from Damien Katz of CouchDB.
I (Erik Peterson) wrote the plugin. Calvin Yu wrote the original class which I refactored quite a bit and made into the plugin.
We did this while working on Skribit, an Atlanta startup (if you have a blog, please check us out at http://skribit.com/)
If you have any questions, concerns or have found any bugs, please email me at erik [at] subwindow dotcom
If you'd like to help improve or refactor this plugin, please create a fork on Github and let me know about it. http://github.com/subwindow/negative-captcha

Those Pesky Expectations

erik@subwindow.com (Erik Peterson) — Sat, 10 May 2008 05:26:04 GMT

I live in Atlanta. A major city in one of the most prosperous nations in the world. I can't get WiMax.

You know where you can get WiMax service? Ghana. A third world country in Africa.

I was pondering a vacation out there just after the rainy season- December or so- and was wondering about Internet access in the country. I figured I'd be lucky to catch DSL-like speeds once in a blue moon. It turns out that not only can I get DSL-like speeds, but I can get them with a mobile device, and I can get them for almost half the cost of my (slower) CDMA wireless card.

Be careful with the expectations you carry about other people and places. Just because you assume you're in a good position, doesn't mean that you actually are. Challenge your expectations every once in awhile. Go to Ghana. See how fast and cheap their Internet access is.

Sometimes, the little things count

erik@subwindow.com (Erik Peterson) — Mon, 28 Apr 2008 22:36:16 GMT

class TrueClass
  def to_exclamation
    "Yes"
  end
end

class FalseClass
  def to_exclamation
    "No"
  end
end

(1+2 == 3).to_exclamation
=> "Yes"
(1+1 == 4).to_exclamation
=> "No"

This demonstrates two things:

Small things can matter. Even something that is seemingly inconsequential can turn into something pleasant.
Just because something (IE monkeypatching) can be harmful, doesn't mean it always is.

Thinking about yield(self)

erik@subwindow.com (Erik Peterson) — Thu, 10 Apr 2008 20:50:33 GMT

Maybe my brain is broken, but when using blocks in Ruby, I keep wanting to have access to the scope of the calling object:

class Foo
  def bar
    baz = "qux"
    yield
  end
end

>> Foo.new.bar { puts baz }
NameError: undefined local variable or method 'baz' for main:Object

Which makes sense, of course. But what if I really needed to know what baz was, in the context of the block? (Assume that I'm determining my accessing needs at runtime, and can't add arguments willy-nilly to the yield call). The solution I came up with was to use yield(self). It seems kind of dirty, and it is- but it works. Mostly.

class Foo
  attr_accessor :accessible_baz
  def bar
    baz = "qux"
    @instance_baz = "qux"
    @accessible_baz = "qux"
    yield(self)
  end
end

#Attempt to get the local variable
>> Foo.new.bar {|container| puts container.baz }
NameError: undefined local variable or method 'baz' for main:Object

#Attempt to get an instance variable without an accessor
>> Foo.new.bar {|container| puts container.instance_baz }
NoMethodError: undefined method 'instance_baz' for #
>> Foo.new.bar {|container| puts container.instance_variable_get(:@instance_baz) }
qux

#Attempt to get an instance variable with an accessor
>> Foo.new.bar {|container| puts container.accessible_baz }
qux

I think this might be an OK solution if you're just trying to get at methods or accessible instance variables of the calling object, but I'm not sure it is worth it to go through instance_variable_get in order to get other instance variables. Is there a better way to get at this stuff? Any method that is more elegant?

The ERP Money-Back Guarantee

erik@subwindow.com (Erik Peterson) — Wed, 09 Apr 2008 06:39:41 GMT

In my last post, I detailed some of the problems with the existing ERP vendor ecosystem. In this post, I want to go over why these ERP vendors fail, and what would be necessary for a successful ERP implementation.

The problem is realizing that if the software to automate a business process is complicated and contradictory and hard to use, then the real cause is a business process that is complicated and contradictory and probably not serving the company well.

--Raganwald

Reginald (can I call him that?) hits the nail on the head. If you haven't read that article, you really should- it does quite a good job at describing this problem.

Applied to ERP systems, the problem is that ERP vendors strive to have an "out-of-the-box" solution, that with "minimal customization" can be applied to any business. What an incredible crock of shit. This, right here- that one sentence, describes the failure of these ERP vendors more than anything. Sure, the salesmen are slimeballs and they send out junior programmers to wrestle these huge systems into conformance. But this is their one grevious error. Trying to pretend that putting an ERP system into place is anything less than a complete reinvention of nearly every operating process of the business is a disservice.

All too often, when trying to automate a business process, software developers take a look at the current (non-automated) system, and merely replicate that in software. This usually entails complicated approval loops, unnecessary data entry, and human gating. This is so, so, so wrong. The problem with the current state of most business software is it asks the question What? when it should be asking the question Why?

A common scenario in business software is requiring approval for something. Purchasing, in particular, is fraught with this bug. Most organizations have a system set up where purchases need an increasing amount of approval for an increasing dollar value. This makes sense, right? You don't want someone buying $100,000 worth of materials completely unchecked. In this situation (and in most approval/gating situations), the process needs to be switched from an approval situation to a monitoring situation. Instead of requiring the approval of the COO for purchases over $50,000, why not shoot him an email detailing the purchase, and he can intervene if something is fishy? The same objective is attained- notification and approval, but this time it is implicit approval, which is no less powerful.

This kind of rethinking of the method is beneficial from both sides. The business benefits because the process is much less manual- any time you can remove a human step from a process the whole organization becomes more efficient. It also benefits the software developer because, all of a sudden, the software is much more simple. Instead of an email notification, an approval/rejection interface, and the appropriate state gating, now you just have the email notification (and maybe a rejection interface). This may seem like a small win, but replicate this win 100 times over the course of an implementation, and the system as a whole is dramatically smaller (and we all know, smaller code bases have fewer bugs), and the processes mapped are dramatically more efficient.

In general, the process of implementing an ERP system is considered a one-way street. The business comes up with requirements, and the vendor/consultant comes up with a system that (theoretically) fulfills those requirements. It should be a two-way street. No, screw that, it should be a freaking cul-de-sac. The business and developers need to work together to create a set of requirements that doesn't map the processes as they are, but maps the processes as they should be.

This might seem like a sketchy proposition from the developer's perspective- "Who am I to tell these guys they need to rework all of their processes?" I don't think this is a problem as long as it is established from the front. The people selling the service need to be frank about the need to rework every process that the ERP touches from the ground up. It might be a hard sell, but once you can demonstrate the advantages of this approach (and the perils of not reworking processes), I think companies might be willing to play ball.

And playing ball is just what this is- the vendor and the client really should be on the same team. I know this sounds like management dreck, but I'm serious. Instead of the developers fighting the clients to come up with requirements and the clients fighting the developers to implement them, everyone needs to understand that it is in each other's best interests to help the other succeed.

Why not take it one step further? Why don't ERP vendors say at the start of the project: "If you're not happy with us a year from now, we'll give you your money back." That sentence sounds like sacrilege in the current environment, but why not have a money-back guarantee? After all, almost every single failure scenario is essentially the fault of the vendor. The common defense for vendors is that the client didn't specify the requirements correctly, or some variation thereof. Hogwash. It is the responsibility of the vendor to know that the requirements the client gives them are worthless. It is the responsibility of the vendor to cut through that and figure out what the real requirements are, and to work with the client to make sure that their processes are going to take full advantage of the system.

An ERP vendor that isn't willing to offer a money-back guarantee is one that has no financial interest in the success of the implementation. After all, a failed implementation that takes two years will probably net them more money than a successful one that lasts 15 months. Shortening this:

An ERP vendor that isn't willing to offer a money-back guarantee is one that doesn't care if the implementation succeeds.

That's absurd. I'm really astonished that the market has gotten this far without realizing this. A credible company that comes into this market that offers this will not only win, but will win in the kind of way that changes the entire industry.

I'm game for that.

Enterprise Resource Planning: Mass epic... Opportunity.

erik@subwindow.com (Erik Peterson) — Fri, 28 Mar 2008 06:03:28 GMT

I've been thinking about Enterprise IT for a long time, but haven't really verbalized my thoughts. After reading and commenting on a post on Obie's blog, I think I'm ready to put some of these thoughts down.

Any sufficiently large business needs some kind of an ERP- a cohesive software system that essentially manages all of the data of the business. Doing without an ERP system is possible, but leads to a crazy amount of excel spreadsheets and paper pushers. Efficiency at a large scale is, nearly without exception, impossible without such a system. [Note that whether it is called an ERP or not, some kind of cohesive data management system is necessary. Google, for instance, essentially has an ERP- they just don't call it that.] There are certain industries where a business can survive without this kind of a system, but obviously companies in these industries will left out of the conversation.

Also, ERP implementations are notoriously expensive. For a large business (>50M/yr revenue), ERP implementations regularly pass the $5M mark. Estimates for the cost of these things are usually a little lower- $2 or $3 million seems to be the common estimate. However, once the vendors screw the pooch enough, businesses will be lucky to get out of there losing less than $10M, all things considered. ERP "vendors" are notoriously bad, in every sense of the word. They are slow, they are liars, they are expensive, and they fail way too often.

Some research has been done on ERP implementation failure rates, and the common wisdom is that only 30% of ERP implementations succeed. Even the rosiest studies propagated by ERP vendors talk about 50% success rates. Can this possibly be true? Take a look at this from the perspective of a business:

I need this system or else my business is likely to die due to inefficiency. Implementing such a system will cost five million dollars and take about two years. At the end of this five million dollars and two years, I have a 70% chance of walking away with nothing.

Man. That sounds like a really raw deal. If I were running a large company, I would seriously resist taking such a risk. It is no wonder most large businesses are still pinnacles of inefficiency- getting rid of the fat is a slow, expensive, and insanely risky proposition.

It goes even beyond this, however. Even if these ERP implementations are successful- i.e., the business somehow doesn't declare bankruptcy as a result of the ERP implementation- the systems are still really, terribly awful. They are horribly unusable systems that are slow, impossible to change, and a huge burden on 90% of the company. Just look at the screenshots for some of these systems. Do a Google search for "SAP Screenshot" (or substitute any other ERP vendor, you'll get pretty much the same thing.) Do any of these look like systems you'd want to use?

So why are these ERP vendors still in business? Because even a slow, horrible, unusable ERP system is an incredibly huge win for a company. There are two reasons for this. First, the amount of power these systems put at the fingertips of decision makers is incredible. Essentially, the data provided by an ERP gives managers ammunition to overcome the status quo in order to make necessary improvements. Never underestimate the power of something that helps a business overcome internal equilibrium. Second, ERP systems nearly completely eliminate the need for paper pushers. Paper pushers, in a pre-ERP company, serve a vital role as a check on abuse and (ironically) waste. Managers simply cannot keep track of all of the activities of the company, and the paper pushers are set up as gates so that they can prevent things from going haywire. Once managers have the ability to keep an eye on things themselves and with software acting as this gate, paper pushers become largely unnecessary. The result? A much more efficient and strategically powerful organization.

Let us recap all of this so far:

There's a product which tens of thousands of businesses need, or else they will die.
Businesses are willing to pay insane sums of money for this product.
Existing suppliers are failing. Hard.

You know what that smells like to me? Opportunity.

Next, I'll cover the reasons behind the endemic failure of ERP vendors, and how that can be fixed and profited from.

Startups and "Ownership of the Idea"

erik@subwindow.com (Erik Peterson) — Wed, 26 Mar 2008 02:16:07 GMT

I want to talk for a minute on why hackers should only work on their own ideas.

Simply put, it is a question of motivation. In almost any startup, 90% of the early effort is product development. Therefore, it is unquestionably the best when the people developing the product are well motivated. In a software startup, the hackers are the ones developing the product, and it is nearly impossible for a hacker to feel motivated to work hard for an idea that is not theirs.

It is a question of ownership of the idea. It isn't a question of equity, or money, or quality of the idea or business plan. If I'm working on an idea that is somebody else's, the natural dynamic is for me to feel like the employee, and for the idea creator to feel like the employer. I don't want to be an employee- I've got my day job for that. Anybody busting their ass until 2 AM with little or no pay isn't going to bust their ass very long feeling like an employee.

With Apple, the idea for the company was Wozniak's. He was a hacker, and he had made this cool computer that he was proud of. As he was showing it off to some people, Jobs saw it and together they started to sell it. Imagine how far Apple would have gotten if Jobs had gone up to Woz and said "Hey, man, I've got this killer idea for a computer- would make it for me? I'll give you some equity!" It wouldn't have happened. The reason Woz was so dedicated is because it was his computer.

The exact same thing happened with Microsoft. Gates and Allen created a product, and only five years later did they get Ballmer to help them sell it. Software startups these days actually aren't all that different than Apple or Microsoft were in the 70's. You need someone- a Wozniak- to build the product. And only after that, do you need someone- a Jobs- to do all the rest. Hackers, no matter how skilled in business (myself included), will eventually need a "beef-headed M.B.A." to help them.

There's a big difference, here, between a startup in development mode and a startup in sales mode. Startups in development mode don't need a business guy. In fact, it is probably harmful for a development-mode startup to have someone involved who is not working on the product. However, once the startup starts making money, a business type is necessary. Someone is needed to pursue funding, to balance the books, and to manage the marketing effort. Hackers may, technically, be capable of these tasks- but shouldn't do them, because their heart is in the product. A startup in sales mode needs somebody whose heart is not in the product- someone to make clear, unbiased and responsible decisions that benefit the whole company, and not just the product.

At the absolute beginning stages of a software startup, two things are needed: a hacker (or two, or three), and his (or their) idea. Anything else is just an impediment.

On Teams

erik@subwindow.com (Erik Peterson) — Fri, 14 Mar 2008 20:20:50 GMT

So, here’s a question:

What’s the ideal size of a software team?

Most people, I don’t care how long they’ve studied the “art” of software development, will come up with a vague answer. “Six to 10” some might say. “About Eight.” “No more than 12.” "Five or Six, I guess.” These are pretty common answers. If pressed to come up with a rationale behind them, you might hear some rumbling about Mythical Man Months, but other than that, nobody really knows why they think that “Roughly Six” is a good number for team size.

Let me take a different approach. Forget, for a minute, about software development. Forget that this is a programming blog.

What’s the ideal size of a team of humans organized to complete a common objective?

Looking at the whole of human existence, wherever humans have to organize to achieve objectives, they always organize in groups of four or five. Four is a bit more common than five. Three is less common. Six is much less common.

Where does this happen?

The military: the smallest unit in the military is the fire team. four to five people.

Baseball: Yes, baseball teams have 25 players on a roster and nine players on the field at a time. However, a team can be divided into smaller units based on their objectives on the field. The top of a baseball lineup is four players, from leadoff to cleanup. The infield is four players whose goal is to prevent balls from leaving the infield. The outfield is three players who all shag fly balls.

Basketball: Each team has five players on the court at a time.

Soccer: Four defenders. four midfielders. four forwards. The set of four is so important for Soccer, that even though they are limited to 11 players on the field (10 position, one goalie), players will move from team to team so that whichever task is most important at the time (defense, offense), that task will always have at least four (and sometimes five) players working towards it.

Music: The vast majority of bands have four or five members. three and six are uncommon. Rarity roughly increases with size. Not just with modern music- the string quartet has long been one of the most popular musical group types. Even in large orchestras, they’re broken down into smaller pieces. Two groups of five violins in First and Second violins. Two groups of four cellos. Three Clarinets. Three Bassoons. Four or five trumpets.

Families: Most families these days have four or five members. Families in history had more members, but that is because they were organized around disparate objectives. Families aimed towards four men (3 boys and one father) to tend the field, and four women (3 girls and one mother) to tend the house. Since humans used to die easily and sex is a coin toss, this ideal team was rarely achieved. However, it was clearly the goal.

Why Does this Happen?

Ok, enough with the non-software comparisons. It should be pretty obvious that the ideal size of a group of humans organized around a single objective is four or five (with three occasionally being preferred). So why do software teams so often exceed this number?

I think the reason is that it is hard to define roles for software developers. In a band, you have very clearly defined roles- one drummer, one or two guitarists, one or two vocalists, and one bassist. Very rarely is this arrangement deviated from. Similar in sports- each player has a very defined role- 3rd baseman, point guard, center forward, sweeper. In families the role is defined by age.

What about roles in a software development team? Sometimes you’ll have a senior developer, but the role division usually doesn’t go any farther than that. I’m quite convinced at this point that each member in a software team needs a clearly defined role, and the size of the team will naturally work itself out.

The roles probably depend on the type of project. For a database-backed web application a good setup might be: Lead, Quality, UI, Data. With each member in that team you have a really good definition of what the role of the person is. If a certain need isn’t being met properly by the defined roles, then you add another person. The existing members naturally resist adding new members when there isn’t a role for them. Team size reaches automatic stasis.

Now, this is obviously pure conjecture at this point. I look forward to putting it in practice and watching how this approach actually shakes out. In the mean time, has anyone had experience with software development with tightly defined roles? How did this effect team size?

What's so DRY about Rails?

erik@subwindow.com (Erik Peterson) — Thu, 06 Mar 2008 02:48:40 GMT

One of the main talking points for Rails evangelism states that it is really DRY. You don't have to repeat yourself. This might be true in comparison to other languages, but when you really get down to it, it isn't very DRY at all. Take a look at your own workflow for common tasks with Rails development, and while line-by-line the code may be DRY, from a conceptual level it isn't at all.

Common task: Adding a column

I need to add in a column for Parts that will hold the Reference number. It is a required column. Here's what my workflow for this kind of task:

script/generate migration parts_add_reference
Open up the migration, add in:
add_column :parts, :reference, :string, :limit => 40, :null => false
Figure out some way (possibly in the migration) of dealing with old records that don't have this new required record. Figure out a way to handle this gracefully both with development/testing fixtures and production.
Add in a test to verify that the validation is there
assert_validation(Part, :presence, :reference, {:good => ["XX-1234"]})
(This uses a test helper package that I may or may not release in the future. Implementation is not important. The important part is that it asserts that the validates_presence_of validation is there.)
Run the test. Test fails.
Add in:
validates_presence_of :reference
Run the test. Test passes.
Add in the functional test to make the field is displayed in the new/edit forms:
assert_tag(:input, :attributes => {:type => :text, :id => "part_reference", :name => "part[reference]"})
Run the test. Test fails.

Add in the code for the form:

<p>
  <b>Reference</b><br />
  <%= f.text_field :reference -%>
</p>

Run the test. Test passes.
Add in the functional test to make sure the field is displayed in the show page:
assert_tag(:div, :attributes => {:class => 'property', :id => 'part_reference'})
Run the test. Test fails.
Add in the code for the show page:
<div id="property" id="part_reference"><b>Reference:</b> <%= @part.reference %></div>
Run the test. Test passes.
Run through all of this in a browser, testing it yourself.
Run the entire test suite.
Commit to source control:
git add db/migrate/XXX_parts_add_reference.rb && git-commit -a
Deploy:
cap deploy:migrations
Check the live version, praying everything went OK.

Jesus fucking Christ. I have to specify the column in 4 different places: the migration, the model, the tests, and the view. Absolutely unacceptable. If I have to change the name of the column from "reference" to "part_number", it would be like performing an appendectomy. Putting it in in the first place requires 21 discrete steps. Just to add in one stinking column? Why in the hell can't it be four?

Write the code:
shows :reference, :required => true, :size => 40
Test.
Commit.
Deploy.

Now I know what you're saying: "That's magical web-app candyland! There's no way we'll ever get to that level! Even if we could, we wouldn't want to!" Bullshit. I think it would be fucking Stellar if someone came out with a web app framework that distilled development down to this tight cycle. Write one line of code. Test. Commit. Deploy. Four steps. That's all you need.

Stay tuned...

Back By Popular Demand: Salesforce scraper

erik@subwindow.com (Erik Peterson) — Fri, 01 Feb 2008 23:54:59 GMT

In a previous incarnation of this here blog, I had a post about how to construct an app in Rails that scrapes Salesforce.com data to a local database. This was about a year and a half ago, and the blog that it was posted on has been defunct for at least a year now. To this day I still get requests for that post, so I figure I should repost it. A lot of things have changed in that year and a half, however. My project went from "integrate with Salesforce so we can get better reports" to "Replace Salesforce.com and then some." Therefore, I haven't used the code I had written about in a long time. I have no idea if it even still works. So while my use-case for this code doesn't exist any more, it is apparently useful to other people.

First off, this uses the activesalesforce gem. I think the last version that I tested it on was 1.0.0, so any updates might break this code. Also, there's apparently a activerecord-activesalesforce-adapter gem now, which works with Rails 2.0. Therefore, things are apparently much different these days. However, this code will probably give you some insight.

Database.yml

sf_production:
  adapter: activesalesforce
  url: https://www.salesforce.com/services/Soap/u/8.0
  username: #your salesforce username
  password: #your salesforce password

production:
  adapter: postgresql #You can probably use MySQL or whatever.  I doubt it matters
  database: production
  username: #whatever
  password: #whatever
  host: #whatever

account.rb

class Account < ActiveRecord::Base
  establish_connection "sf_#{RAILS_ENV}"
end

class Account_Postgres < ActiveRecord::Base
  set_primary_key "id"
  establish_connection "#{RAILS_ENV}"
  set_table_name 'accounts'
end

Opportunity.rb

class Opportunity < ActiveRecord::Base
  establish_connection "sf_#{RAILS_ENV}"
end

class Opportunity_Postgres < ActiveRecord::Base
  establish_connection "#{RAILS_ENV}"
  set_primary_key "id"
  set_table_name 'opportunities'
end

I think you get the point. You'll need to do the same for any other Salesforce objects you plan on using. It really sucks having to have the Salesforce version have the normal AR name, and the Postgres version having to have the altered name, but the activesalesforce gem depends on this naming scheme. Next, Here's a big fat dump of some of the helper methods that I use. You might want to put this in a file in lib.

lib/salesforce_helpers.rb

#Creates a "migration" for a specified Salesforce class.  Eval this in a ActiveRecord::Schema.define() block
def dumpclass( aclass )
  dumpstr = "  create_table \"" + aclass.table_name + "\", :id => false, :force => true do |t|\n"
  for column in aclass.columns
    dumpstr += "    t.column \"" + column.name + "\", "
    if column.type.to_s.eql?('text') && column.limit.to_i < 1000
      dumpstr += ":string, :limit => " + column.limit
    else
      if column.type.nil?
        dumpstr += ":string, :limit => 255"
      else
        dumpstr += ":" + column.type.to_s
      end
    end
    dumpstr += "\n"
  end
  return dumpstr + "  end\n"
end

#Scrapes a single record from Salesforce to the local database
def scrape( aobj, aclass, adbclass )
  begin
    new_obj = adbclass.new(convert(aobj, aclass))
    new_obj.id = aobj.id
    new_obj.save!
  rescue
    return 1
  end
  return 0
end

#Scrapes a whole Salesforce Object harshly- deleting all the local data, and the dumping.  Good for empty tables
def hard_update_class(aclass, adbclass)
  count = 0
  adbclass.delete_all
  for aobj in aclass.find(:all, :limit => 0)
    scrape(aobj, aclass, adbclass)
    count += 1
  end
  return count
end  

#Scrapes a whole Salesforce Object softly.  Only looks for objects that were created/updated since the last scrape.
def update_class(aclass, adbclass)
  #I honestly don't know why I did it this way.  It isn't very DRY.  There must be a reason, so tinker with caution.
  begin
    lastcreated = adbclass.find(:first, :order => 'created_date desc')
    lastmodified = adbclass.find(:first, :order => 'last_modified_date desc')
    for aobj in aclass.find(:all, :limit => 0, :conditions => 'createddate > ' + (lastcreated.created_date - 18000).to_s(:iso_8601_special))
      scrape(aobj, aclass, adbclass)
    end
    for aobj in aclass.find(:all, :limit => 0, :conditions => 'lastmodifieddate > ' + (lastmodified.last_modified_date - 18000).to_s(:iso_8601_special))
      adbclass.delete(aobj.id)
      scrape(aobj, aclass, adbclass)
    end
  rescue
    begin
      lastcreated = adbclass.find(:first, :order => 'created_date desc')
      lastmodified = adbclass.find(:first, :order => 'last_modified_date desc')
      for aobj in aclass.find(:all, :limit => 0, :conditions => 'created_date > ' + (lastcreated.created_date - 18000).to_s(:iso_8601_special))
        scrape(aobj, aclass, adbclass)
      end
      for aobj in aclass.find(:all, :limit => 0, :conditions => 'last_modified_date > ' + (lastmodified.last_modified_date - 18000).to_s(:iso_8601_special))
        adbclass.delete(aobj.id)
        scrape(aobj, aclass, adbclass)
      end
    rescue
      puts "Skipping " + aclass.to_s
    end
  end
end

#Converter for a single object
def convert ( aobj, aclass )
  hash = {}
  aobj.attributes.each { | key, value | 
    hash[key] = value if aclass.column_names.include?(key)
  }
  hash
end

#For some reason the Salesforce didn't interpret the ISO 8601 date format spec correctly
ActiveSupport::CoreExtensions::Time::Conversions::DATE_FORMATS.merge!(
  :iso_8601_special => "%Y-%m-%dT%H:%M:%S-05:00"
)

Ok, whew. Sorry that was so long. Now we can go about using these methods. Most of this stuff is in rake tasks.

#Create local schema for Account
ActiveRecord::Schema.define() do
  eval(dumpclass(Account))
end

#Prefill the data for Account
hard_update_class(Account, Account_Postgres)

#Update the data for Account
update_class(Account, Account_Postgres)

Conclusion

I hope that was useful to someone. I'm sorry that the code is in such a decrepit state. If you have any questions, just comment here and I'll try to help you out. I highly encourage anyone to modify/update/cleanup this code and make a Rails plugin or Gem for it.

Nested methods make me all DRY

erik@subwindow.com (Erik Peterson) — Wed, 30 Jan 2008 01:00:22 GMT

Here's a cool little thing I ran into when investigating scoped_struct You can nest methods on ruby:

def a
  def b
    return "b"
  end
end

That way, when you do a.b you get "b" as the result. Pretty neat, huh? This would be much more useful if you used it in classes to shorten up method names. Going with the example in Mike's blog, let's have a Player class that represents a football player. We want methods to give us some stats back:

class Player
  def fumbles
    def dropped
      1
    end
    def lost
      2
    end
    def recovered
      3
    end
    self
  end
end

This way, you get my_player.fumbles.dropped and my_player.fumbles.received all nested and defined all DRY like. One little catch is that you have to put that self return right after the method definitions in the fumbles scope. Otherwise, you'll get all kinds of nil errors.

Git over HTTP walkthrough

erik@subwindow.com (Erik Peterson) — Fri, 25 Jan 2008 07:47:58 GMT

This is a separate post to publish how I solved the problem highlighted in my last post. Specifically, I wanted a read-only Git repository available over HTTP through Apache. I already had Apache set up and had a subdomain set up via Pound. All I needed to do was set up the Git repository so that it could be accessed over HTTP. Easy, right? Apparently not.

If someone has an easier solution, or I'm just approaching this problem incorrectly, feel free to let me know. I'm perfectly open to the idea that I'm a retard. After all, I am a terrible programmer.

Ok, so the deal is that, apparently, for this kind of repository, you can't just use git-init like you could with a normal local or remote repository. From what I've gleaned, you have to use git-init-db Which is silly, because the man entries for the two appear to be almost identical. Again, maybe I'm just a tard, but I really don't understand the difference.

Even after that, git-clone wouldn't work, and I had to run git-update-server-info Ho, ho, ho! But that wouldn't even work! I had to do git-update-server-info --force for some ungodly reason. %i Keep in mind, this is still on a completely empty repository.

Ok, so without further ado, here are the steps to get it actually working:

mkdir myrepo.git
GIT_DIR=myrepo.git git-init-db
cd myrepo.git/
chmod +x *
git-update-server-info --force

Not exactly straightforward.

On Git

erik@subwindow.com (Erik Peterson) — Fri, 25 Jan 2008 07:31:17 GMT

Over the past few weeks, I've been evaluating Git. I'm sure you have heard of it, but in case you haven't, I'll give it a quick run down. Git is the leading distributed version control system out there, mainly because it is what is used for the Linux kernel and Linus has been doing some pretty heavy promotion of it. I figured I would try it, seeing as I'm a sucker for things that are hyped (and Git has definitely gotten its fair share of hype lately from all over the blogosphere).

Installation

Git was pretty easy to install. I downloaded and compiled the source, and it went really well. There is also a unofficial binary available, if you're afraid of make. There aren't (to my knowledge) any nifty GUI tools for Git, or a Textmate bundle. But that hasn't bothered me much- I never used the Subversion equivalents. It always seemed faster to do it via the command line (while occasionally piping svn diff to Textmate).

Learning Curve

Overall, Git was really easy to understand. Setting up a repository and committing things locally is pretty much identical to Subversion. Setting up a remote repository and pushing to it was also pretty easy, and about on par with an svn+ssh setup. Getting Git to work with Capistrano was also surprisingly easy. It took about 20 minutes total to read up, install and start working with my basic Git setup.

Some bumps

I wanted to set up a public-facing code repository that would make it more likely that I would share some of my code. I figured that this would be a strong suit of Git since it is all about sharing and its decentralized nature means that pretty much every contributor is going to have their own publicly-available repository. However, I was sorely mistaken. The online FAQs for setting up this kind of repository over http were sparse, incorrect, or both. After 3 different tries and about two hours of work, I finally got one working. I'm still not sure if it is set up the right way, but at least git clone works, and that's what really matters.

Overall, this portion was a pain in the ass and it would be nice if using git init would work for a git repo over http. The confusing combination of git init and git init-db while apparently requiring git update-server-info --force for no reason makes this a big failure for Git. Maybe there's another explanation- maybe I'm retarded. I don't know. But setting up something that should have been easy was instead horribly difficult and poorly documented.

The Strong Suit: Branching

I will have to say, however, that branching a repository in Git is incredibly easy. It really is impressive how easy it is to create a branch and to switch between the two. This is useful even for only one developer, because I can maintain a "public" branch of some code. Now I can, for instance, keep my database.yml intact without publishing my database passwords to everyone. I don't even have to use nasty workarounds like database.yml.mine- I have my own branch that my own database.yml is checked in to. What a breath of fresh air.

Conclusions

Overall, Git is nothing revolutionary. There are some really nice features, and I'm sure some other big gains for large open source projects. But if you're working with your own code base or a small team, this won't change the way you work in any significant way. What's more is that I don't see any reason why the biggest advantages of Git (branching, local repositories) can't be included in Subversion or any other centralized version control system. The biggest obstacle is politics.

Still, I highly suggest that you switch to Git for all of your development if you can. The ease of branching and the local repositories alone are enough to warrant the 10 minutes of work it will take to switch from Subversion. Really- 10 minutes. That's all it will take.

That said, will I be switching to Git for all of my development? No. While all of my personal development has already switched to Git, for my day job I'm kind of tied to Trac. The ability to associate tickets with discrete versions in Trac is invaluable and actually required by law in our particular case. So until Trac or another equally good bug-tracking system supports Git, I'm sticking with Subversion for my day job.

ZSFF

erik@subwindow.com (Erik Peterson) — Tue, 15 Jan 2008 10:35:12 GMT

Zed Shaw has introduced a new syndication format called ZSFF. I think it is a pretty cool idea, because RSS is generally pretty shitty. As an author, it is hard not having control over your content- it being at the mercy of whatever RSS reader is going to fuck it up. Plus, I have a natural aversion to anything XML.

So, I've bought in. That's one of the key advantages to rolling your own blogging system- you can whip up something like that in 5 minutes and not worry about it. It was literally trivial to create the ZSFF feed. No libraries, no complex logic. Here's my entire index.zsff.erb file:

ZSFF 1.0
  Author: Erik Peterson
  Title: subWindow Blog
  Site: http://www.subwindow.com/
  Subtitle: Just another crappy programmer's blog
  Copyright: 2008
  Content-type: text/plain
  
  <% for article in @articles -%>
  <%= article_url(article) %>
  <% end -%>

I've also built a validator. Here is the code as well, if you like.

Update: I've refactored the code quite a bit and packaged it as a gem. It is now available through the command:

gem install zsff

Don't fuck with my expectations

erik@subwindow.com (Erik Peterson) — Tue, 15 Jan 2008 07:09:19 GMT

My hard drive died as part of some kind of karmic clusterfuck, and right before it hit the Cobain barrier I was trying out git. Now that I'm back on my feet I wanted to download and recompile the source of git again so that I could continue my geekery. There's this thing I like to do when downloading/compiling something: I get the URL for the tarball and:

wget {whatever} && tar -xzf {whatever} && cd {whatever} && ./configure && make && sudo make install

That way, the thing can download and compile while I don't have to tend to the whole process one step at a time. I can go back to coding or whatever I was doing.

Much to my chagrin, OS X doesn't come with wget. What the fuck? wget has to be one of the most fundamental programs in existence- it is completely nonsensical to keep it out of a distribution that is Unix-based. Sure, it has curl, but curl has more flags than Jimmy Carter has liver spots. I'll be god damned if I'm going to remember all of those when I just want to download a fucking file.

Anyways, the point of this is that I went to get wget by going to the site, copying the url of the source, going to the terminal and typing in

wget http://ftp.gnu.org/gnu/wget/wget-1.10.1.tar.gz && tar -xzf wget-1.10.1.tar.gz && cd wget-1.10.1 && ./configure && make && sudo make install

Needless to say, I about had a fucking conniption fit.

The point here being: don't fuck with your user's expectations. Especially the ones that are hard-wired into their brains so extensively that when they try to get around the expectation failure, they fail again.

Way to make your users feel like tards, Apple.

I'm a Terrible Programmer, And You Should Be Too

erik@subwindow.com (Erik Peterson) — Sun, 13 Jan 2008 10:44:34 GMT

This being the first post on this illustrious blog, I figure I'd explain that subheading you see right up there. Let me lay it on you, short and sweet:

I am a terrible programmer.

Seriously. I'm not very good. About half the lines of code that I write are broken. The way I do things the first time is almost always the wrong way. It is rare that I look at a piece of code that I've written more than a month ago without saying "Damn. That is some gross-looking code." And you know what? Somehow, I manage to ship rock-solid, scalable and nearly bug-free systems. And you can too! All you have to do is embrace the terrible programmer that resides within each and every one of you.

The key to milking quality code out of a terrible programmer is that you first, before all else, have to admit you are a terrible programmer. Only through that admission and the terrible fear of being discovered can you, a terrible programmer, create some fantastic systems. The idea is that by simply knowing this, you've already taken the first step on the long path to not being a terrible programmer. It's kind of like the twelve steps of crappy code addiction, where you first have to admit that you have a problem.

This philosophy is kind of an offshoot of what I like to call the Dunning Effect. This is the idea that dumb people tend to overestimate their intelligence or ability, and smart people tend to underestimate it. Therefore, if I am smart enough to realize that I am dumb, I am actually smart. Applied to programming, If I am a good enough programmer to realize that I am a terrible programmer, I am actually a good programmer. But this isn't all hocus-pocus "believing will make it happen" bullshit. This isn't The Secret®. This is serious stuff we're talking about here. Without further ado, let me introduce:

Erik's Terrible Guide to Not Being a Terrible Programmer By Realizing You're Terrible

Part 1: Paranoia
The first step toward exploiting your mediocrity in order to achieve greatness is to be really, incredibly paranoid about it. What if my wife finds out that I'm a terrible programmer? She'll divorce me. I can't let that happen! I must test my code! I must refactor obsessively! I must fix bugs before they're found!

The most important part of this is to test your code, and I don't mean "run through it once before you commit." I mean create some killer unit tests (and functional and integration tests, if your framework is in to that sort of thing). I'm so afraid that my coworkers will find out that I'm an awful programmer that I always run greater than a 1:1 code to test ratio. For stuff that would surely out me if it were to break (accounting software, life-saving medical devices, etc), I usually run at about 1:2. Believe me, these tests get run quite often. Beyond that, I always, always use multistage deployment. This means that I stand up a staging environment that is mirror image of the production setting, and I go to town. I do shit that would shame a two-year-old. I try to break stuff so hard you'd think I was Rick Allen doing a drive-by high five. If anything breaks, I fix it and write a test that duplicates the bug to make sure it never comes back. Why? Because I'm so terrible, I'd probably let it happen again!

It goes beyond just testing though, buddyboy. I'm such an awful programmer, I don't even have confidence that my exhaustive automated testing practices will prevent a bug from ever reaching a user. I make sure that I know about any errors the same instant that my users do. That way, by the time Frank shoots me an email saying "Man, Erik, this shit is broken. You're such a terrible programmer!" The bug will already have been fixed and deployed, and I can tell Frank that he must be seeing things, because it works fine now. He'll never suspect that underneath these clothes, I'm actually wearing Terrible Programmer underwear. It'll just be my dirty little secret.

Part 2: Grand Theft Code

The second part of Being a Great Programmer by Being a Terrible Programmer™ is realizing that pretty much anyone else out there who has ever written any code is a better programmer than you. That means you must copy code like mad. Lucky for you, people freaking love sharing code. They call this shared code "Libraries", "Frameworks", or maybe even "Plugins". If you're doing anything, you have to first realize that somebody's probably already done it, and that it's probably been done better. You just have to find out if they've published it, find it and then mercilessly rape and pillage that code until they're crying themselves to sleep at night.

By using other people's code wherever possible, you can rest assured knowing that you've completely minimized your chances of fucking up. And rest well you will, my friend. Like your dog who sleeps so soundly because he spent 36 minutes chasing his own tail, you will sleep like a baby due to your overwhelming incompetence.

Well, that's it for today. I hope you've learned that you are an awful programmer, and it is the most important thing you know.

subWindow Blog

Ruby Module Reference

Places where Rails is not DRY

Summer Rails Plugin Series #3: acts_as_referenced

Installation

Usage

Advanced Usage

A Note on Racing

Conclusion

Summer Rails Plugin Series #2: needs_approval

Instructions

Other useful functions:

Note

Conclusion

What You're Doing is Not New, Not Revolutionary

finger is to Friendster as Friendster is to Facebook

The Trouble With finger

Lessons are for Hares

Background Processing in Rails Using 'at'

Enter at

English as Code as Pseudocode

The Tradition of Pseudocode

Taking it a bit Further

Are We There Yet? Are We There Yet? No!

Summer Rails Plugin Series

What is a Negative Captcha?

How does a Negative Captcha Work?

Main Part #1: Honeypots

Main Part #2: Real fields

Ancillary Part #1: Timestamp

Ancillary Part #2: Spinner

Ancillary Part #3: Secret key

How does the Negative Captcha Plugin Work?

Possible Gotchas and other concerns

Credit

Those Pesky Expectations

Sometimes, the little things count

Thinking about yield(self)

The ERP Money-Back Guarantee

Enterprise Resource Planning: Mass epic... Opportunity.

Startups and "Ownership of the Idea"

On Teams

What's so DRY about Rails?

Common task: Adding a column

Back By Popular Demand: Salesforce scraper

Database.yml

account.rb

Opportunity.rb

lib/salesforce_helpers.rb

Conclusion

Nested methods make me all DRY

Git over HTTP walkthrough

On Git

Installation

Learning Curve

Some bumps

The Strong Suit: Branching

Conclusions

ZSFF

Don't fuck with my expectations

I'm a Terrible Programmer, And You Should Be Too

Erik's Terrible Guide to Not Being a Terrible Programmer By Realizing You're Terrible

Enter `at`