Collective Idea Blog

Batch Number Increment in Vim

2013-05-23T12:18:46-04:00

We’re fans of Vim at Collective Idea, with the largest percentage of our staff using it as their primary editor. We’ve even added a few recent ¹ converts. Go Vim!

Setting the Stage

This morning I needed to quickly increment multiple sets of numbers. Too tedious to edit each number by hand, I wanted to figure out how to best do this with Vim.

We’ll use a simplified example to demonstrate the task at hand — globally increment a series of numbers by a given value. The first block of HTML is the original code and the second block of HTML is the desired output:

<tr>
   <td><img src="1.jpg" data-alternate="alternates/1.jpg">
   <td><img src="2.jpg" data-alternate="alternates/2.jpg">
   <td><img src="3.jpg" data-alternate="alternates/3.jpg">
</tr>
<tr>
   <td><img src="1.jpg" data-alternate="alternates/1.jpg">
   <td><img src="2.jpg" data-alternate="alternates/2.jpg">
   <td><img src="3.jpg" data-alternate="alternates/3.jpg">
</tr>

<tr>
   <td><img src="1.jpg" data-alternate="alternates/1.jpg">
   <td><img src="2.jpg" data-alternate="alternates/2.jpg">
   <td><img src="3.jpg" data-alternate="alternates/3.jpg">
</tr>
<tr>
   <td><img src="4.jpg" data-alternate="alternates/4.jpg">
   <td><img src="5.jpg" data-alternate="alternates/5.jpg">
   <td><img src="6.jpg" data-alternate="alternates/6.jpg">
</tr>

Here are the steps I took and what I found.

A First Pass

I initially reached for Vim’s CTRL-A, a very handy tool for incrementing² numbers. Anywhere on the line, 3<c-a> would increment by 3 the first occurrence of a number relative to the position of the cursor.

Awesome! Except that that only increments the first number and Vim’s single-repeat repeater, instead of moving to the next number, re-increments that number again by 3.

Perhaps one of Vim’s other repeaters, a macro, or single-repeat with an added motion would have worked, but as with most tasks in Vim with excessive repeated keystrokes there’s usually a better way.

Enter Sub-Replace-Expression

Vim’s sub-replace-expression allows the second half of a substitution to contain logic to be evaluated. Prepending a \= signals to Vim that what’s to follow should first be evaluated before being used as the replacement value in the substitution.

In our example, visual line select the second set of <td>'s and execute the command :'<,'>s/\d\+/\=submatch(0)+3/g. Presto!

# match against 1 or more digits
\d\+ 

# begin replace expression
\=

# submatch({nr})
#
# return the nth submatch of the matched text, 0 returns
# the whole matched text; add 3
submatch(0)+3

# perform the substitution globally
/g

¹ A convert in progress. I believe in you, Jonathan.

² CTRL-X handles decrementing.

Turning Off

2013-05-06T11:16:33-04:00

Now is the time to disconnect. Not forever, not always, but we need to disconnect so we can reconnect with the world.

There’s lots of new writing on the idea of moving away from our always connected state, from Paul Miller writing for The Verge about taking a year off of the Internet, to Douglas Rushkoff’s Present Shock which argues that we’re trying so hard to live in the moment that we’re actually living in the past and missing the present.

I’m also becoming a fan of the Slow Web movement. Not everything we do needs to be instant; it instead needs to be timely. I fell into this trap this week, as we launched our new app and I was immediately refreshing Facebook, Twitter, and various analytics services trying to see how fast it would go viral. I couldn’t turn off, because I wanted to see things happen in real time, even though I’m actually seeing what just happened in the “near past.” Fortuntely, I was touristing in Belgium, so once I left the hotel I was free from my addiction, but I’m sure I would have had mostly unproductive days had I been in the office looking at all the stats.

One of the most frustrating statistics when launching an iOS app can actually be somewhat freeing: Download numbers. Apple only updates your download & sales numbers once per day, so you really only get to see what happened yesterday. At first I was frustrated that I couldn’t keep checking and see live progress. After a couple days, I realized the delay meant I could stop obsessing. Minute-by-minute changes wouldn’t affect our plans, and would only serve to distract me further.

We (especially as developers) live in a world of push notifications, live-updating pages, and instant results. This speed gives us immense power but, most of the time, the distractions aren’t worth the opportunity cost of slowing down. With an iOS app it takes us about a week to push an update out to our customers so, unlike a website, we have little need to be so reactionary. Maybe my vacation was well-timed after all.

I think we’re coming to a new phase in our tech lives where we discard our always-on lives for a more deliberate usage model. We’ll applaud people who have super-productive spurts of connectivity more than the “inbox zero” champion who can respond to any message in seconds. I don’t see us ever removing the phones from our pockets, but I see us leaving them in the pockets more often.

We built Downside to combat the problem of people always being on their phones when in public, but our team has found it fairly easy to play. Something about having your phone in front of you helps remind you that you don’t need it. You can check that fact on Wikipedia later. You can answer your email after you get back from lunch.

I wonder if we’ll see a split between the super-connected and the “disconnecters” coming soon. The super-connected are excited about Google Glass, where the disconnecters are skeptical. The disconnecters are the ones who have found productivity gains by closing their email & Twitter clients during the day, and have turned off most push notifications. The super-connected are feeding their appetite and can’t go 5 minutes without checking at least one device. Here’s a tip for disconnecters: play Downside with the super-connected and wager on the outcome. You’ll get a lot of free lunches.

Meet Downside, our First iPhone Game!

2013-04-30T18:02:36-04:00

I’m extremely excited to introduce Downside, our first foray into building a game for iOS.

Downside started as most great ideas do: as a crazy idea at happy hour. We had played a version of the game for almost a year, but Tim and Brian realized that it would make a great App too and they’re right.

The premise of Downside is simple: Most of us are used to staring at our phones instead of talking to the people in the room. Downside makes a game out of keeping your hands off your phone. We’ve been beta testing at lunch and happy hours for a few weeks and we usually go through an entire meal without anyone touching their phone. If we can do it, you can too.

Downside uses Game Kit to find people in the same physical space as you, and you can quickly create a game with anyone around you without having to be friends on any kind of social network. Once you’re in a game, everyone flips their phones face down and the game begins. As soon as you pick it up, you lose.

We have lots of plans for additional game modes too, but we want you to see our first version and tell us what you think! Download it today free on the App Store and drop us a comment or a review. We’d love to know how you like it.

Some of our early users will wager a round of drinks, whereas families are getting their kids to pay attention around the dinner table. There’s even been talk about using it in business meetings to focus on the task at hand. We know people are doing this already without Downside, but now there’s an app for that!

Conference Season: 2013

2013-02-27T11:12:24-05:00

We’re gearing up for Conference Season. Want to see us in the flesh? Avoid us? Here’s the start of our schedule so you can plan accordingly. We hope to see you at these and many more!

GR Dev Day

This coming Saturday, both Tim Bugai and Jason Carpenter will be presenting. Tickets are sold out, but maybe they’ll start showing up on StubHub.

SXSWi

Chris Gaffney, Jason Carpenter and I will be at the Interactive Festival this year, and though there will be far too many people, we’d love you to say hi if you’re going!

MountainWest RubyConf

Jason Roelofs is speaking in the DevOps track at MountainWest, April 3–5.

Great Lakes Software Excellence Conference

Tim Bugai will be speaking at GLSEC, and you’ll likely see a few more of us there on April 29.

RailsConf

We’re sending a big crew to RailsConf this year. Brian Ryckbost, Steve Richert, Chris Gaffney, David Genord, Ryan Glover, Eric Milford, and Jonathan Pichot will all be there, and looking to hang out April 29–May 2.

Scottish Ruby Conference

Finally, I’ll be at the Scottish Ruby Conference May 12–13. I’ll be the American looking for as much Haggis as I can get. See you there!

Becoming a Rubyist

2013-02-25T10:00:23-05:00

I joined Collective Idea six months ago. In that time, I’ve had the chance to throw myself into a new language. Just like learning a new spoken language, learning a new programming language means learning a new way of thinking. Here are some of my reflections on becoming a Rubyist.

Coding is Human

My initial attraction to Ruby was the community’s emphasis on clear, human-readable code. You can only write so many nested foreach loops in PHP before you begin to look for a more succinct and rational language. When I began exploring Ruby, I was struck by Matz’s human-centered philsophy of increasing progammer happiness, and of the principle of least surprise. I was also itching to program in an Object-Oriented language. (Though PHP does have objects, the framework I worked in previously is almost exclusively procedural.) Even now, when I write an each loop in Ruby, I can’t help enjoying how nice it looks.

Reading Russ Olsen’s excellect Eloquent Ruby, I realized that programming could be an aesthetic exprience as well as a logical one. Steve Klabnik has spoken on these themes, putting programming in relation to literature and philsophy. It’s very typical when talking to other Rubyists to hear code described as ‘beautiful’ or ‘clever’. For me, coming to understand programming in these terms connects it with other human pursuits. After all, a community that produced _why must understand art as well as engineering.

Patterns are (educated) Opinions

I learned quickly after joining Collective Idea that there is rarely one right answer to a programming problem. It’s been revealing to hear the differing opinions in the office over everything from Ruby conventions to Rails patterns. Simply listening to these conversations has made me a better programmer. It’s also made me want to take the time to develop my own well-supported opinions.

The freedom that comes from not having one answer (and a language that lets you open any class!) is incredibly empowering, but it can also be paralyzing. When I encounter decisions that have architectural implications, I’m still uncertain on how best to move forward. I’m afraid of making a decision that I regret later. I don’t yet have the benefit of experience to help me weigh the pros and cons of any one approach.

I encountered my first significant technical debt a few months after starting. I was working on an app that functions primarily as a glue between several different services. On certain actions, it makes calls to at least 3 separate services, having to coordinate and process the responses from each. My controller had gotten out of hand, and I had mixed up some business logic in with my models. My tests were straining, edge cases were breaking the app, and I knew I was Doing It Wrong™.

Discussing with colleagues how best to move forward, it was suggested I try a pattern one of them had had recently written about. Called the Interactor pattern, it seperated all my interactions with outside services into a seperate class. It was beautiful, it just felt better, and my tests agreed.

Testing is Craft

You can’t spend much time working in Ruby without hearing about the importance of testing. At first, I had a typical reaction, wondering how writing more code would make me more productive. The more I did, though, the more I realized how good tests lead you to write better code. My first tests, of course, weren’t very good. But they got better. And after watching Geoffrey Grosenbach excellent series on RSpec (he’s recently updated it for RSpec 2), I began to fully grasp isolation and why small methods are awesome.

I also began to understand what the industry buzzword around ‘craftsmanship’ really means. I’ve worked in software long enough to know firsthand how easy it is for projects to underperform and go over budget. Building software is unlike building almost anything else. It’s essentially collaborative logic-making. Because its product is incredibly malliable, it needs clear and specific guidelines. Becoming a ‘software craftsperson’ is having the humility to realize that one person cannot fully conceptualize the complexity of the entire software stack. We write tests because we know we’ll make mistakes, because we’ll have to maintain what we’ve built, and because we don’t know who might have to work with our code in the future. This may be old news to TDD diehards, but that doesn’t make the idea any less powerful.

In short, becoming a Rubyist has been one of the most rewarding processes in my career. It’s made me appreciate what I do on an aesthetic level, it’s brought me a creative freedom I didn’t expect from programming, and it’s given me the tools to be proud of what I build.

Chrome Profile Settings and Capybara 1.1 Stable

2013-01-25T11:51:50-05:00

We previously blogged about how to test file downloads with capybara and chromedriver that explained how to create a profile that specified where to send downloaded files.

We had take a similar approach with a recent project, except instead of declaring a different download directory we were setting the window placement to alter Chrome’s size. This seems to have broken with a recent version of Chrome.

Instead of relying on settings like this to adjust Chrome’s window size:

profile['browser.window_placement.top'] = 0
profile['browser.window_placement.left'] = 0
profile['browser.window_placement.right'] = 10000
profile['browser.window_placement.bottom'] = 10000

Set the --window-size=width,height command-line switch by including the args hash option:

Capybara::Selenium::Driver.new(app,
  :browser => :chrome,
  :profile => profile,
  :args => ["--window-size=10000,10000"]
)

Long Lost Allocation

2013-01-23T19:57:02-05:00

There’s a great little method in Ruby that you might not be familiar with. It’s used all the time but it’s hardly ever called directly. The method is Class#allocate.

Even if you’re not familiar with Class#allocate, I’m sure you’ve seen Class#new:

me = Person.new("Steve")

If you’ve ever been confused about how calling Person.new ends up calling Person#initialize, here’s what’s really happening behind the scenes:

class Class
  def new(*args, &block)
    instance = allocate
    instance.initialize(*args, &block)
  end
end

The allocate method just builds a new instance of the class, completely skipping the call to initialize.

So what?

While you probably shouldn’t call allocate directly in your production code, I think it can be incredibly useful in testing. You are unit testing… right?

Consider the following class:

class Animal
  def initialize(phylum, class_, order, family, genus, species)
    @phylum  = phylum
    @class   = class_
    @order   = order
    @family  = family
    @genus   = genus
    @species = species
  end

  def name
    "#{@genus} #{@species}"
  end
end

To test the Animal#name method, I need an instance of Animal. But for this method, I only care about the animal’s genus and species.

describe Animal do
  subject(:animal) { Animal.allocate }

  describe "#name" do
    it "combines the genus and species" do      
      animal.instance_variable_set(:@genus, "Homo")
      animal.instance_variable_set(:@species, "sapiens")

      expect(animal.name).to eql("Homo sapiens")
    end
  end
end

Rather than initializing a full animal instance to test only two of its internal values, I simply set those values directly.

For me, using allocate has been a great way to keep my unit tests short, clear and isolated. It’s especially useful when a class’s initialize method requires many arguments and/or preprocesses those arguments.

What else?

I find that developers often don’t test their own initialize methods. And if they do, it might look like this:

describe Animal do
  describe "#initialize" do
    it "sets the phylum, class, order, family, genus and species" do
      animal = Animal.new("Chordata", "Mammalia", "Primates", "Hominidae", "Homo", "sapiens")

      expect(animal.instance_varaible_get(:@phylum)).to eql("Chordata")
      # …
    end
  end
end

What stands out to me is that while we’re testing initialize, we never call it directly. Using allocate I can test initialize like I would any other private method:

describe Animal do
  subject(:animal) { Animal.allocate }

  describe "#initialize" do
    it "sets the phylum, class, order, family, genus and species" do
      animal.send(:initialize, "Chordata", "Mammalia", "Primates", "Hominidae", "Homo", "sapiens")

      expect(animal.instance_varaible_get(:@phylum)).to eql("Chordata")
      # …
    end
  end
end

Allocating in RSpec

Currently, calling subject within an RSpec unit test will give you the result of calling new on the class you’re describing. I’d love to see this default change to use the allocate method so that unit tests can be properly isolated.

In addition, calling allocate will always work, while calling new only works if your class’s initialize method accepts no arguments, which I don’t find to be the case very often.

While my hopes aren’t high that the default will change, at the least I’d like the ability to configure RSpec to use allocate rather than new when calling subject.

What do you think?

Reverse Search with Elasticsearch

2013-01-14T14:08:28-05:00

I am working on a Rails application that allows construction companies to manage a portfolio of projects (AscribeHQ.com). It also has the ability for other users to create a “Group Portfolio” to display projects that were uploaded by other users, based on a potentially complicated criteria. (ex. ‘City of Chicago Group Portfolio’ wants to show construction projects with budgets over $250k and are located within the city limits)

The problem

A Group Portfolio searches the existing projects to show the user which projects they can display, which we are using accomplishing with Elasticsearch. The real challenge is, we want the company user who uploads a new project to find all the Group Portfolios that matches their new project. Essentially we need a “Reverse Search”.

The solution

Elasticsearch has an amazing feature called Percolation, which allows us to save the complicated searches of the Group Portfolio into an index, then when a new project is added, we can ‘search the searches’ to return the Group Portfolio ID. Even better, there is a ruby gem called Tire that supports this percolation feature.

I’ll not go into all the set up of Elasticsearch itself as there are many great blog post on that part already, but this is how we set up the Percolation using the Tire gem:

UserProject model:

class UserProject < ActiveRecord::Base
  include Tire::Model::Search
  include Tire::Model::Callbacks

  # Magic method that returns all the search that match the new project
  def find_matching_groups
    Portfolio::Group.find(UserProject.index.percolate(self))
  end

  # The normal mapping that is required to set up ElasticSearch
  mapping do
    indexes :title,            :as => 'title', :boost => 2
    indexes :custom_title,     :as => 'custom_title', :boost => 2
    indexes :owner,            :as => 'owner.company_name', :boost => 2
    indexes :subtitle,         :as => 'subtitle'
    indexes :street,           :as => 'street'
    indexes :city,             :as => 'city'
    indexes :state,            :as => 'state'
    indexes :zip,              :as => 'zip'
  end

end

Group Portfolio model:

class Portfolio::Group < Portfolio
  include Tire::Model::Search
  include Tire::Model::Callbacks

  after_save do |group_portfolio|
    if group_portfolio.project_criteria.present?
      group_portfolio.save_query
    end
  end

  # Adds and updates query in ElasticSearch database 
  def save_query
    UserProject.index.register_percolator_query(self.id) do |q|
      params = {}
      
      # project_criteria is saved on the Group Portfolio object.  ex: [{"filter_type": "state", "states": ["MI"]}, {"filter_type": "proj_type", "types": ["28", "29"]}]
      self.project_criteria.from_json.each do |criteria|
        params = params.merge(criteria)
      end

      q.filtered do
        query do
          boolean do
            must { terms :phase_id, params['phases']} if params['phases']
            must { terms :project_type_id, params['types']} if params['types']
            must { terms :green_id, params['greens']} if params['greens']
            must { terms :delivery_method_id, params['delivery_methods']} if params['delivery_methods']
            must { terms :project_definition_id, params['project_definitions']} if params['project_definitions']
            must { terms :state, params['states'].map(&:downcase) } if params['states']
            must { terms :city, params['cities']} if params['cities']
            must { terms :zip, params['zips']} if params['zips']
          end
        end
      end
    end
  end

end

Action returning Group Portfolios:

class Manage::PublishingsController < ApplicationController

  def index
    @project = Project.find(params[:project_id])
    @user_project = UserProject.where(:portfolio_id => current_portfolio.id, :project_id => @project.id).first
    @group_portfolios = Portfolio::Group.where(:id => @user_project.find_matching_groups)
  end

end

Results

We were previously trying to accomplish this same type of results with Delayed Job and some very complicated code. It often took around 5 minutes to do this ‘reverse search’. Now the user sees the results in a half of a second and with simpler code. This is a big win for us and will help us offer better service to our customers.

Applications

There seems to be endless applications for this. On dating sites, a new user can be told how many (and even who) searched for them before they signed up. Auto dealerships could easily see if the car they are could buy matches any recent searches for vehicles on their site. An advertising site could estimate how many views an ad will get based on previous queries. All of these things could be done in other ways, but the code will likely be very complicated and slow.

references: Percolation

Running HAProxy behind Amazon's ELB

2013-01-11T15:34:09-05:00

Amazon’s Elastic Load Balancer (ELB) is a great tool for serving traffic across availability zones in an AWS region. It can do SSL termination, can handle any amount of traffic (in a ramp-up-over-time format), and your only variable cost is the bandwidth in question. However, it is a complete black box with no ability to shape or manipulate traffic for your application’s needs (say, to send DDoS traffic into oblivion instead of blasting your web servers). For this reason it can make sense to run your own load balancer, such as HAProxy, right behind ELB.

In general, configuring HAProxy to sit behind ELB is no different than any other hosting setup but there are two settings that can improve operational efficiency and prevent some common problems.

Monitoring

We are big fans of Passenger for serving our Rails applications but until the release of Passenger Enterprise there was always a few second delay on deploys where Passenger shut down existing processes before starting up the new code, leading to queued requests waiting for the new processes to boot. When we had ELB’s Health Check hitting a direct application URI, this pseudo downtime was registering as full downtime in ELB, removing the servers from the rotation, leading to actual downtime or reduced capacity until ELB picked up a valid request again.

To work around this, we moved the monitoring URI to HAProxy itself using the monitor-uri option, and updated ELB to Health Check against this URI. To ensure that we aren’t ever serving requests to a HAProxy server with no valid backends in a given availability zone (unless everything is down), we configure every HAProxy server to talk to every Web server in our EC2 account, but we increase the weight of servers in different availability zones. For example, for a HAProxy box in us-east-1a we have the following:


backend 
  server ec2-us-east-1a.server:80 weight 25
  server ec2-us-east-1c.server:80 weight 100

With this every level of the application in question has redundancy. From availability zones to ELB to HAProxy to our app servers, every layer can survive a failure without taking the application down.

Timeouts

In this setup, ELB is always going to be your “client” to HAProxy and if your timeout client value is too small you are going to see a massive amount of <BADREQ> <NOSRV> errors flooding your HAProxy logs. It turns out that the monitoring side of ELB likes to open up a couple of sockets to the backend servers without sending any data down those sockets. These sockets have a timeout of 60s and reconnect immediately upon closing. We previously had a four second timeout on these connections. This resulted in logs getting spammed with the following:


http http/<NOSRV> -1/-1/-1/-1/4000 408 212 - - cR-- 17/17/0/0/0 0/0 {} "<BADREQ>"
http http/<NOSRV> -1/-1/-1/-1/4001 408 212 - - cR-- 16/16/0/0/0 0/0 {} "<BADREQ>"
http http/<NOSRV> -1/-1/-1/-1/4001 408 212 - - cR-- 17/17/0/0/0 0/0 {} "<BADREQ>"
http http/<NOSRV> -1/-1/-1/-1/4049 408 212 - - cR-- 16/16/0/0/0 0/0 {} "<BADREQ>"
http http/<NOSRV> -1/-1/-1/-1/4001 408 212 - - cR-- 17/17/0/0/0 0/0 {} "<BADREQ>"
http http/<NOSRV> -1/-1/-1/-1/4001 408 212 - - cR-- 17/17/0/0/0 0/0 {} "<BADREQ>"

The -1/-1/-1/-1/4001 408 is saying that the request timed out at 4001ms, and haproxy returned a 408 Request Timeout. The cR-- section tells us that HAProxy force closed the connection itself (c) while waiting for headers that never came (R). You can read more about the log format at the official documentation’s Logging page.

After changing the timeout to 60 seconds, we saw the following drop in overall log volume (courtesy of Loggly):

These few changes have given us a very reliable server setup and is working to keep all logging as relevant as possible, and with HAProxy in the mix we have even more control over application scalability on top of what EC2 already provides.

Inter-Service Authentication with SSL

2012-12-17T15:56:56-05:00

At Collective Idea, we love building web services. Oftentimes we also build the client applications that consume those services.

One of the major challenges with a service-oriented architecture is authenticating communication between the client and the service as well as between services.

Oh, You Mean OAuth!

OAuth is a good candidate for this type of authentication. It provides flexibility in terms of scoped authorization levels, securely signed requests, token expiration and all sorts of other goodies. And these things are great, especially for public-facing applications, where anybody could register through your OAuth provider.

But if you’re only dealing with a small number of controlled, trusted clients, implementing an OAuth provider for each service might be a bit much.

Then What?

Every request to the services needs to identify its client application and more importantly, it needs to be verified. What we need is a tried and tested way to communicate trust.

Enter: SSL

We usually see SSL in the form of an “https://” URL in our browser, and a little lock icon to make us feel all warm and fuzzy. That feeling of security comes from the fact that our browser has identified the website and we can trust that they actually are who they say they are.

This sounds awfully close to what we’re looking for. If an SSL certificate can verify the recipient of a request, we should be able to reverse the process to verify the sender of a request.

SSL Authentication

To set up our SSL authentication, each client application needs its own SSL certificate. So first thing’s first: we need a private key with which to sign all of these SSL certificates… one key to rule them all. Signing all of the client certificates with the same private key will allow us to verify their authenticity later.

openssl genrsa -out master.key 1024

Now that we have our master key, we can start generating SSL certificates. Each certificate will require a new CSR (certificate signing request). CSR generation will prompt for information about the certificate and we’ll take that opportunity to identify the client application that this certificate belongs to.

# Enter "web-client" for "Organizational Unit Name."
# The remaining attributes aren't important.
openssl req -new -key master.key -out web-client.csr

With the CSR in hand, generating a signed certificate is easy.

openssl x509 -req -in web-client.csr -signkey master.key -out web-client.crt

This certificate is given to the client application and every request originating from the client will include the contents of that certificate in a custom header.

Meanwhile, in the service… when a request is received we’ll need to assert that the certificate provided is authentic. We do this by verifying that the certificate is signed by the master key. The best part is that the service doesn’t even need the private, master key to do that verification. It can use a public key derived from the master key instead.

openssl rsa -in master.key -pubout > master.pub

Every service gets this same public, master key and uses it to verify all incoming requests.

Finally, Ruby

Assuming your service is written in Rails, here’s how that verification might be implemented:

class ApplicationController < ActionController::Base
  before_filter :require_authentication

  private

  def require_authentication
    unless current_certificate.verify(public_key)
      head :forbidden
    end
  end

  def public_key
    @public_key ||= OpenSSL::PKey::RSA.new(ENV['AUTH_PUBLIC_KEY'])
  end

  def current_certificate
    @current_certificate ||= OpenSSL::X509::Certificate.new(request.headers['X-SSL-Auth'])
  end

  def current_client
    current_certificate.issuer.to_a.assoc('OU')[1]
  end
end

The public key used for verification is kept in the server’s environment at ENV['AUTH_PUBLIC_KEY'] and the current request’s certificate is pulled from the headers.

The certificate responds to the verify method which accepts the public key and simply returns true if the certificate is authentic or false otherwise.

You’ll notice the current_client method as well, which extracts the organizational unit from the certificate that we set during the CSR generation. This identifies the client application making the request. With that information, the service can implement any sort of access controls that may (or may not) be required.

In the Wild

At Collective Idea, we think SSL certificate authentication is a cool idea and in some cases, a great fit. It’s also comforting to know that Square uses this same approach for authentication between its own services.

In fact, this post was inspired by a talk by Square’s Chris Hunt at RubyConf 2012. His talk omits some of the details of certificate generation and verification but it’s a fantastic resource for good service-oriented architecture.

I’m personally new to SSL-based security so please leave a comment for how this might be improved or expanded upon. Thank you!

So We Bought a Printer

2012-12-12T11:44:54-05:00

We needed a printer for our new office.

Well, we didn’t really need one. We don’t print all that much these days, not even boarding passes anymore. Really, I wanted a scanner so we could reduce our paper floating around. I wanted one with a document feeder so I could scan multiple pages at once. If it came attached to a printer for those few times we wanted it, even better.

Shopping for printers is hard.

Really hard. Especially since I don’t care about DPI, PPM, or any other marketing BS. Also, I don’t care about the software. I want it to work with my Mac. Work, not make toast, just work.

Maybe I’m weird, but I haven’t cared at all about printers in years. At our last office, we shared a Dell laser that worked well enough, and I didn’t have to worry about it.

Research

Dear god there are a lot of printers on the market. Ok, I limited myself to wireless, AirPrint-compatible models with a document feeder. Still way too many. Ok. Pick a few at random and start reading reviews¹. Not very helpful. Anything under $150 seems to be flimsy, and no good idea about how they really work on WiFi with Macs.

Software

Finally deep in the HP excuse for a website, I found what I was looking for: No need to install the software on a Mac, just connect and Apple’s Software Update will do the rest. As I said, my big thing was scanning. No idea how this actually works. I’ve had bad experiences in the past².

I was happy to discover that Preview.app can scan direct! As long as you’re on the network, you get a “Import from…” menu item. Ok time to go for it.

Preview’s file menu

Preview ready to scan my multipage document

Buying

I was sick of research, so time for impulse buy. HP had the best feature + review combos, so I was all geared up to get a HP Photosmart 7520 but the guys convinced my that laser would work better with our low print volume. Time to drive to the store so I could get it done.

After hemming and hawing over the price and size, I gave up and grabbed the HP Laserjet Pro 200 M276nw. Color laser, wireless, scanner tray, all the goodies. Also the thing is freaking huge. The wireless module must be the size of a cinder block, but whatever.

Setup was easy, and it felt great to toss the 3 CDs straight in the trash³.

Does it work?

Like a champ. Slack printed up a “Test Page” as soon as I got the wireless setup, and then I tried scanning a 3 page document. Added the printer in System Preferences, no issues, and then went to Preview and it Just Worked. No hassels, it got all the pages and I was happy.

So now we have a printer. Hope this helps someone out there decide. Either way, I’ll be able to print up some passive aggressive signs for the new office and scan the few papers I still have around.

¹ This is how I found out that the HP 7520 dropped the Facebook support that the 7510 had. What’s that, you didn’t know printers had Facebook? Me either.

² Seriously, in 2012 scanning is still hard?

³ Well, except for the waste & environmental impact…

Desks 2.0

2012-12-11T11:47:01-05:00

At Collective Idea, we’ve been hard at work moving into a bigger, new and improved office space. We have big plans for the space, including refinished hardwood floors, Flor carpet, Nest thermostats and more. But before all the bells and whistles, we need to be able to… work.

We don’t need much: a solid internet connection, a handful of cinema displays and some good stand-up desks. That’s where I come in.

When I was 12 years old, I built a wood shop in the back of my parents’ garage. I made and sold Adirondack chairs throughout high school, taking orders from all around the U.S. and even Norway. In fact, building the website for my carpentry business was my first foray into web development.

So when Daniel realized the lack of stand-up desk options out there and started talking about the possibility of building our own desks for the new office, I jumped at the chance.

The Goal

The stand-up desks that are available for purchase are generally pretty expensive. Plus, the options are limited so there’s always that nagging feeling of having the same thing as all the other standers out there.

So the goal was simple: build an inexpensive, one-of-a-kind desk that fits our space and style.

The Top

The first picture that came to mind is of the monolithic wooden tables in Apple retail stores.

So I started looking at butcher block tabletops. The problem was that the 4×6’ size we wanted was either unavailable or terribly expensive. That’s when Brian saved the day with this suggestion to use two inexpensive IKEA countertops:

We all loved the idea and decided on the NUMERÄR countertops in beech because they would pretty closely match our maple hardwood floors.

The Power

The difference between the desks we wanted to build and the conference table above is that we need power; to our cinema displays as well as whatever gadgets we’ll be charging during the day.

The easiest option was to do nothing at all. We could simply drape the power cords off the back of the desk. But we quickly ruled that out because, well… it’s ugly. Plus there’s no “back” of the desk since we intend on setting up stations all around the desk’s perimeter.

The next idea was a trough. By separating the two countertops a few inches, we could create a channel in which to mount outlets and stow power bricks. The downside is that we lose a little of the monolithic look we were going for by dividing our tabletop in two. Plus, it wouldn’t allow us to easily set up stations on either end of the desk.

The solution we landed on was a mix of the two; to cut out a slot in the middle of the tabletop. Eventually, we’ll install a “floor” to the power management slot where we can install outlets, throw our power bricks, etc. but for the first iteration we’ll simply drop our power cords through it.

The Base

We tried to contact the builder of the IKEA conference table above to ask how he built the base of the table, but we got no response. So we had to go it alone.

We briefly considered wooden legs to match the tabletop but it would have required a lot of supporting structure and/or fancy joints that we didn’t really have the time (or patience) for.

Our next stop was stainless steel hairpin legs. We found some that looked great and could be made custom, hopefully to accommodate our stand-up height. They also have levelers for uneven floors (which we have).

Unfortunately, we couldn’t find anyone who would make us legs that were quite as tall as we were looking for, at about 41". Most topped out at around 36".

Next strategy: frantic Googling. And we came up with simple 3" tubular legs.

These legs can be custom cut but already come in a 40¾" height, standard. Each leg can support a 300 lb. load which is plenty, even for our heavy 200 lb. tabletop.

If we use a half-inch spacer between the leg and the tabletop, the surface of our desk will be 42¾" high, which is the height we’ve all become comfortable with.

So now that the legs are chosen, it’s time to start building!

The Build

I recently moved into a new house and I don’t have a wood shop set up yet so the plan was to build one of the six desks from start to finish and use it as a workbench for the remaining five. It would also give me some practice so that hopefully, the rest would come together more smoothly.

Routing the Tabletop

In order to make the slot for power management, we need to cut a groove in the long side of each countertop that would make up our tabletop. A good router is the perfect tool for the job, along with a pattern bit.

Using a straightedge and a soup can, I drew the rounded shape of the groove on a scrap of MDF (Medium-Density Fiberboard). I cut it out with my jigsaw and sanded to remove any bumps or imperfections. This one cut would be the template for all twelve finished grooves so it’s important that it’s done carefully.

After hoisting the countertop onto my workbench, I clamped the MDF template onto the edge and traced the inside of the curve with a pencil. Beech is a hard wood so I’d have to remove most of the material with my jigsaw before I could use the router to finish it off with a clean edge.

The curves of the groove have a tight 1½" radius which is difficult to cut with a jigsaw, especially when the stock is 1½" thick. To make it easier, I cut into the curve at several places along the radius. This makes cutting the entire curve easier by simply connecting the dots.

With the curves cut, I finished the groove by cutting lengthwise down the countertop. This gives me my rough cut that’s ready for the router.

The pattern bit I’m using in the router has a ball bearing at the top that rides along the MDF template and cutters below that remove the material. Slowly working from left to right, the router has no trouble eating through all of the excess material that the jigsaw left behind. We’re left with a clean edge.

A quick sanding removes any ridges left by the router and knocks down the hard edges and corners.

That’s it for this one countertop. One more just like it and we’re ready to start assembling the tabletop.

Attaching the Stringers

The first thing to do was flip the countertops over so that we’re working with the underside. Sounds obvious, but important! Each countertop has one better side intended to be the top surface. IKEA stamps the underside.

I cut the stringers from the same MDF I used for the groove template. These will attach the two countertops to each other and provide locations to attach the legs. The MDF is ½" thick, giving us our desired height once the legs are attached. Each stringer has four holes pre-drilled at each end to accept leg brackets. The holes should be big enough for the screws to pass through easily. We don’t care about the screw threads gripping in the MDF.

The stringers are positioned so that the center of each leg is 8½" from the edges of the table. This will put the surface of the 3" legs 7" from the edges of the table to allow enough room between them for stools but also help avoid accidental kicking. Once the stringer is in position, we drill through one of the holes in the stringer and attach the bracket and stringer to the countertop. Then we make sure the other end is still in position and repeat. When one screw is attached in each countertop, we can pre-drill the remaining holes.

Starting with the opposite corner of each bracket, I secure the remaining screws. I’m using #12 1½" self-tapping screws. They have great grip and are easily driven in using a power nut driver.

MDF is fairly flexible so in order to minimize the length of the stringer that is subject to bending forces, we’ll attach the length of the stringer to the countertops using standard #10 1½" wood screws. Each screw hole is marked, pre-drilled and countersunk.

You can gauge how far to countersink by drilling until the head of the screw fits in the opening.

Then all screws are driven.

What Not to Do

I learned the hard way that if your pre-drilled holes aren’t big or deep enough, the beech countertops are so hard that you can spin the heads right off of the screws. As usual, trial and error wins the day!

Adding the Supports

Butcher block is resistant to flexing lengthwise but our tabletops have a structural weakness. The load of cinema displays near the center of the table will subject each countertop to twisting forces. To distribute those forces across both countertops, we’re adding angle iron supports near the center. Angle iron is extremely resistant to flexing.

Each support is attached to the countertops using four #14 1¼" self-tapping screws; two at the far ends and two near where the countertops meet.

Be careful when pre-drilling these holes. Without the addition of the ½" stringer, it’s easy to pre-drill too deep and drill through the top surface of the tabletop. To prevent that, wrap the drill bit in tape at the proper depth so you know when to stop drilling.

Once pre-drilled, we tighten the supports to the tabletop with screws and washers.

Wrapping Up

At this point, this tabletop is done! Flip it over and admire.

Placing the table in position is a two-person job (at least). First we put the tabletop upside-down on the floor where we want it to be. Two people are needed because each tabletop weighs around 200 lb. and they’re big! The stainless steel legs are then hand-twisted into the brackets.

Finally, we flip the table over into its upright position. It’s important to keep an eye on the legs while flipping the table. If a leg gets caught underneath the table while flipping, the lateral force can damage or break the leg and bracket. These legs are built for vertical forces.

The Result

We’ve been working from our new office for a couple months now and we love it. After applying a few coats of a clear satin finish, the tables are working and looking great. All told, we built two sit-down height and four stand-up height desks.

We also finished installing power management in each desk and made them chainable to clean up cord clutter underneath.

So far, we’ve been loving them. Come on by and see for yourself. We’ll save a spot for you!

Building Office 2.0

2012-12-10T16:50:34-05:00

I had long had my eye on our new office space. The two previous tenants are friends and I loved the location.

We first looked at it over the summer. It was just an empty space but it had the advantage of being fully built-out and well-kept.

blurry photo from our first visit

I knew we had to make a plan or we’d be sitting on the floor, so we brought in our secret weapon: Interior Designer Jenn Ryckbost. Jenn quickly had an accurate mock-up of the entire space in SketchUp and we started designing what the space should look like.

early renderings

We put the SketchUp file in Dropbox and the whole team started to give input. We tried to make the entire process transparent. We had a Basecamp project for discussions and tried to involve everyone in any part they were interested in.

Like building software, spending time in the design process got us a better result. We iterated daily, changing anything we didn’t like, and only placing orders when we were satisfied.

most of the furniture is in

It was also great to have help with picking colors. If left to me, I probably would have left everything off-white. Jenn built us a great palette, and now we have great splashes of color, with a chalkboard wall too!

an early color palette

Steve got to work building most of our tables and we ordered everything else we needed. The day we got wifi setup, we didn’t have much more than our stand-up tables but we started working out of the new space. Ordering furniture can be slow (finally got our last batch a week ago) but it was great to know what we had in store.

The final result looks amazingly (but unsurprisingly) like our designs. We’ll post more photos soon or you can come to our Open House to see for yourself!

Final rendering

Office 2.0

2012-12-12T11:44:36-05:00

We’re now working out of our Office 2.0 (and having an Open House Dec. 13^th). We found a beautiful spot in the heart of downtown Holland with a bit of space to grow.

We plan to share a lot about what we did, decisions we made, and details about how we work. We’ll use this post to collect those future posts too.

Posts in this series:

Posts about our first office, Office 1.0 include:

Playing with Go: Embarrassingly Parallel Scripts

2012-12-03T10:03:27-05:00

I recently needed to take a list of domain and find which ones point to a specific IP address. For a small list, say less than 10, manually running dig in the console would work great, but this list had almost 800 domains so I needed a script. As domain lookup is a network request and thus very slow, setting up the domain requests in parallel made sense. I could easily just do this in Ruby, my language du-jour, but I’ve done this type of thread work before and frankly it can be tedious to set up, fragile, and still won’t have access to all of my system’s resources due to the GVL¹. I’ve been keeping an eye on Google’s Go for some time now and decided to see how it handled this problem.

I’ve been intrigued by Go since it was originally announced about three years ago. Here was a compiled, fast, light-weight, low level language with many of the features we take for granted these days, such as garbage collection, while also adding on a very sophisticated concurrency model similar to what’s found in Erlang: very lightweight internal processes managed by the runtime. Sounds like a perfect fit for my requirements.

The code I ended up with is here: https://gist.github.com/4170926. For the sake of comparisons I built a sequential version of the script as well as the parallel version and added timings for running both scripts against the full list of domains.

Running these scripts for yourself is a one-liner: go run [script.go]. The input file domains.txt needs to be a newline-delimited list of domains. I’ll go over the more confusing parts of the two scripts to help with understanding what’s really going on here.

Objects?

Go’s object model is very close to C’s: structs with data and methods that operate on said structs. Both scripts only use a small, two-element struct, DomainMap, to keep track of the IP address found for a given domain. I use the short-form to initialization new instances of the DomainMap structure. The order of values maps directly to the order of the defined fields at the top of the scripts.

type DomainMap struct {
  Domain string
  IpMapping string
}

object := DomainMap{domain, ipAddress}

object.Domain == domain
object.IpMapping == ipAddress

Error handling

Go does error handling by returning multiple values from a function, where the second return value is expected to be a value of type error. You can ignore this with the _ variable.

rawIpAddresses, _ := net.LookupIP(domain)

Parallelism

The parallel version of the script has some new concepts that need explaining, particularly goroutines, channels, and channel communication.

A goroutine is a very lightweight process, sort of like a Ruby Fiber. Creating one is simple:

go domainLookup(responseChannel, domain)

Go will grab the function call after the go keyword and execute it in parallel. However, given that we’re no longer in the main process, we can’t just return values from the function. We now need a different way to get the return value. This is where channels come in.

responseChannel := make(chan DomainMap)

As Go is a statically typed language, we need to define the type of channel being created. Channels can only accept data of the same type as the channel. Communication through channels is done with the reverse-stabby operator <-, which should be read as “the data on the right side is flowing to the left side”:

// Write into a channel
returnChannel <- DomainMap{domain, ipAddress}

// Read from the channel
domainMap := <- responseChannel

And that’s all the special syntax. The only real difference between the parallel and sequential scripts is the map-reduce-esque setup to wait for all the goroutines to finish. I didn’t need to worry about thread pooling, system capabilities, or thread safety. Go makes it so easy to write truly parallel code that there’s no excuse not to anymore. I was able to run almost 800 goroutines (one per domain) all throwing out DNS queries and coming back in less than 10 seconds, in a script that doesn’t even look like it’s running in parallel.

Now that Go 1.0 stable is out, it’s a great time to get familiar with this language. I highly recommend checking out the Tour of Go for basic introductions into every major feature of the language, and there’s a ton of documentation on the main website golang.org. For the little bit of time I’ve played with Go now, I see a very bright future for this language.

¹ Global VM Lock, more about Ruby’s concurrency here: http://www.engineyard.com/blog/2011/ruby-concurrency-and-you/