My professional life has run roughly in decades: a first decade of getting into the software industry, doing lots of consulting and contracting, and observing businesses being run. The second decade grew out of the first decade and was spent founding a product company and figuring out - alongside a lot of really smart people - how to grow Spreedly into something successful. The boundaries between these vocational periods are of course fuzzy, but over the course of 2018 I’ve unknowingly been wrapping up my second decade, and the end of 2018 has become a clear break from my time working on Spreedly on a day-to-day basis.

Here’s the thing: Spreedly’s in a fantastic spot to win. I feel really good about stepping away right now with the company profitable, compounding sales growth, a great leadership team, and a product that has one of the lowest tech debt loads of any production system I’ve ever worked on. And I’m not disappearing into a black hole: I’ll remain a shareholder, continue to advise the company via my position on the board, and be cheering the team on from the sidelines every step of the way.

Why am I leaving? As these things usually are, it’s complicated. Here’s the short version: there were differences of opinion on my role as Spreedly grows past 50 employees. And I mean “opinion” sincerely - I haven’t even been present at a product company as it went from 50 to 100 employees before, much less been an executive working on it at that stage, so my perspective shouldn’t weigh more. And since I wasn’t able to get on the same page about my role going forward, I didn’t want to get in Spreedly’s way as it continues to grow.

What’s next for me? I ask rather, “What better time for a mid-life crisis?” and I’m only half-joking: while I’m not in crisis, I have about a million ideas for what I might want to work on next, but 2019 is all about getting back to the basics of creation: writing consistently and programming consistently. These two skills have formed the bedrock of my career to date, and I’m excited (and terrified) to pull my creative focus away from Spreedly and put it into a series of smaller projects designed to hone these two fundamental skills while exploring a series of hypotheses about what my next major endeavor might be.

Alongside the fundamentals, I also expect to do a lot of networking in 2019, so do drop me an email if you want to grab coffee or even just video chat. This goes double if you’re a CEO, CTO, or VP of Engineering in a company in the 10-50 employee range trying to figure out how to wrangle engineers and the software they build - that’s where all my scars are from and I feel like I can speak with some confidence about what works and what doesn’t.

Weep not for me! I’m super excited to dig into the third decade of my career. And worry not for Spreedly: it’s set up to win, top to bottom, with an amazing team building an amazing product. Instead, grab some sunglasses: the future is bright everywhere you look. 😎

This little vignette starts with Warcraft. See, I have kids. Quite a few kids. And they’re getting old enough it’s fun to play games with them. Now, if I was a normal father, I’d buy them modern computers and we could play modern multiplayer video games like normal people. I am not a normal father; rather, if you’ll recall, I’m a nerd, which makes me a nerd father. Which means, when I think, “it’d be fun to play a multi-player video game with the kids,” my mind harkens back to some of the most fun I had playing multiplayer as a kid: Warcraft.

More specifically: Warcraft II: Tides of Darkness.

OK, first step is, can we even play it on a computer made in the last decade? Turns out we can - and quite easily - using DOSBox. DOSBox is a nerdy dad’s best friend when it comes time to introduce his kids to the classic experience of a LAN party. Since around here even our “old computers” run OS X, I’m a big fan of Boxer for making DOSBox a cinch to setup and manage.

So far things aren’t too nerdy, but we’re nowhere near done yet. The next step is networking, and this is where it gets interesting. Old games mostly use IPX, and IPX is pretty much a dead protocol on modern networks. DOSBox has a cool trick up its sleeve, though: it translates IPX to UDP, and it works without a hitch.

While hitchless, this IPX-over-UDP setup is kind of a pain; one player has to set up their DOSBox to be the server, and the other players have to know and enter the IP address of that player to connect. Kind of a pain in the tuckus to do every single time we want to play, so surely we can do better, right? Wouldn’t it be awesome if the network at the Talbott house had an always-on IPX server on tap? Why yes it would!

Since we’re not the first ones on the internet to wish for such a thing (is it possible to be the first at anything on the internet these days?), some searching around turns up ipxrelay, which is such a beautiful, dependency-less, single-purpose tool it makes me want to weep for joy. A quick:

1
2
3
4

git clone git://git.zytor.com/games/dosbox/ipxrelay.git
cd ipxrelay
make
./ipxrelay

And we have a working standalone IPX server. Now, I wouldn’t be comfortable running this on the big bad internet without a deeper understanding of its operation and security, but for internal LAN parties? Sign me up!

But that brings us to our next challenge: where do we run it? I don’t maintain an always-on server on our home network, nor do I really want to. But actually that’s not true, since I have three routers running Tomato Shibby, and they’re nothing but mini-servers that have lots of idle capacity that could be put to good use. So I decided to get ipxrelay running on my biggest/fastest Tomato router. No sweat, right?

No sweat, except that running a program written in C on a device like an Asus router means cross-compiling that program. My highest-powered router is a MIPS device, and Tomato itself has a very distinct and limited set of libraries available, all of which must be taken into account. But people do this all the time, so should be easy, right?

I’ll save you the multiple weekends and evenings of futzing, hair-pulling, puzzling deadends, and searching in circles. For some reason, nowhere on the internet could I find: “To cross-compile a program for a Tomato Shibby target do A, B, and C.” Until now, since that’s exactly what I’m going to do in as clear a terms as I can for the sake of future searchers.

Tomato Shibby Cross-Compiling for Dummies

It’s pretty simple really:

1. Be on a Linux box
2. Checkout a copy of the Tomato Shibby code
3. Use the included, pre-built compiler to build your application

Well, it’s simple now that I know what to do. Follow these steps to avoid great pain!

1. Be on a Linux box

So first I need a Linux box to do the building on since I’m on OS X and I don’t need or want the pain of getting cross-compiling working in an environment different than what’s already used by the Tomato Shibby developers. In real life I built a VM from scratch, but with time for reflection and a blog post to write I’ve remembered I have Vagrant set up and ready to go here already. Here’s a little ‘cast of me setting up a Debian Vagrant box:

2. Checkout Tomato Shibby

Now that I have a Linux environment to build in, I need to clone the Tomato repo and checkout the tag my router is at:

Using --branch and --depth reduces the size of the checkout somewhat.

3. Use the Cross-Compiler

One rule of step-wise recipes is that one step will always be more complicated than the rest. This is that step. We need to take the program we want to compile, tweak its Makefile to use our cross-compiler, and build the program:

Wow, it looks easy now that I know exactly what to do. sigh

You can copy and paste from the screen cast above - slick, right? - but just in case, the patch referenced is available here: https://gist.github.com/ntalbott/75a8aa269b2d310bf02f, and the ipxrelay repository is here: http://git.zytor.com/games/dosbox/ipxrelay.git/.

Running ipxrelay on the Router

Now that we have a cross-compiled ipxrelay, we need to get it saved somewhere safe on the router, and ensure it’s running with the right options. I’m running mine from a USB flash drive I have plugged in, but I’ll leave the setup of that as an exercise to the reader. Once it’s somewhere safe - your options are /jffs (wiped each time you restart the router), some kind of added storage like a my USB drive, or a memory card mounted at /mmc if your router supports it - the next step is to get it fired up. Here’s the Firewall script I have in the Administration->Scripts area of Tomato to start it up:

1

/mnt/ROUTER/bin/ipxrelay --port 2130 --address 192.168.1.1 --pidfile /var/run/ipxrelay.pid

My IPX relay is going to run on port 2130, only bind to my internal IP, and put out a pidfile. The shutdown script then uses the pidfile to properly clean up:

1

if [[ -e /var/run/ipxrelay.pid ]]; then kill `cat /var/run/ipxrelay.pid`; rm /var/run/ipxrelay.pid; fi

And there you have it, an always persistent IPX relay on your internal network! You might think we’re done, but we can go even deeper!

DOSBox Automation

What’s better than having IPX networking consistently available? Using that fact to make our DOSBox games auto-networked, of course! The final thing I did was to create a couple of batch files to auto-configure the IPX setup so that when we launch the game, it Just Works™. We need to put three files in our DOSBox C: to make this work: first, FIND.COM from FreeDOS since DOSBox doesn’t come with a FIND built-in. Second, START.BAT, which tries to start IPX networking, and fails out if it can’t:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

@echo off

del ipxnets >nul

ipxnet disconnect >nul
ipxnet connect 192.168.1.1 2130 >nul
ipxnet status >ipxnets
find "Client status: CONNECTED" ipxnets >nul
if errorlevel 1 goto neterror

call game
goto end

:neterror
echo Error starting networking...
echo Is ipxrelay started on the router?
goto end

:end
del ipxnets >nul

And finally, the super-simple GAME.BAT, which just makes START.BAT generic:

1

launch war2

DOSBox can be configured to launch START.BAT when it starts, and voila! You’re all set to crush your offspring - or siblings, or whoever you can convince to have a LAN party with you - at Warcraft 2!

The conclusion of this little piece is the same as the beginning: I am a total nerd. And as aggravating as it sometimes is, it’s a lot of fun, too!

But racing is such an odd thing. Running in and of itself isn’t odd; it’s just another activity that I undertake on a regular basis. I train, and I push myself, I try to get better, and I return to it as an enjoyable process for its own sake. I can run socially, carrying on a solid conversation while maintaining a pretty good clip, though I’m generally alone listening to an audio book. And I run at my own convenience: I pick the time, the place, the distance, the pace, etc.

When I line up to start a race, though - then it gets weird. For one thing, I’ve chosen to get up and run at a crazy hour on a Saturday morning, when I could’ve just as easily waited a few hours and run out my front door. Or it might be a Sunday afternoon when I’d normally be enjoying a quiet nap. Or maybe I’m at a programming conference, dragging myself out of bed after a late night of talking tech to wander through a strange city and try to find the starting line.

And it’s not like I’m actually going to run socially in a race - ha! When I race, I like to actually race, so I’m spending all of my breath and focus on pushing my body to the limits of what it can do, and I can’t spare an iota of it on the people around me. Racing is super self-centered: it’s you, the road, and an ever-ticking clock. I don’t even try to listen to anything while I race, since the necessary apparatus is too much of a distraction. And if I go to a race with a friend, I’ll wish them luck right before the gun and not see them again until one or the other of us is cheering the other across the finish line.

Then there’s the fact that in any race large enough to have an official course and chipped time tracking, there’s no way I’ll come in first. I go out and run a race knowing that I’ll make a good showing, but also knowing I’ll get trounced by some 23 year old that was smart enough to take up running 5k’s before he turned 30. I’m not even really a risk to the other guys in my age bracket since, while I love running, I love coding even more and so I won’t be spending those extra hours training that I’d need to in order to place even bronze for 30-35.

But here’s the thing: I love to race. Every time I get out there I think, “What in the world am I doing here?” and then the gun fires and I’m off and I’m loving it. Someone else has laid down a very specific set of constraints - run this route, start at this time - and now I get to push myself to do the absolute best that I can within those constraints. I’m not being social, and yet I have this whole host of humanity in so many (admittedly all very fit) varieties, and I get to run beside and behind and around them. And while I’ll never come in first, I get to race myself on the course, seeing if I can best my own best and show that I have continued to improve and grow since the last time I crossed the line.

Racing is so bizarre, and I love every minute of it.

I’ll admit, we’ve tried to do this a couple of times before, and every time I’ve been stymied by two things: first, the overwhelmingness of both trying to choose between a lot of well regarded options, and at the same time trying to figure out what we should be tracking. Second, hooking up metrics is not a trivial task, and when there are other things I can be doing that at least appear to add more value to the bottom line, I tend to quickly get distracted and pulled back into “regular work”.

The big difference this time around was that I found a couple of tools that cut way down on the difficulty of getting started, and I’m super excited to tell you about them.

Biz Metrics - Segment.io

Remember how I mentioned having lots of interesting choices, but being overwhelmed trying to pick? Using Segment goes a long way towards solving that problem. To put it in the terms of my day-to-day world, Segment is basically the “Spreedly of Business Analytics” - you hook up one time to Segment’s API, and then you can turn on dozens of different services in your site with just a quick configuration change in their dashboard.

This had two huge advantages for me: first, it simplified the API that I had to implement against, since by looking at a whole pile of API’s and building a solid over-arching API, Segment has come up with something incredibly simple and easy to use. They also have a library for just about any language, making the implementation even easier.

The second big advantage with Segment is that it makes it a cinch to trial multiple analytics services simultaneously and see which one(s) are the best fit. Tools like KISSmetrics and Mixpanel aren’t cheap, and yet they take a fair amount of effort to integrate from scratch. Using Segment in front means I can trial them in parallel without biting off two significant integration efforts.

And speaking of KISSmetrics and Mixpanel: both tools are pretty awesome. Justin is like a kid in a candy store, taking the raw information and events filtering in and building out all kinds of interesting funnels, counters, and segments. And I can tell he’s deep in the numbers, since he keeps finding places where I’m counting things wrong!

Devops Metrics - Librato

The biggest stumbling block to me with devops metrics has always been how many moving pieces there are. You have to figure out what to collect, collect it somehow, get it into some kind of data store, then hook up something to consume that data and eventually expose it in some kind of useful format. It’s all a bit overwhelming when you’re just getting started, and at the local Devops Meetup I was explaining some of my frustration to a few sympathetic ears from Github and Heroku, which both have excellent internal devops metrics.

One of them suggested I get started with Librato, and boy am I glad they did. Leveraging Librato’s awesome librato-rails gem, I had some basic metrics showing up in a dashboard within a few hours of getting started. And since Librato is taking care of the infrastructure, I’m in a great place to add more metrics and incrementally increase our visibility into the health of our infrastructure.

We’ve always had some visibility into our business metrics and our infrastructure, but like so many things when you’re in startup mode, it was one kludge after another. It’s so exciting to feel like we’re on a solid footing now, both in terms of having better metrics today and in being able to iteratively expand what we’re monitoring going forward.

Relix 2.0.0 brings a major version bump, due to the fact that it now requires Redis 2.6. This allows us to leverage Lua scripting, which is a big win for some use cases, especially since Relix does all it can to pipeline Redis requests.

The new feature driving the usage of Lua scripting is the ability to index and retrieve the list of values being indexed by a multi index:

1
2
3
4
5
6

class User
  relix.multi :account_id, index_values: true
end

# Enables this call
User.lookup_values(:account_id)

So whereas normally you’d look up all the users with a given account_id, lookup_values allows you to look up all the account id’s that are indexed. This is super handy for doing aggregate lookups by a multi index:

1
2
3
4

User.lookup_values(:account_id).each do |account_id|
  users_for_account = User.lookup{|q| q[:account_id].eq(account_id)}
  # Aggregate processing for the users in the account
end

2.0.0 also adds a deprecation mechanism and uses it to deprecate direct access to IndexSet#indexes in favor of IndexSet#[].

You can grab 2.0.0 hot off of Rubygems, report any issues you encounter on Github, and contact me via the details on my Github profile if there’s anything I can help with.

And let me know if you’re using Relix - I’d love to hear about anything and everything it’s being used for!

For me, the “do things” part is easy. Well, easy is grossly oversimplifying my relationship with creating, but suffice it to say that if I’m not regularly making something I start getting depressed. I am wired to derive satisfaction from the act of creation, and so I have inbuilt feedback loops that ensure I return to “doing things” with regularity. I still struggle to make sure the things I’m doing are on the whole worthwhile - the occasional diversion into writing a Lua script to run a Minecraft bot are fun but can easily be overdone - but accumulating a list of things I’ve done isn’t hard.

It’s the “tell people” that I so easily neglect. As I’ve worked on Spreedly over the years, not talking enough about what we’re up to is far and away my greatest failing. Thankfully Justin now does an awesome job of getting the word out, but that doesn’t mean I can slack off, just that I can focus in on talking about the technical side of what I (and Spreedly) are up to, which is the part I’m best equipped to talk about anyhow.

And so after we got the new spreedly.com out the door and I needed some time to decompress, I decided to spend the next week taking all the cool things I and the team had done over the preceding weeks and start writing about them. It actually beats taking time off in a lot of ways since it actively calms the brain down by taking all the stuff it’s full of and putting it down in words.

And with a week of posts under my belt, and dozens of topics queued up (it’s been too long!), I’m getting the “tell people” ball rolling again. I hope if you’re reading along you’re enjoying the posts, and I’d love your feedback on app.net, via email, or any other way you can get to me (there are quite a few). And a submission or upvote on Hacker News is always welcome; much as it can be a distraction, I still love participating in the HN community.

For now my plan is to keep the weekdays pretty technical, and publish less technical posts (such as this one) on Saturdays. Of course, my goal is that it doesn’t stop with the posts I already have written and waiting to publish; while I don’t plan to keep up daily posts, I’m doing my best to build a habit such that “tell people” is one of the “things I do”. Writing is an act of creation I enjoy, and the more consistently I do it the more I enjoy it and the better I am at it. So expect to hear a lot more from me in the coming days!

Why add the overhead of extra services? And yes, they do have overhead - adding more moving parts always adds complexity. I want to talk a lot more in the coming days about what we did, how we did it, and what compelled us to go down this path, but today I’m just going to tease with the bottom line payoff of getting this first piece deployed:

I just stripped over 4,000 lines of code from this app. That code doesn’t have to be loaded, supported, or secured any longer in this context. That’s what I call a win!

Thing is, you can have the best automated test suite ever, and you can have a great set of passive monitoring tools, but nothing beats an active check of the application after a deploy. After yet another time of making an insignificant tweak that turned out to work just ever so slightly differently in production in such a way as to take one of our apps down post-deploy, I finally said, “Enough!” and made Duff code us up a solution.

That’s right folks, this one isn’t about tooting my own horn, but rather about tooting the horn of fellow Spreedly developer Duff’s since he doesn’t do it nearly enough. After we fixed my latest gaffe (which thankfully “only” affected the UI), Duff tackled the problem of adding an active sanity check to our deploy processes that would just do a basic check (or checks) and tell us as part of the deployment itself if the check failed.

I love the term smoke test for this check; originally used in plumbing, the smoke test was when they’d force smoke through newly installed pipes to see if any smoke seeped out. It was later adopted by the electronics industry as well: “You plug in a new board and turn on the power. If you see smoke coming from the board, turn off the power. You don’t have to do any more testing.” ¹ The smoke test is that first test you run on a newly assembled system just to make sure it’s not immediately and obviously broken.

So Duff dug in and added a new recipe to our internal deployment gem:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

Capistrano::Configuration.instance(:must_exist).load do

  def all_smoke_tests_passed
    send_to_hipchat("All smoke tests were successful.", color: 'green')
  end

  def smoke_tests_failed(messages)
    puts "\n\n******************\n\nSMOKE TEST FAILURE!!!\n#{messages}\n\n******************\n\n"
    send_to_hipchat("http://f.cl.ly/items/2T3c2B2m3H2l3R2Y3o2z/Image%202013.02.15%201:50:04%20PM.jpeg", message_format: 'text', color: 'red')
    messages.each do |message|
      send_to_hipchat(message, color: 'red')
    end
  end

  task :smoke_test do
  end

  after "deploy", "smoke_test"

end

It’s pretty simple, it just hooks in after deploy and runs a (by default empty) task. It also provides convenience methods for taking action upon success or failure. Usage is simple:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

require 'potluck/recipes/smoke'

task :smoke_test do
  failed_tests = []

  curl_result = `curl -s -w "%{http_code}" https://id.spreedly.com/signin -o /dev/null`
  failed_tests << "Id smoke test FAILURE." unless (curl_result == "200")

  curl_result = `curl -s -w "%{http_code}" https://spreedly.com/ -o /dev/null`
  failed_tests << "Public smoke test FAILURE." unless (curl_result == "200")

  if failed_tests.empty?
    all_smoke_tests_passed
  else
    smoke_tests_failed(failed_tests)
  end
end

All we have to do for a project is override the :smoke_test task, do whatever makes sense for that particular app, and use the convenience methods to report the result. Now if a smoke test fails, the whole team knows right away as it gets blasted into our Hipchat room:

We of course also see if the smoke test passed (though it’s much less “in your face”):

It’s amazing how comforting it is to see that line show up after a deploy.

Of course, there are plenty of other ways we test our apps, both pre- and post-deploy, but the smoke test is a great addition to the overall lineup and I’m grateful to Duff for getting them going for Spreedly. And I’d encourage other developers to consider adding an active check to the deploy process to see if anything starts smoking once everything’s live.

¹ Smoke Testing on Wikipedia back

Only one problem: spreedly.com was currently handled by our Subscriptions application, and we didn’t want to change either the API urls or the hosted payment page urls for that service. And it’s not just a temporary problem, either: we’re not discontinuing the Subscriptions product, and while we may eventually move it to its own subdomain, that will be a slow and careful transition that could easily last over a year.

So we were clearly going to have to serve up two different applications off of the same domain. The plan was to continue defaulting urls to the Subscriptions app, but to hijack a specific set of pages and paths and point them at the new app. I worked up a list of paths that should point at the new path, and our handsome and talented devop John worked out how to do the mapping in our frontend load balancer. The production setup actually ended up being super simple since we already use nginx proxying to unicorn, and it was simply a matter of having a particular set of paths proxy to a different backend. And because there’s no special infractructure to support it there’s also no performance overhead to sharing the domain.

But… what about when we’re developing locally? While I’d eventually like to see us using Boxen or something like it to closely mirror the production setup in our local development environments, we’re a ways off from that yet. Today we use the awesome pow to handle our local development serving, and while it does all kinds of cool things, it won’t serve multiple apps on the same domain.

I wanted to get something going quickly, which meant I needed a simple solution that integrated with our existing setup with a minimum of fuss. Enter rack-proxy. This handy tool lets you create a piece of Rack middleware that will proxy the whole request over to another app of your choosing. Using the excellent Rails guide about Rack and a nice template in a Stack Overflow answer, I quickly had something up and running:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

if ENV["STITCH"] == "1"
  require "rack-proxy"

  class Stitcher < Rack::Proxy
    EXACT = %w(/ /terms /privacy)
    PREFIX = %w(/pricing /about /support /gateways /assets)

    def initialize(app)
      @app = app
    end

    def call(env)
      original_host = env["HTTP_HOST"]
      rewrite_env(env)
      if env["HTTP_HOST"] != original_host
        rewrite_response(perform_request(env))
      else
        @app.call(env)
      end
    end

    def rewrite_env(env)
      request = Rack::Request.new(env)

      return env unless(request.host == ENV["PUBLIC_FULL_DOMAIN"])

      unless(
          EXACT.include?(request.path) ||
          EXACT.include?(request.path + "/") ||
          PREFIX.include?(request.path) ||
          PREFIX.any?{|prefix| request.path.starts_with?(prefix + "/")})
        env["HTTP_HOST"] = ENV["SUBSCRIPTIONS_INTERNAL_DOMAIN"]
      end

      env
    end

    def extract_http_request_headers(env)
      headers = super
      headers["Host"] = ENV["PUBLIC_FULL_DOMAIN"]
      headers
    end
  end

  Rails.application.middleware.insert_before ActionDispatch::Static, Stitcher
end

Lets walk through this real quick; it’s pretty straightforward, but there are a few nuances I want to highlight. First of all, you’ll notice this only runs if a particular ENV key is set. Doing so leverages our extensive use of environment variables for app configuration, and ensures that this will only run in development and never in production. Eventually I’d like to write more about the benefits we’ve seen from switching almost exclusively to configuration via environment variables, but for now you can just note that they’re used in multiple places in this little proxy to pull the right value.

Next, note that I had to overwrite #call - this is necessary to allow passing through unproxied requests to the “hosting” application. It’d be nice if rack-proxy provided a way to do this out of the box, but it’s simple enough to implement regardless. The decision of whether or not to proxy the request is made by looking at whether the HTTP_HOST for the request has changed; if it has, then we want this request to go to the proxied app. Also note the call to #rewrite_response - this is a method provided by rack-proxy to allow adjusting the response before it is passed back to the browser. It’s not strictly necessary for me to call it here since its default behavior is to do nothing, but I used it multiple times during debugging to get a peek at what the proxy was doing, so it’s handy to keep it in the chain.

The magic happens in #rewrite_env, where I check against the set of exact and prefix paths I want rewritten and determine whether the request matches them. If it does, the HTTP_HOST is set to the proxied app’s host, and we’re good to go. The one other check I am doing here is that I’m only ever proxying when on the public/marketing domain, since this app actually serves a couple of subdomains and I only want to rewrite on one of them.

Finally, there’s a bit of trickery going on in #extract_http_request_headers to make everything work smoothly. Even though I’m rewriting the host that the request should go to, I don’t want the Host header that’s passed to the proxied app to be affected, since it should be as clueless about the fact that it’s being proxied as possible. rack-proxy uses this method to prep the headers for submission upstream, and so I just hook it and make my own last minute adjustment.

And then the middleware is inserted at the very beginning of the middleware stack, and BOOM! it all just works. Well, not quite: I spent hours debugging an issue with the headers, but other than that I really did have a solution working in a couple hours. And best of all, we now have a way in our local development environment to simulate the exact domain structure in production.

If you need to stitch another app into a Rack stack, I’d definitely recommend checking out rack-proxy.

The problem was the “mostly” part. The proxy was working, the right things were loading on the right paths, but I couldn’t log into the proxied application. Logging in was actually working and redirecting me into the app, but then I was immediately being redirected back to the login page. It was like I couldn’t write to the session. Maybe a cookie problem?

I do most of my browsing with Chrome, and am generally a fan of the developer tools it ships with. So I started poking, going back and forth between the server logs and the browser’s request log, trying to figure out what was going on. It didn’t take long to confirm that the server was definitely writing out to the session correctly, but that for some reason the browser wasn’t returning the right session information back to the browser. I also confirmed that the site worked fine when not proxied, and it was something proxy related that was breaking things.

And then I spent multiple hours spinning my wheels, trying to figure out what was different between the working direct requests and the broken proxied requests. I tried looking at requests in Chrome for a while, and then I switched to Firefox and tried a few different tools there as well. It was getting late, and I was burning out, and I was making no progress, and I should just quit…

But my conundrum was that while I was pretty sure this was a problem that was specific to my hacked up proxy, I wasn’t 100% sure, and I didn’t want to flip the switch on Monday in production only to find out that nobody could log in. But finally, with no real progress to show for a few hours of debugging, I gave in and gave it a rest.

And sure enough, while I didn’t figure the answer out after quitting, I did realize I’d been approaching my debugging the wrong way. Debugging is an art, and I’d been doing the equivalent of trying to play a violin with a hack saw. What I needed was a tool that would let me textually compare the headers of the working and broken requests.

Enter curl, which I should’ve pulled out right away. I knew at this point that something was up with the cookies, but I had no earthly idea what. So I dug into the curl man page and started reading through the options (there are a lot of them) looking for how I could see what was happening with the cookies. OK, actually I just ran man curl | grep -A10 cookie since I didn’t have time to read the whole tome.

Working with what I learned from the man page, I first got myself set up so I could easily run the same request against the (working) unproxied app and the (broken) proxied app. Then I used curl’s handy -c option to have it write out a cookiefile when I made a request:

$ curl \
  -i \
  -c cookies.working \
  -d "login=bob" -d "password=test" \
  http://workingapp/login

Nothing magic here; -i tells curl to print out the headers along with the response, -c stores the cookies in a curl-formatted file, and the -d switches pass in the login information (and make curl do a POST). I then repeated this against the broken version of the app, storing the cookies in a different file. By looking at the headers and responses from the two requests, I could see that both redirected me into the app, indicating that as far as the server was concerned, I had auth’d successfully.

Next step was to verify that curl would re-present the cookies correctly, and that it could reproduce the login issue. To do that I simply passed the written out cookie file to the app and noted whether I got redirected (indicating a missing session) or a 200 OK:

$ curl \
  -i \
  -b cookies.working \
  http://workingapp/innerpage

The -b switch does the magic here, reading the cookie file written out by the previous command and passing the cookies to the app just like a browser would. And sure enough, the working version gave me back a 200 and the broken version redirected me back to the login page.

Now to dig into the cookie files themselves and see what was different about them! It took a few minutes of staring and poking, but very quickly I noticed that the broken version only had one cookie in its file, while the working version had written 4+ cookies. How was that happening?

Long story short, rack-proxy uses Ruby’s Net::HTTP heavily, and Net::HTTP tries to treat headers like a hash. But headers aren’t a hash; you can pass multiple headers of the same type, and that’s exactly how Set-Cookie works. Net::HTTP papers over this by joining with a comma the values of header keys there’s more than one. And thus I was ending up with one big invalid cookie rather than the four cookies I should’ve had.

Now it was as simple as figuring out how to get Net::HTTP to give me the header values individually and submitting a pull request with a fix. I locked rack-proxy to the fixed revision in our Gemfile, and everything just worked. And while this problem wouldn’t have affected us in production (since nginx is doing all the proxying for us), I went into the launch on Monday morning much more confident since I knew this wouldn’t bite us when we flipped the switch.

Moral of the story? Browsers are the wrong tool for debugging HTTP. You need to be able to see and textually compare what’s traversing the wire, and even though the developer tools provided by modern browsers can give you a peek, they often obfuscate issues rather than making them obvious. curl and other text-based tools like it are the best way to see what’s really happening and quickly track down the problems that arise with HTTP.

One of the key things about how we bill is that a lot of it’s made up of very small metered charges. Because they’re so small, it’s not a huge deal if a metered fee here or there falls on the floor, and so as long as we never over charge a customer we’re more concerned with fixing issues that come up than we are with capturing every last penny. Thus there are various places in the chain of “add a fee” where things could go wrong that we just let them fail and notify ourselves rather than writing fancy recovery code.

But of course, this only works if you notify yourself in such a way that you can fix the problem, and that means a stack trace should show up in the notification. As I coded along I found myself in a situation where something could go wrong and yet it wasn’t due to an exception but rather just due to an “unexpected input” situation. That meant I had no exception to grab the stack trace off of, and for a few minutes I was stumped.

Now, I knew about Kernel#caller, but the problem with it is that it starts with (as indicated by the name) your caller’s stack, and in this case I wanted the stack to start right where I was creating the notification. Turns out there’s a simple solution:

1
2
3
4
5

module Kernel
  def backtrace
    caller
  end
end

I dropped that code in a Rails initializer and then just called backtrace when building my notification, and it works beautifully.

Now, I’m pretty cautious about going in and adding methods to Kernel, but I think this is a good example of why it’s nice that Ruby lets us do so when we want to. If you find yourself needing a backtrace without raising an exception, I hope this helps you out!

Fact is, kids thrive on repetition. Knowing what comes next lets them anchor their constantly changing minds and bodies in something steady and unsurprising. If you give them a drink every night before they get in bed and then forget one night, they’ll be up five minutes later reminding you that you missed your cue. If they take a nap every day in the afternoon and for some reason it’s missed, the whining will commence and you’ll know exactly from whence it springs.

The dawning realization I’ve had over the past few months is that it’s not just children that are well served by thoughtful repetition. Turns out I’m a lot happier and healthier when my life has a cadence as well. By dint of living in a house full of little ones I’ve found myself slotted into a whole bunch of different repeating cycles whether I like it or not, and I’m finding that I do like it, and even thrive on it.

Of course as adults we’re used to having a degree of ritual in our life. We typically have some sort of weekday routine - get up, get pretty, get fed, get to work, get working, get home, get relaxed, get to bed - and the weekends are another cycle that repeats every week without fail. But often these are seen as indicative of drudgery, and something we should always be looking for an escape from. “Leave the work-a-day routine behind!” they say, and we nod our heads in agreement as we begrudge each time the wheel goes around and we find ourselves still on it.

But what if the problem isn’t the repitition, but rather our attitudes about it? What if what’s really needed is a more child-like mind, resting in the stability that a routine provides, and delighting in the things that can be learned and experienced within it? Going back to fundamentals, who gets tired of breathing a few times a minute, or eating a few times a day, and wishes they could escape from the terrible sameness of repeating those activities again and again?

I’m finding that my mistake as I left the routines of my parents’ household was that I let routines happen to me, instead of purposefully building routines that I could embrace and enjoy. I know I’ve certainly gone months feeling dislocated each time a particular cycle repeated itself, largely because I’d allowed it to be imposed upon me without ever thinking through whether it was worthwhile or if it could be tweaked to make it more enjoyable. If you’re going to do something over and over again, it ought to be something you enjoy!

When properly applied, repetitive practices are powerful, and an inescapable part of what it means to be alive. I’ve decided to reclaim the word “ritual” from its spot on the list of “things you’d never intentionally add to your life” and use it to describe these patterns that can be really meaningful and powerful if embraced. And in future posts I want to walk through some of the rituals I and our family as a whole now enjoy, and talk about how we’ve both accidentally stumbled into them and at other times deliberately instituted them.

And hopefully, with a bit of “ritual” applied to my writing, it will become regular enough for those future posts to actually happen!

This morning I did exactly that: I hopped on Facebook, removed all my non-family friends, deleted all my “Likes”, deleted all the Apps I had given access, left all groups, and went through and (laboriously!) turned off every email notification Facebook has. I now exist on Facebook, but really only as a pointer to where I’m actually participating online.

So two questions immediately come up: why? And why not go all the way and delete the account altogether?

The why is pretty easy: I’ve come to despise the social model of Facebook. The simplistic, synchronous “friend” model is good at putting content in front of me that I feel compelled to read, but bad at putting content in front of me that actually enriches my relationships. Add to that the giant sucking sound that is Facebook’s effort to grab as much personal content and then make that content as public as they can get away with, and I end up with a tinge of self-loathing every time I lose a half-hour to reading about the intimate details of the lives of people I don’t even know any more (if I ever did at all).

Facebook feels (to me) like Pleasure Island in the story of Pinocchio - a fun place designed to draw you in and make an ass of you, so that eventually you’ll turn a profit for the owners. I don’t feel that there is much empathy from the organization for their users, rather that the trajectory of the company is to exploit baser human tendencies to “drive higher engagement” and make a quick buck. Even the usage of an advertising business model in a social context just feels crass in a way that (for instance) Google ads never have; sort of like that person you found out invited you into their social circle so that they could sell you Amway products.

This is of course just my personal take: I know many folks get a lot of mileage out of Facebook, and feel that it has merit, and I say, “More power to them!” Which brings me to the second question, well put by Mr. Bass:

Why not go all the way and just delete the account? Well, first of all, there is no such thing as “delete” on Facebook, at least not as far as I can tell. While going through and cutting my Friend list down, there were two people, including one Matthew Bass, who were still in it even though they had “deleted” their account. So it seems kind of like a pointless gesture.

More importantly, my goal here is to stop putting value into Facebook, not to punish myself by disappearing from it completely. People can still find me by my email addresses, see where I’m actually hanging out via the links in my profile, know that I am related to that other Talbott they know, etc. If anything my super minimal ongoing presence on the site makes more of a statement than deleting the account would, by saying, “Yes, I know about Facebook, and I actively choose not to participate in it.”

And, with a still valid Facebook account, I can continue to check out the services that require it, take a peek when Facebook does something new to tick off the internet (I bet your listed email address on the site is @facebook.com, isn’t it?), and generally not get totally out of touch with non-geek consumers, which is an important set of people to understand. And I get all of these ongoing benefits without much more attachment to the site than I would have had I “deleted” my account.

So, going forward, you can find me on App.net and Twitter. And contact me via email. And read my latest thoughts here on the blog. And you can probably even send me a message on Facebook… but if a message gets sent via Facebook and there’s no email notification, will it ever get read? Probably not.

Turns out it is possible, but you have to dig pretty deeply into Capybara’s guts to make it work. If you’re using Cucumber, just drop this into your support directory:

Then use it by calling it within a step definition with whatever fields you want enabled.

It should also adapt easily to non-Cucumber environments; just make it accessible and call it alongside the code that attempts to set the fields.

Hopefully this saves someone else some time; I’ll never get that half-hour back myself!

The problem is, if I don’t like the place that I have to blog, I’ll tend not to write anything, which means I miss out on the thinking process involved with composing longer writing. It’s been a couple of years now since I’ve been happy with my blogging setup, and just recently I’ve been getting the writing itch really bad and its finally culminated in me getting my blogging house in order.

The whole thing is complicated enough that I actually sat down and mapped it out. I had two key goals: first, I wanted to separate out a family blog (birth announcements, Christmas letters, funny kid quotes) from my personal geeky musings. And I wanted to put both of those blogs onto platforms that made sense for them, rather than forcing Katie (and eventually the kids) to use the same tools as myself.

I also had a key requirement: do not break the internets. In other words, preserve permalinks, comments, etc. I’m not willing to abandon old content just because I’m moving to a new platform, and as they say, “We have the technology.” So a migration plan was key.

Here’s my final map:

• blog.talbott.ws -> Page sending folks to either family.talbott.ws or nathaniel.talbott.ws
  • blog.talbott.ws/[permalink] -> Octopress serving archived postings
  • blog.talbott.ws/archives -> Octopress archive page
• family.talbott.ws -> Tumblr
  • Link to blog.talbott.ws archive page
  • Link to talbottkids.tumblr.com archives
  • Link to nathaniel.talbott.ws
• nathaniel.talbott.ws -> Octopress

I used the Jekyll migrator to export the old Mephisto content into an Octopress repo, then deployed it to Heroku. Thankfully Mephisto used a sane url structure, so it was easy enough to preserve the article links. I tweaked the default theme a bit so that it looked similar to the old one, and also put in some handling for legacy (pre-Disqus) comments imported from Mephisto.

Next I set up a Tumblr blog for the family; Tumblr has a nice iPhone app, looks good with short snippets of content, and is just generally maintenance free. Still on the list is figuring out a solid way to back up the content there, though I think that’s doable via their API.

Finally, I’m getting my personal blog rolling. It’s also Octopress, and I’m deploying it to Github. The plan is to shortly replace the default Octopress theme, though that’s not my main priority at the moment.

And now I have a solid place to write - w00t! As they say on the internets: