Jake Scruggs

Clojure ‘def’, ‘declare’, and ‘concat’

2016-01-31T23:02:00.001-06:00

I hit the Aeron hard and asked my pair Liz “what’s up?” I’d been sick and she had been working on a tricky Datomic datalog query in my absence. She claimed to have solved the whole problem but the tests would not pass when run as a whole vs. when run individually.

A classic problem with a big Clojure twist. I, being no stranger to this terribleness, eventually shook the sick out of my head and asked if she had tried running the singular test file by itself. Good news: it failed too. What? I’ll explain. We use a classic trick here at Backstop when running tests in the Clojure REPL: ‘:reload-all’

(require '[clojure.test :refer [run-tests]])
(require 'your-ns.example-test)
(run-tests ‘your-ns.example-test) ; First time

(require 'your-ns.example-test :reload-all) ; Picks up new changes in the files
(run-tests ‘your-ns.example-test)

Otherwise we’d spend all day waiting for the JVM to start up every time we run our tests. It’s a cool trick but it has its limitations. We were clearly dealing with one of those limitations as the tests would fail once in a fresh REPL but succeed on a second run. Those who’ve already guessed the answer please apply for a job here. For all us other mere mortals who need to think it through I’ll present this example of what we were trying to do and what happened:

(declare a-vector)
#'user/a-vector
(def time-bomb (concat [1 2 3] a-vector))
#'user/time-bomb
(def a-vector [4])
#'user/a-vector
time-bomb
IllegalArgumentException Don't know how to create ISeq from:

clojure.lang.Var$Unbound  clojure.lang.RT.seqFrom (RT.java:505)

But if you asked for time-bomb again:

time-bomb
(1 2 3)

Everything is, er, fine?!? Except 'a-vector' has disappeared and a 'function' has returned different things for the same lack of input. So, of course, “concat” is part of the problem here. As Stuart Sierra pointed out in http://stuartsierra.com/2015/04/26/clojure-donts-concat, “concat” is a lazily evaluated join so it can hide many bombs.

If we had instead used “into,” which is not lazily evaluated, all would have blown up in a nice normal manner:

(declare b-vector)
#'user/b-vector
(def now-bomb (into [1 2 3] b-vector))
CompilerException java.lang.IllegalArgumentException: Don't know how to create

 ISeq from: clojure.lang.Var$Unbound,

 compiling:(form-init1552771854893412833.clj:1:15)

Well, not a great error but at least the line number would have given us a clue as to where the problem was instead of the stack trace starting where ever the “time-bomb” was evaluated.

However, “a-vector” is defined by the time “time-bomb” is evaluated. So why doesn’t the lazy “concat” use the available “a-vector”? “Concat” seems to freeze the temporary nature of clojure.lang.Var$Unbound and not evaluate a later defined “a-vector.” Weird. Is this a bug? Or just a known evaluation order thing? I verified this behavior in 1.5,1.6,1.7, and 1.8 so it seems here to stay.

In case you are wondering, without a “declare” this code would blow up right away on the right line number:

(def nope (concat [1 2 3] nothing-at-all))
CompilerException java.lang.RuntimeException:

  Unable to resolve symbol: nothing-at-all in this context,

  compiling:(/tmp/form-init1552771854893412833.clj:1:11)

Why did we use declare instead of just order the code so it compiles? As a team, we’ve adopted a code standard where we try to push private functions down to the bottom of a file and “declare” helps us declare things that will be coming to the compiler/jvm/macro-magic-factory. It works really well with “defn” so we applied similar principles to public “def”s (for use outside the namespace) and “def”s we didn’t think should be part of a namespace’s API.

The combination of testing in the REPL, code conventions, the lazy nature of “concat,” and when “def” gets evaluated created a lot of confusion for us. Hopefully, this blog post helps you figure out some similar problems.

Lutz’s Super Truthful Murder-ball adventure

2015-11-25T15:06:00.000-06:00

For backstory checkout out the previous DnD post.

Landing in a pile on a large stone table in a massive room from out of a swilling green magic portal is very disorienting. However, it did lead to Lutz shaking off his magical blindness. First sight: Many intense dudes some covered in lacquer armor — Lutz didn’t even know lacquer was a thing you could make armor from. While Lutz pondered the truth of his eyesight, Lord Valparaiso threw a little fit. One: He was mighty pissed that his evil portal trap has ensnared him in addition to our heroes. Two: The portal had clearly been hijacked to another mysterious and far away land. The land was so strange and far from their home that Raest couldn’t feel his god Palore anymore which meant no spells. Essentially this changed his role in the party from from healer/crappy fighter to, um, crappy fighter. That single set of footprints on the beach are not from Palore carrying Raest during his darkest hour.

The team had many questions but lacking a common language they were ignored, shoved out of the big room, down a hallway, and into a smaller room with a chalkboard. A very painful session of explanation ensued. Drawings of stick figures and angry gestures soon left both sides exhausted. The time had come, once again, for Lutz to put the team on his back. Working with noted jerky-owner and consumer Bran they managed to extract salt from beef jerky infused water with heat from a torch. That pinch of salt combined with a pinch of soot plus Lutz’s NEW FOUND ARCANE TRICKSTER POWERS meant he could cast a Comprehend Languages spell. Clutch.

Now that they could understand what was being said, via Lutz, the strangers in a strange land understood that they must play a spirited game of foots-ball with big bats and a medicine ball. There would also be giant walking trees that shoot hurt-y needles if you get too close. Hitting other players was allowed and encouraged but the ultimate goal was to get the 2 foot wide medicine ball into the goal by any means necessary. Winner gets not to die. Best 2 out of 3. Classy.

Our ambassador/coach/warden gave the team a (perhaps) last meal. It kinda seemed she might have an interest in seeing team Lutz win. Was betting involved? Who’s to say? Mind your own business. The important part of this last feast was that everyone got a rest and Lutz hid away a small amount of alcohol.

Upon reaching the field for the first match the team saw their competitors. From far away they appeared to be a bunch of imps and a man. As they came closer the dread increased as our brave band of adventurers saw one 9 foot tall center and a bunch of good sized dudes.

At the start of the match more bad news came pretty fast: Clustering around the ball made the team susceptible to a sleep spell from the safely 50 foot back magic user. The other team had “forwards” running down the wings, defenders hanging back, and an actual strategy. Peren, Lutz, Bran, and Raytheon had more of a “swarming bees” strategy that kinda played right into team bad guys’ “cast a sleep spell and send a man down the side lines” strategy.

You might think the gang got out of this one with some clever thinking, but no, they mostly just whacked everything nearby which included the ball. Ray-ray, in particular, took out his frustrations over losing all of his promising new cult by smashing every opponent excessively: Often from behind. Somehow this aggressive flailing got the ball past the team jerk face’s front line and rolling. The big bad, being not very big on smarts, crushed the ball towards the wall which was about 90 degrees off from his target goal. Lutz used his superior speed and agility to do 2 things in the time another would merely perform one. He wake up the sleeping members of his team and flanked the opposition. Bran got all “angry barbarian” and made like Lutz with the 2 to 1 actions. Unfortunately the other team had a double attacker too - lots of hustle displayed on the pitch that day. The first match was won by Lutz picking up the ball, using his crazy moves to break the ankles of the other team, and bursting on through to the goal.

During the break Lutz cast a False Life upon himself with his fly arcane trickster powers using the dram of alcohol he had acquired from the “last supper.” Palore, wherever he may be, did not favor Raest with the heal spell he so desperately needed. Only a jerk would have pointed out that necromancy had succeeded where religion had failed. A bit more rested than before the team braced for the coming match. How would the other team solve the “Lutz problem?”

Well how could you? He’s just too damn fast and agile. Like they used to say about famed foots-ball player Mycroft Jordan-Hammer: You can’t stop a Lutz, you can only hope to contain him. Once again Lutz woke up sleepy members of the team, picked up the ball, and made a break for it. Problem was that this time a terror tree decided to get between Lutz and the goal. Now a normal thief, pick pocket, 2nd story man, or even assassin would perhaps have turned tail at that point. Lutz sucked up his courage, cast “Thunder Wave,” and knocked the terror tree aside. An entire walking tree pushed back 10 feet by a skinny studded leather wearing punk of an 18 year old. Just picture it. Now picture a bunch of jerks chasing Lutz for a few long seconds as it is became apparent to all that it was going to be a 2-0 sweep. Clutch.

Now that they won the right not be slaughtered for a god’s favor, how will these heroes be treated in a land even Palore has forgotten? Tune in next time to find out.

Our Purely Functional Microservice in Production

2015-11-17T17:44:00.000-06:00

Recently I was interviewing a candidate for a job on my team at BackstopSolutions and he said something like: “I read your blog post about Microservices and was wondering if the lack of follow up meant the project was a failure.” He actually expressed this sentiment in a much nicer way as this was an interview but still it got me thinking: "I should do an update on the status of our microservice and explain how it could be purely functional."

The service is implemented in Clojure, uses Datomic for persistence, and integrates via REST.

The service has been in production for several months now and has gone through a number of versions of its API. Implementing HATEAOS worked out well as it allows us decouple having to keep the service and front end in lock step. Since this is a microservice, we could keep both applications in step with each other as we own both ends of the interface. However this co-ordination is not without cost. Release dates slip b/c of bugs in either system. There are less bottlenecks in production releases as the front and back end can be deployed to production independently of each other. Not having to co-ordinate releases is a wondrous decoupling. It didn’t take long to implement HATEAOS and maintenance costs for the discoverability aspect have not materialized.

Datomic is a special snowflake of a database and that it why I believe it is the future. That is a bold statement deserving of a whole other blog post so I will merely list a few of Datomic’s features that we exploit:

It never forgets. Think of it like source control but with transactions replacing commits.
Time is a first class citizen. This is really an outgrowth of the above feature but is worth pointing out on its own.
It uses the incredibly compose-able Datalog query language instead of SQL and can include clojure.core expressions in its queries.
It can easily be used as a graph database.

Those are some crazy advantages. Since Datomic never forgets, ANY query to our service that specifies a time can never have a different return value. That is what I mean when I say that our service is “purely functional.” This enables the Rails app that works with our micro service to cache every query it makes FOREVER. The service without caching is still crazy fast considering the incredibly complex queries is has to run. Like any graph database, speed is determined by what percentage of the graph a node can access. Even if a customer stretches the engine to its maximum, our ability to cache forever means they will take that hit but once. Of course you can query our microservice with no time specified where it will default to now and not cache. This is used only for debugging and is never used by our Rails application.

The graph database aspect of Datomic combined with its understanding of clojure.core enable us to spend days instead of months (or even years) implementing custom business logic. Backstop Solutions is a Business to Business (B2B) Software As A Service (SAAS) firm. In B2B there is but one constant: All business’ processes are unique. Period: End of story.

There may be some similarities and generalities between companies in the same competitive space but these similarities are revealed to be mostly illusionary when you try to implement specific customers. Drill down into any business’ rules and you will find unique requests from every client. Think of it this way: In an efficient market what use is there for complete duplication? The clients who's processes are a competitive advantage hate any SAAS offering that can not implement aspects of their system in the SAAS product. With this microservice we can. The actual challenge is to get clients to simplify their rules enough so that they themselves can understand the rules. This may seem confusing until you consider that Backstop Solutions has clients with many offices in many different lands so we are often the first people to implement a logically consistent system that fits all of their needs, people, and processes.

Datalog is a subset of Prolog. Prolog is a logic based language as is SQL. Where it crushes SQL like a bug is its compose-ability. You can create a bunch of “rules” and mix and match these rules just like you are used to composing functions. This is amazing. I can not stress this enough. I continually implement stories in a fraction of the estimated time because I discover that most of the query is already written and I just have to tack a little on the end, or beginning, or middle.

What is even more wonderful is that even if I ever get into a tight corner with Datalog I can fall back on all the functions in clojure.core. How many times has it infuriated you that the version of SQL you are using can’t support a completely logical function and so you have to do something horrible? Now imagine being able to use all of clojure.core in SQL. Exactly.

I should stop and point out that I had the idea for this service 4 years ago during a conversation with David Chelimsky. As he described a problem he was solving with a graph database I realized that we had a completely different but functionally similar problem where I work. Thanks for always being inspiring David.

Lutz’s completely accurate DnD summary for those who weren’t there

2015-11-09T21:14:00.001-06:00

The “Curious Case of the Tiny Thieves” is being joined in progress and so a little backstory is necessary:

The dexterous and rather handsome rouge Lutz was investigating these “Tiny Thieves” with his band of mostly merry adventurers: Raest a cleric of Palore, Bran the mighty barbarian, Torment the super naive atheist paladin, a wild magic user called Very Forgettable Name, and Iesha or Aida or Ada AKA: the old lady magic user who casts a lot of sleep spells. They discovered the little thieves trying to abscond with their solid gold cat statue which they received as a gift for returning a talking cat to an old lady who was really a dragon in human form — long story.

Turns out the whole town hated these mini scofflaws as they has been on a veritable spree of the crime variety. A reward was offered and pursued by our fairly fearless heroes. As far as they knew the criminals were: 6-8 inches tall, never seen in full light, and immune to sleep spells. Lutz made a super cool trap for these minute miscreants but they avoided it perhaps because the gang had left the lights on. Seems obvious in retrospect… They did find tiny foot prints in the fireplace the next day! They were also found the same footprints the next room over where a very expensive necklace had been stolen. Tiny thieves: 1. Adventurers: 0.

That was all prologue, believe it or not.

Upon investigating the roof and dirt surrounding the “Sleepy Dragon” the team supreme found evidence of rope used to climb down the chimney and footprints leading away from the inn. Following the trail of regular sized footprints the gang pretty much got lost right away. Sigh. The trail was definitely headed south and that’s about all they knew. In that direction lay: The bad part of town, The duke who had suspiciously never been robbed, and a new unknown church that ministered to the poor in said bad party of town. “Why not stop in at the working class church and ask around?” thought pretty much everyone. Raest, who apparently has more money than sense, decided that 5 gold was an appropriate bribe/offering to a dirt poor chump. Not 5 copper, mind you, but 5 gold. As in 500 copper. Bran had been invited in to the small place of worship by Raest in a move that was so very sensible it belied his previous insane tipping/bribing strategy. I feel like I’m focusing on this too much… But seriously 5 gold. FIVE DAMN GOLD. It makes a rouge wretch to think of it.

Iesha noticed the lay priest was totally lying about everything with her wicked awesome old lady perception skilz. However that information was disclosed after Raest and Bran had disappeared into the hovel/church.

Lucky Lutz, the super awesome leader of the group, had wisely decided to keep his ear to the door and therefore heard screams from within. First through the door and down the stairs Lutz sprinted only to be held up by the barbarian’s massive size in the stairwell. Bran had been stopped by…

The Tiny Thieves! They were clearly automatons (explaining why they could not be put to sleep) and surprisingly strong as they managed to subdue the brick shit house that is Bran and even Lutz: the slipperiest rouge in the land. Unfortunately for those wee jerks, Torment has a bewildering talent with his net. He effortlessly swept them off of Lutz from 10 feet away!
But what of Raest and his terrible ability to estimate bribes? He was in another corner of the savage basement of heck getting all perforated by the lay priest of a bad guy god. That’s right: Raest had paid 5 gp for an invitation to an underground murder party.

Raest cast a quick command spell which, as you may know, can be only one word. That word was “retch.” Cult-y McJerk Face then heaved up his lunch while getting pulverized via mace to the face. He died fast in a bewildering flurry of vomit and steel. Despite this early success Raest was lit up like a Christmas tree by a bunch of magic bolts from a pair of magic jerks. Torment barely missed falling into a pit trap and then charged the head cult guy who's name probably starts with a V. Sadly V(icky?) introduced Torment to the Xmas tree experience mortally wounding the paladin. Torment collapsed to the ground unconscious and bleeding out. Bran and Raest kicked ass for awhile but succumbed to the combined forces of the tiny thieves and dickhead magic users. Low on health, they were put into a magical sleep and were soon to be easy pickings. Surely Lutz could have saved the day had he not been magically blinded and therefore unable to help his friend Torment dying just a few feet away. Aida was upstairs drinking water and that other guy was investigating the latrine. Really. They may or may not have had a slap fight while getting down the stairs.

Things looked dire for our rag tag group of misfits until the cult leader made a mistake. Which was good because the only thing stopping his dance of diabolical destruction was the late arriving, but very hydrated, team of Aida and wild magic guy. Did V(oldemort?) over estimate the strength of his magic portal’s vacuum power? Or did he simply slip like the jackass he was and always will be? Maybe we’ll find out next time because everyone but the old bat and a mostly dead Torment got sucked through the portal into another land.

Another land of suck. 5 seconds of being in the new place revealed 2 things:
1) The new bad guys are plentiful and way into lacquer armor.
2) Lutz and company are to be offerings of some sort. Which doesn’t sound good at all.

Cliffhanger!

Microservices vs David Heinemeier Hansson

2015-04-30T08:13:00.000-05:00

Hearing that David Heinemeier Hansson had slammed microservices in his Rails Conf 2015 keynote was a bit disorienting as I was developing a microservice at the time. I leaned over to the other senior developer and said: “What’s his deal?” Toby Tripp, who had just watched the keynote, explained that when David’s team implemented microservices they had terrible trouble managing all the versions between 3 systems. “Isn’t that why we use a lookup call to get the URI-map?” I asked.

“Yep” he said.

“That and never remove a return key/value (pair), only add new ones until you can prove they are not used anymore” He added.

I returned to adding in some caching to our service calls.

Later, I watched the video which is here:

https://www.youtube.com/watch?v=oMlX9i9Icno

David has a meandering style but he mostly discusses microservices between 43:00 and 52:00. The “what’s in your backpack” stuff is all about his earlier zombie apocalypse metaphor that you can watch if you like. He claims his team implemented microservices right, loved them on day 5 but hated ‘em in year 2 (paraphrase). I don’t understand how I will regret my microservice architecture decision if we implement HATEOAS (Hypermedia as the Engine of Application State). You may remember HATEOAS as the thing from REST that all the eggheads complained was missing from Rails’ “RESTful-routes.” “Sure, the routes were REST but what about the discoverable URI-map?” they may have said. They were, of course, right (jerks).

I’m not saying HATEOAS is perfect. I was in the middle of adding LOTS of caching to our client so it almost never has to call our service for one of the two calls you must, at a minimum, make with a HATEOAS service. Why two calls?

To get the URI-map (A map of how to call the service for all resources)
Actually make the call (Using the above map)
What? There’s a 3? Yup, that second call could return another URI-map
It could continue

I know this seems insane, to make 2 or more http calls for every one request, but the benefits outweigh the costs. Old calls will always work. Less bugs. More confident software. Also, you can cache the hell out of all the URI map requests from the client side. When you need to expire that cache you can do it upon release of new functionality because, this being a microservice, you own all sides of the interface. Just work it into your deploy script and you’re done. Extra bonus: No URI ever has a stupid v7 in it.

The last call is a bit of a bummer over http. Even servers right next to each other in the cage have important amounts of latency for many services. HATEOAS isn’t for every service. Although, with virtual machines, I run my service on the same physical box in another vm: Somewhat mitigating network latency. Double Extra Bonus: My service is purely functional so I can cache all of its responses forever. That bold claim, however, is another blog post.

Anyway, Tripp is leaving my team to go be the king of software at some company I refuse to promote out of spite. How would you like to work at Backstop Solutions? I’m looking for a senior Ruby dev with a bunch of Clojure experience but, to be honest, I don’t really expect to find one. So if you’ve got some decent Ruby skills and can learn Clojure while pair programming you should apply. https://www.backstopsolutions.com/careers

Mention you read this article in the interview and you’ll get to see a man in a crazy shirt blush.

Why Group Texts Must Die

2014-09-16T19:46:00.000-05:00

Recently I re-tweeted this thought from Pete Holmes:

top reasons for divorce. 1984: 1. financial problems. 2. religious differences. 2014: 1. constant inclusion in group texts. 2. no wifi.
— Pete Holmes (@peteholmes) September 14, 2014

because I'm in the middle of a communication crisis. I do "knowledge work" so it’s considered inappropriate to have a device constantly making little beeps and boops while the person next to me is working on some insane LibXML (you don’t wanna know) bug. Programming requires extreme concentration and distractions are to be avoided.

The rub of it is that I’m “on call” if our software product bursts into flames. Therefore, I must keep my phone in a mode capable of disturbing me so I don’t eat lunch, play ping-pong, or just code right through the disastrous failure of our software.

This used to be fine back when text messages were either:

Time sensitive
Important
From one person

In first two cases the beeps my phone made were appropriate and in the last case I could simply text back to that person that I was busy if they didn’t get the me ignoring them hint. However, as we all sadly know, the plague of group texts has completely disrupted how we handle notifications. Now when my mother in-law, a nice person who likes to share, sends me and 10 other contacts a pic of a lake I spend the rest of the afternoon getting “Nice!,” “Did you take a swim?,” “No, it’s too cold”, “where are you guys?” etc. texts. Every one of those I should probably check to make sure it isn’t an important work thing — just in case.

Why not a mass email? Why do I get group texts trying to plan something days or weeks away? Shouldn’t that be an email thread? The answer is as obvious as it is sad-making: People have given up on email. For every Inbox Zero zealot, there are a hundred people who’ve essentially let their inbox run wild with clutter. Missed important emails because of that clutter? — better move to text messages For Everything. Of course, this leads to cluttered text messages. Plenty of times I’ve been furiously searching through my email to find someone’s reply only to remember “Oh, they’re one of those text-y people, I better scroll through all the various group texts threads they’ve been involved in… Sigh.”

A small aside: Some people don’t have unlimited text messages on their phone. The horror, right? My boss consistently crashes through her text message limit b/c of “friendly” group messages.

The problem of communication clutter isn’t going away. Ever. Abandoning email for texts just moves the problem. I implore you to take charge of your inbox, people. Getting emails you never read? Unsubscribe or Block! Can’t bring yourself to do that? Well, how about creating a rule that moves your daily…

Hold on, got a text… yup, a silly friendly chatty one from a friend who never emails anymore. I really like this guy but such frivolity is a tweet or an email or a Facebook or a Whatever but not a text that demands my attention.

Where was I… Oh yeah, create rules in your email client of choice that automatically move “less often read” messages to a folder for later (read: never) reading. If you’re still getting a flood then you must keep going. Unsubscribing, blocking, and rules are your new watchwords. If you can’t handle this fire hose of information now what do you think is going happen when everyone can send “animated emojis” on a whim from their Apple Watch (or some other wearable computer thing)?

Your “inbox” is not just your email inbox. It is Every Damn Message you receive in Any form. They all via for your attention and giving up on any one communication form only annoys your friends while providing, at best, temporary relief.

Apology: I’m sorry if you’ve sent me a group text in the past and see this as an attack. I really do want to hear about your cats, see your kid’s pics, and ponder your stray thoughts… In an appropriate medium. Does that sound bitchy?

The Definition of Garbage

2014-06-09T22:53:00.000-05:00

The views and opinions expressed here are my own and don’t necessarily represent positions, strategies, or opinions of Backstop Solutions Group.

Recently we released episode 3 of the Software Apprenticeship Podcast but had to pull it back for re-editing because of some problems with how developers talk to each other. Developers are not kind to ANY code. Even our own. Especially our own. Sitting next to a dev while he or she discusses the code they are working on can be a shocking experience. Words like “Crap,” “Junk”, “Garbage” and many worse are used often. A lot of this type of talk was on episode 3 and when someone at Backstop (who’s job it is to protect us from ourselves and comments taken out of context) heard it they asked us to edit the podcast to take out some of the more offensive comments. This is why episode 3 sometimes fades into music and then comes back mid-conversation. Sorry about that.

I don’t know where I first heard the definition of developer as “Whiny Optimist” but it is uncannily accurate. We developers are forever complaining about previously written code. Code is awful. Code is crap. Code is the worst spaghetti wrapped around horse manure we’ve ever seen.

And yet…

We couldn’t go on if we thought we’d have to live out our lives fighting the very thing we create. There is this optimism about future code. It will be bright and shiny. The next project to re-write the is going to make everything better. So much better… The code will be pristine and new features slide in like rum into coke. Ponies and rainbows are coming.

Also…

Every year I get better at what I do, so even code I thought wonderful 3 years ago can be “crap” to me today. I look back and see a developer who didn’t keep orthogonal concepts separate who coupled code that should not be coupled and I am sad. I regret my past inefficiencies and curse them.

But…

How bad is this code really? Backstop’s code is rigorously tested many times automatically before being poured over by humans. Any code change in my product gets tested first on my machine (by automated tests) then on another “Build Server” (which runs the tests I was supposed to run and a bunch more), then another series of “Regression Servers” will run some even longer regression tests that literally use the app as our customers do. If it passes all that then we’ll have our Quality Assurance people go over it again to make sure the machines haven’t missed anything. The last thing the Q.A. people do is write a new automated regression test to make sure this functionality doesn’t break in the future.

What the heck are we complaining about then? The software works! It helps many people make a lot of money, it makes the company money, and is a leader in the industry. We developers are, in some ways, a bunch of ungrateful jerks.

Let me see if I can explain why. Writing software that solves hard problems is hard. Duh. There are only so many people who can do it and we struggle through. Writing software that solves hard problems and can continue to accept new features easily is the HOLY GRAIL of software development. Rarely has it been done even though every company claims their code is the “best in the industry.” If you were to get your hands on the unedited version of episode 3 you would hear a lot of developers complain how we wish we had written code in the past that could be easily changed today. We might even call such code “garbage” but what does “garbage” really mean? In our app it has come to mean code that works, is well tested, but resists change more than we would like. We are whining about having to do more work. If only our past selves had properly separated the concerns more, if only there was more time for refactoring. But some day we will reach that shining castle on the hill. And there will be ponies and rainbows for all.

Software Apprenticeship Podcast Episode 3 is Out

2014-05-14T22:54:00.000-05:00

This week we start off by throwing Jonathan into the deep end of pool where he pairs with an experienced developer on a 10 year old Java project that is the core of our signature product: Backstop. Of course the company is called Backstop Solutions and so, in order to avoid confusion, we gave the project a different name for internal use: Fund Butter. The mystery of how such a terrible thing came to pass is revealed in this very episode.

There’s no way we couldn’t discuss DHH’s Rails Conf declaration and blog post: “TDD is dead. Long live testing.” This, of course, leads to a discussion of our team’s test philosophy.

You can find it on iTunes (search for Software Apprenticeship Podcast), any podcast app’s search function, Google, this temp page: https://itunes.apple.com/us/podcast/software-apprenticeship-podcast/id868371146 or use the RSS feed directly if your into that sort of thing: http://softwareapprenticeship.libsyn.com/rss

Our panel (composed of Toby Tripp, Matt Pyra, Eric Johnson, Jonathan Howden, and I) meander through quite a few topics. Here’s a breakdown by time of the various topics:

01:27 Backstop’s 2 day bug bash
02:00 The apprentice has to tackle a 10 year old Java program
06:22 “Everything needs to have an ID if you’re going to make it Hibernate-y” - Jake “super technical” Scruggs
08:21 Why we call Backstop “Fund Butter” within the company
10:50 The apprentice encounters Selenium
12:42 Troubles with regression/integration testing through the web browser.
13:43 “Unit Testing is Dead” - DHH
20:00 Pairing all the time vs code review
21:51 Toby talks about the Hill Air Force Base pair programming study mentioned here: http://www.informit.com/articles/article.aspx?p=29410
26:40 The Wall of Awesome - Backstop’s way for employee’s to recognize and thank other employees
47:21 The anti-college movement
49:36 The “Expose your ignorance” apprenticeship pattern with examples/confessions from Jonathan, Jake, and Toby
51:14 The C3 project comes up with near 100% frequency when >= 2 die hard XP'ers are in the same room.

Announcing the Software Apprenticeship Podcast

2014-04-28T20:52:00.000-05:00

In the summer of 2004 I did an apprenticeship of sorts at a place called Object Mentor. At the time “Uncle” Bob Martin and his son Micah Martin were in leadership positions at the company and I somehow convinced them to let me work for free over the summer in exchange for teaching me software development. It wasn’t a very structured program, nothing like what Micah would later put together for 8th Light, but I was a pretty motivated learner. I also had the advantage of coming from a teaching background so I knew how to learn.

All this has been covered in daily detail, if you'd like to read more.

After ten years of software experience I’m becoming a mentor to an apprentice and documenting the experience via podcast. Backstop Solutions has graciously allowed me to pay our apprentice (the same rate we pay interns) as he is doing real work on a daily basis in addition to outside learning experiences. From 9-5 he will be working on production code with 100% supervision as he will always be pairing with an experienced developer. It’s a six month program with 2 month check-ins.

The apprentice, Jonathan Howden, knows that should he fail to meet expectations we may end our relationship at 2, 4, or 6 months. This is a bit scary, but if Backstop is going to be taking a chance on an un-credentialed employee we, as a company, need to be able mitigate the risk of such a person polluting our codebase. It is therefore our responsibility to provide constant feedback to the apprentice so that he will know exactly what is needed to succeed. So far he’s been doing great: He just finished his 3rd week of apprenticeship so we just recorded our third podcast and will be on a weekly schedule. Assuming he survives the six month apprenticeship, Jonathan will be offered a full time job at a damn good starting salary. Interested in such a job right now? Check out https://careers.backstopsolutions.com/

In the first episode, Jonathan talks quite a bit about Dev Bootcamp (DBC). I’ve known, worked with, and read the book of one of the founders so it seemed natural to reach out to Dave Hoover and DBC to help Backstop find its first apprentice. We asked their “fairy job mother” to spread the word that we were looking for apprentices and ten applied. They were all given coding homework challenges which were evaluated code review style with the whole InvestorBridge team allowed to attend. We judged three submissions good enough to warrant an in-person interview. Jonathan made it through this gauntlet and was rewarded with a brand new, much longer, gauntlet. Of learning. Look, there's no way not to make this sound hokey as we're trying to do a good thing here.

On a weekly basis I hope to capture what an apprenticeship is from the inside and perhaps provide some value to proto-developers and prospective mentor companies who may be wondering what this “apprenticeship” business is all about. Changing careers is, like it or not, the future. I did it in 2004 and I hope Jonathan will too in 2014.

Software Apprenticeship Podcast:
iTunes Page: https://itunes.apple.com/us/podcast/software-apprenticeship-podcast/id868371146?mt=2
RSS feed: http://softwareapprenticeship.libsyn.com/rss

SICP Wasn’t Written for You

2014-02-27T21:38:00.001-06:00

The number of software luminaries who sing the praises of “Structure and Interpretation of Computer Programs” (referred to as SICP) is such a long list that you might think only a crazy person would take issue with it. However, to ignore SICP’s problems and continue to blindly recommend it seems just as crazy.

SICP was the textbook for MIT’s introductory programming class and was a bit of a departure from other into to computer science textbooks at the time. Wikipedia sums it up nicely: “Before SICP, the introductory courses were almost always filled with learning the details of some programming language, while SICP focuses on finding general patterns from specific problems and building software tools that embody each pattern.” Which sounds awesome, but does essentially say that abstract principles will be introduced before the nuts and bolts of a language. If you think about that for a minute, you may see where the problems will be.

When I was training to be a teacher I took a bunch of education courses. I got good grades but when I got into the classroom to actually teach I flailed around just trying to keep the class under control and mostly forgot to apply the principles I had learned. The knowledge was in my head, but it floated, disconnected, from anything in particular. When I learned these ideas I had no teaching experience, and so, nowhere to place these abstract principles.

SICP’s first chapter explains the basic form of Scheme (a Lisp), some basic operators (+, -, *, /, etc), defining/calling a function, different ways a compiler might evaluate code, and conditionals over the course of a few short pages. That’s a bit much to swallow all at once, especially the comparative evaluation stuff but that should be easily sorted out with some examples. Right? Well, that’s not really SICP’s thing. SICP will give you a few trivial examples and then toss you right into the deep end. Then first 2 problems for the reader are pretty easy, but it’s the 3rd that will let you know what yer in for: “Define a procedure that takes three numbers as arguments and returns the sum of the squares of the two larger numbers.” Which seems pretty easy until you realize there are no variables. You’ll need to figure out an algorithm that can take 3 numbers and, without any intermediate state storage, return the 2 biggest numbers in such a way that you can sum their squares. I’ll be real honest here, after about 30 min of trying to do this (I have zero functional background so I’m a complete novice here) I gave up and tracked down the answer online. Of course the answer was simple and concise and made me feel like a chump. Which is fine, but not really what I was expecting in the first chapter, let alone the 3rd problem of the entire book.

But that’s what SICP is all about -- challenging problems. The rest of the chapter introduces Newton’s method for square/cube roots and lexical scoping just for fun. Chapter 2 is recursion vs iteration in terms of execution speed, resource usage, and transforming from one to the other. Logarithmic, linear, and exponential growth are dealt with in a few paragraphs and then we’re off to Exponentiation, Greatest Common Divisors, Primality, and implementing Fermat's Little Theorem for probabilistic prime determination. My favorite question from chapter 2 asks the reader to formulate an inductive proof that Fib(n) is the closet integer to ((golden ratio)^n)/5.

Which brings me to another criticism of SICP: It assumes a familiarity with math that most people just don’t have. A first year MIT student would probably be swimming in math classes so the book assumes that knowledge on the readers part. Abstract programming principles can be very difficult to find examples for so I’m sympathetic to the plight of the authors, but when you just go straight at math you’re explaining an abstract thing with another abstract thing.

There’s a certain sort of person who gets excited by complicated abstract but internally consistent logic with no real connection to the concrete. In my experience as a physics teacher, these students do exist but are very rare. Most people need a bit of connection to something tangible in order to have the ideas connect in their brain.

What then is my point about SICP? Simply that its explanations are overly terse and its problems are large steps past what little is explained. In light of those things I have recommendations for those who attempt to work through it.

If you intend to do every problem, realize that this will take a LONG time and involve a bunch of research.
Set a time-box for how long you’re going to spend on a problem before you go look up the answer. If you’ve spent enough time trying to solve a problem you will still value the answer enough to remember it. 30 min is a good number. Increase or decrease as your sanity allows.
If you feel like something hasn’t been explained: You’re probably right. After you find the answer, a close re-reading will reveal a cryptic sentence that you now realize was trying to tell you something. This will infuriate you and is perfectly normal.
Work through the book with a group. This will hopefully allow you to commiserate about how lost you are and get some help. If there’s someone in there that loves this book and thinks everything is explained perfectly, ignore them. If they subtly imply that you’re stupid for not getting it: Leave the group. You don’t need that static in your life.
Do not feel bad about not knowing all this math stuff: Remember that this book was written for students who would be surrounded by math at the time they read it.
Consider learning Lisp before starting this book. The really important concepts in the book come easier if you’re not also learning Lisp at the same time

Why I abandoned MetricFu

2012-08-08T20:16:00.001-05:00

My last commit to MetricFu was March 2... of 2011.

So.
much.
shame.

Since I am throwing in the towel and handing over leadership (perhaps to you), I thought I should explain why I've been such a terrible maintainer and why I'm walking away.

Reasons:

1). MetricFu is a mess. At some point, with all the submissions coming in, the architecture and testing got away from me. Now it's pretty depressing to see crap code quality in an app that is supposed to be about good code quality. I can not overstate how embarrassing and frustrating this is – mostly because it's my fault (I let in things I shouldn't have. Stupid feature blindness).
2). In 2008, when I created MetricFu, I was fascinated by metrics and now I'm not because I don't need the tools to see the problems anymore. Metrics made me better but now I'm leaving them behind: Selfish.
3). I'm not a consultant at the moment so the exploratory nature of MetricFu isn't very useful to me. I know where the bad parts of town are in my app.
4). I think metrics can be a great teaching tool, but you can learn all the wrong things if you're not careful. You can reduce Flog scores without making the app better (Simple: arbitrarily divide up every large method) . Same with code coverage. Even code duplication detection can lead to bad code. DRY-ing things up in a bad way can often be much more harmful then leaving them moist (perhaps you removed some duplication by creating a crazy amount of indirection – a common problem).
5). Ultimately, creating software is a craft. It's not pure science and it's not pure art, it's a combination of both. MetricFu leans a too little hard toward numbers and that implies a precision we just don't have. How to get from bad code to good is a very delicate dance between content, form, and business requirements.
6). Integrating over a dozen meta-code gems is a god damn pain. The gems that make up metric_fu use code to analyze code (intense meta Ruby) and so they sometimes conflict. Also since MetricFu mostly uses regexes to parse the (command line or html) output, every change in one of MetricFu's underlying gems means that MetricFu needs a version bump.
7). The last year has been, personally, very busy and difficult for me.

For these reasons I'm going to be stepping aside as metric_fu maintainer – I'll still be around to answer questions and such but I probably won't be contributing much code. If I've somehow disappointed or frustrated you because of last year's neglect then I apologize. Seriously. I'm sorry.

Want to lead MetricFu? Make your case in the metric_fu google group:
https://groups.google.com/forum/?fromgroups#!forum/metric_fu

Oracle Foreign Key without Index Test

2010-12-07T09:50:00.004-06:00

We've been having some Oracle deadlock issues that have been hard to reproduce locally. After a lot of investigation and solving of important problems that happened not to be THE problem we figured out that while we've been pretty good creating integrity constraints in the database we have not been very good about making sure that every foreign key has a corresponding index. And that can lead to problems.

So we had a situation were our documents table had a foreign key on the accounts table that was not indexed. So updating an account row lead to a whole table lock on documents (instead of just a row lock which would have happened if there was an index) and that was very bad when we had two separate processes where one was doing a bunch of accounts stuff and the other was doing a lot of documents stuff. Deadlocks for everyone!

The sad thing is that if we had just drank the Rails cool-aid about having no integrity constraints in the db we would have been fine but we got into trouble by only implementing constraints and not the indexes they work much better with.

So we found this cool bit of SQL that helped us find all the foreign keys missing an index here: http://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:4530093713805#26568859366976

And that help us solve the deadlock problem. But what about the future? If only there was a way to run this check periodically and automatically... Time for a test!

As an additional bonus, the failing output of the test tells you how to write the index you need. Big thanks to Dave Bortz for tracking down this problem -- I just swooped in and wrote the test. And took credit in this here blog.

Ruby Kaigi 2010 Day 3

2010-08-28T23:09:00.010-05:00

Last day of Ruby Kaigi! Sad to see it go, it's been a great conference. As per usual Tweets are in bulleted italics and the rest is after the fact commentary.

First I've got to show you the commemorative fans they were handing out:

It's Matz and... Someone else (sorry if it's obvious - I don't know). And of course they are programing in the bath.

BigDecimal: You can handle numbers as large as can fit into memory as opposed to the IEEE double #rubykaigi
BigMath is the Math module for BigDecimal #rubykaigi
Lots of different rounding modes in BigDecimal #rubykaigi
BigDecimal.mode is global per process - not thread safe #rubykaigi
So BigDecimal is Fiber unsafe #rubykaigi

Bummer.

.@mrkn implemented a solution storing mode in thread local #rubykaigi

Oh good. Wait, is that in 1.9.2 or trunk. I'd have to be in 1.9.2 right? To the Google! ... Yep, seems like it made it in: http://redmine.ruby-lang.org/issues/show/3285

You can now change modes in a block (added to trunk yesterday) #rubykaigi
Effective digits/ Significant digits determines which digits to keep and which to round off #rubykaigi
BigDecimals don't know their own effective digits #rubykaigi

But I think the point of talking about sig figs was that it's coming soonish in Ruby. 1.9.3?

It is dangerous to mix floats and BigDecimals #rubykaigi
BigDecimals can't convert from rational, integer, or float. just strings #rubykaigi
And you can't do BigDecimal(BigDecimal) !! #rubykaigi

That is one crappy interface. Taking in only strings is frustrating and weird. How did this happen?

Now a preview of the future of BigDecimal #rubykaigi
No library for irrational numbers in Ruby #rubykaigi
Perhaps we can represent irrational numbers as algorithms and on convert when needed. #rubykaigi
BigDecimals don't have real significant digits implemented right now. #rubykaigi
The issue of BigDecimal not being able to handle anything other than a string as input will be fixed as it is a bug #rubykaigi

The irrational numbers thing sounds pretty cool. Since I mostly sling around strings for my day job I don't know that I'll ever use it. But purely for Ruby pride I like to see ruby challenge Python's rising dominance in the sciences and maths.

I really don't like the lack of travel time between session. You should have at least 5 minutes to change sessions. #rubykaigi

Look, this has been a great conference and any slights I've tended to overlook because they've really done a heroic job of keeping the admission price down but that lack of a passing period is just... Well I don't see why they did it. Just having the time as a buffer in case talks run over is reason alone. Moreover, no passing period traps people in sessions that they'd rather not be in and restricts choice.

Apparently there is a lot of chatter in IRC by ruby commiters that NArray should be added to strd lib. #rubykaigi
NArray is 28x faster and uses 8x less characters in 1.9.2 #rubykaigi
NArray is faster than what? Array, I assume, but I may not be following the translation correctly. #rubykaigi

Translation is volunteer and best-effort so it adds some challenge to attending sessions in a foreign language. So if I make any mistakes in this here blog, that's what I'm blaming it on.

Pwrake is parallel distributed Rake - being developed here in Tsukuba. #rubykaigi

Just a little tidbit dropped at the end of the NArray presentation -- sounds pretty cool. Masahiro Tanaka is using it drive his workflow in his scientific research.

yarv2llvm tries to speed up Ruby by implementing type inference (yarv is the vm for 1.9x) #rubykaigi
There are times in Ruby when type can not be inferred and in those cases yarv2llvm is often slower than Ruby 1.9x #rubykaigi
Fixnum overflow to BigNum is one of the hardest problems to solve #rubykaigi
Btw, I think the title of this talk: "How Did Yarv2llvm Fail" is wonderful. Way to keep everything above board. #rubykaigi

Great talk and very honest. The type inference thing looks promising but there are a couple of tough hurdles to clear. Maybe some day. Until then there's always Mirah.

Ruby AOT complier is "mostly compatible" with Ruby 1.9 #rubykaigi
AOT == "Ahead of time"
Ruby AOT compiler is passing 7847 of 7850 Ruby Spec tests as of now. #rubykaigi
On average Ruby AOT compiler is 3.5x faster than 1.9 for common benchmarks #rubykaigi

Cool, let's all move to Ruby AOT!

It doesn't fair quite so well in the real world Ex: Rails. Slightly slower than 1.9 #rubykaigi
Something about cache misses makes it slower #rubykaigi
The Ruby AOT compiler team wants to reduce the generated code size to increase speed. #rubykaigi
Rails can be compiled in 77 min and size of compiled code is 92MB with Ruby AOT #rubykaigi

Oh. Never mind. Plus, compiling. Boo. Hiss.

This memory profiler's gui looks amazing! #rubykaigi
Seriously, this is the sexiest profiler ever #rubykaigi
Can attach to running Ruby programs from another machine #rubykaigi
Designed to have as small as possible impact and be easy to use #rubykaigi

I gotta tell ya the audience was chomping at the bit to get a hold of this profiler. You can run it in production for christ's sake! And the GUI is to die for.

Not available yet!?! #rubykaigi
Needs to get patch accepted into ruby core - maybe in 1.9.3. RATS! #rubykaigi

Very cruel of Tetsu Soh not to mention this up front. I'm sure it was an oversight but everyone was crazy disappointed. Still this was one of the best talks of the conf. Tetsu Soh is one to watch.

Dear Ruby Core please, please, please, please, please, please, please, please accept Tetsu Soh memory profiler patch #rubykaigi

Indeed. I'm famous (on the internet (in one corner)) so you should listen to me.

Lots of requests to publish Tetsu Soh's memory profiler on Github. I agree! #rubykaigi
Automatic sliding doors in Japan trigger much latter than Americans would expect leading to a lot of stopping and hand waving

I'm not the only one who noticed this so I'm not crazy.

More awesome techno in the #rubykaigi main hall. I want, nay, need the playlist.

Please post it on the 'Goodies' section of the Ruby Kaigi site. Or tweet about it. Something.

Shay Friedman has spent the first five minutes of his talk apologizing for being associated with Microsoft. Stop. #rubykaigi

There was this weird hostile vibe coming from Shay. I felt like he hated us for liking Apple products. The weird thing is that there are a lot fewer Apple computers here than at a normal Ruby Conf. Maybe he thought we were all Microsoft haters and so he might as well fire first. Seemed like a mistake.

Just saw a nice hello world creating windows and dialog boxes in IronRuby #rubykaigi
IronRuby currently passes 85% of Ruby Spec. #rubykaigi
IronRuby 1.0 is 1.8.6 compatible. 1.1 (coming soon) will be 1.9.2 compatible. #rubykaigi
Windows Presentation Foundation's view templates are written in xaml which is like html. #rubykaigi

Another markup language because we needed more. I feel like the IronRuby team really needs a win. They've been the slowest progressing Ruby VM for a long time now.

<script language="ruby"> Whaa? Gestalt hides a hidden bit of Silverlight in the page so you can replace Javascript with Ruby. #rubykaigi

Pretty neat trick that. Of course it means making your site dependent on silverlight. No worse than depending on Flash I guess.

IronRuby.Rack is Rack implementation on ASP. Currently in beta. #rubykaigi

It's all coming along, I'd just like to see something working at this point in its life.

If key is symbol then only one instance - less object creation. Which is why everyone uses symbols. Or because everyone else does #rubykaigi

So did you know why everyone uses symbols? Or did you just do it because that's the convention. Yeah, me too but then I learned. I'm not going to say how long ago that was.

Presenter's computer went down and it's taking forever to reboot. Stupid Mac. #rubykaigi

That's one of those rough moments where you really feel bad for the presentor. But he handled it like a champ -- continuing the presentation while he waited for his machine to boot back up. Well done. I wish I got more out of the talk but it was one of those inspirational talks that are hard to translate. The Japanese speaking audience seemed to love it.

Please use the overburdened network to download the english version of the slides. Umm... What? #rubykaigi

Huh. Why did that seem like a good idea. Since when does asking a room full of people to download something all at once ever work. And yet, it seems it happens once a conference. Repeat after me: Never ever depend on the network at a conference.

Funny story time. All through the conference I spent a lot of time staring at the IRC screens on either side of the stage where the translations happen. But since it's just IRC, anyone can join the room. There was this one guy who posted A LOT and everyone of the posts seemed to be either:

888888888888

wwwwwwwwww

I later found out that 888888 means clapping and wwwww means laughter. OK but stop cluttering up the screen that I'm trying to read translations off of. Then I found out who sora_h was: He's 14 and his name is Shota Fukumori. He got up and gave an entertaining lightning talk. Turns out he's a Ruby commiter so he's got that on me.

And, all of a sudden, it was time for the final keynote by Chad Fowler. He gave a talk about how to live a remarkable life and it was good stuff.

The two things that are necessary to live a remarkable life are: To have intention and a system of realizing that intention #rubykaigi
Living intentionally means being mindful of your actions. It's easy to coast through life. #rubykaigi
"My co-workers laughed at me when I learned Ruby" - Chad Fowler #rubykaigi
"You don't want to be in a situation where you are competing on price" - Chad Fowler #rubykaigi
Programming Cobol is like working in a hospice. There's a market for that. - Chad Fowler #rubykaigi
"Always be the worst musician in whatever band you're in" Pat Matheney #rubykaigi
"If you're playing things that sound good then you're not practicing" lesson from music that applies to programming #rubykaigi
"I needed a project that was useless" - Chad Fowler #rubykaigi
"Was today better than yesterday?" All you have to do is get a little better every day #rubykaigi
"Passion is a resource that you have to conserve" - Chad Fowler #rubykaigi

Lots of gems in there.

RubyKaigi 2011 will be in July and in Tokyo. #rubykaigi

There was something about the theme being "The Last Ruby Kaigi" but I think it was sort of a joke making fun of some guy who posted a rant about how the Ruby Kaigis need to end. I didn't have enough background to get it.

Oh, I forgot to talk about the guy hawking "The Last Google Wave Book Ever Published." It seems he had been working on a Google Wave book and it was printed on the same day Google announced the end of wave. His response? To shamelessly promote the book at Ruby Kaigi in a lightning talk, at some sort of hallway session (with 30 people gathered around), and pretty much everywhere else. And I actually saw some people carrying the book around. Did they buy it? Who knows -- his positive personality was so powerful I wouldn't be surprised if they did.

Ruby Kaigi was an excellent time. I thought it might be crazy intimidating but everyone was super nice and there were enough english speakers so that I could always get my ideas across. Go if you have the chance.

Also, check out my every growing set of Ruby Kaigi photos at Flickr:

http://www.flickr.com/photos/jakescruggs/sets/72157624815648014/

Ruby Kaigi 2010 Day 2

2010-08-28T07:27:00.008-05:00

Holy crap am I tired. It's been a long awesome day. It started out with some excitement:

Just found out I have an hour time slot when all I prepared was 30 minutes. Ok, time to write some more. #rubykaigi
Panic! Maybe I'll talk about metric_fu a bit. #rubykaigi

I must have looked at that schedule 20 times and never realized that I had an hour slot. Everyone else had 30 minutes so I thought I did too.

Attendees of "The Importance and Implementation of Speedy Tests" will also get "Metrics Based Refactoring" at no additional cost! #rubykaigi

I did something I almost never do: Look at crisis as an opportunity (crisi-tunity). I had to write "Metrics Based Refactoring" anyway for Windy City Rails so why not write it now. In less than 4 hours. While watching my friends present at a conference. Looking over those sentences now I can't believe I didn't freak out.

Ted Han used publicly available data to settle reviewing bais accusations against "The Edge" of hating the PS3 #rubykaigi
Turns out there was no bais. Math to the rescue! #rubykaigi
bais? bias? baise? no idea.

You are probably not surprised to find out that I can't spell. Even Eito Katagiri, who did a wonderful job translating my slides, found a bunch of spelling errors and English is not his first language.

I just pulled out a table in the main hall and the table police where all over me. My mistake. Sorry. #rubykaigi

There were announcements everywhere about not doing this and yet I did. In my defense: I'm an idiot.

They're giving out an award to the person who committed most to Ruby 1.9.2: Yusuke Endoh #rubykaigi

A very nice gesture. Lots of class here at Ruby Kaigi.

Time for Matz's keynote!

Matz is talking about Ruby 2.0... again. #rubykaigi

This topic is a staple of Matz's speaking career. I think the first time I saw him talk, years ago, he was talking about Ruby 2.0. Someday...

"Right now ruby is just good enough" - Matz #rubykaigi
Matz hates local variable propagation (the lack thereof) but no one else seems to care so he's abandoning it. #rubykaigi
Ruby's private is not private: it can be accessed from subclasses and overridden by accident #rubykaigi
Monkey patching modifies the class globally. #rubykaigi
Classbox is the solution to global monkey patching #rubykaigi

Lots of talk about the mysterious 'classbox.' What is it? Well here's a paper on the subject:

http://www.iam.unibe.ch/~scg/Archive/Papers/Berg05aclassboxes.pdf

And some pertinent lines from the abstract:

...Unfortunately existing approaches suffer from various limitations. Either class extensions have a global impact, with possibly negative effects for unexpected clients, or they have a purely local impact, with neg- ative results for collaborating clients. Furthermore, conflicting class extensions are either disallowed, or resolved by linearization, with consequent negative effects. To solve these problems we present classboxes, a module system for object-oriented lan- guages that provides for method addition and replacement. Moreover, the changes made by a classbox are only visible to that classbox (or classboxes that import it), a feature we call local rebinding. To validate the model we have implemented it in the Squeak Smalltalk environment, and performed benchmarks.

Now for more Ruby 2.0 preview:

5/2 => 2 should be 2.5 or 5/2 (rational) #rubykaigi
Inheritance in ruby is more for connivence than for other merits #rubykaigi
Matz is thinking about moving mix-ins to a traits like solution which would have conflict detection (unintentional overrides) #rubykaigi
You could declare the 'mix' and specify how to deal with conflicting methods. #rubykaigi
mix Foo, [:*] would copy all constants from the mix, or you can specify which ones you want and rename them #rubykaigi
mix raises error on method/constant name conflict or removing #rubykaigi
From Matz's slides "Ruby 2.0, just started, small step from 1.9, should be done soon" #rubykaigi
Ruby 2.0: Traits, Classbox, Keyword arguments, a few other nifty features #rubykaigi
There some fierce discussion going on in IRC about the proposed changes in Ruby 2.0 #rubykaigi

The debate on 'mixes' got pretty hot and heavy. Keep in mind that all this IRC chatter is displayed behind Matz while he was giving his keynote because the translations are done in IRC. So you'd be reading a translation of what he said right along with people discussing it.

My presentation on "Speedy Tests" and "Metrics Based Refactoring" starts at 13:30 (10 min) in room 200 #rubykaigi
The 'slides' from my presentation on "Speedy Tests" http://is.gd/eHNTN #rubykaigi

I think the talk went very well. Especially considering that I wrote the second half fairly fast.

I just disparaged integration tests in favor of unit tests. Next up is @p_elliott talking about how he only does integration #rubykaigi
.@p_elliott does a lot of things to make his integration test fly. I would like to see one of their suites. I could learn a lot. #rubykaigi

So I asked Paul how long their suites take and he said that they develop on 8 core machines and use Specjour to utilize 4 more cores so they tend to run between 5-8 minutes. Pretty damn fast for full stack integration testing.

Hey, someone else made the 30 minutes vs. 60 minutes mistake. I feel better now. #rubykaigi

Except he found out at the end of his presentation when he asked: "How much time do I have left?" and go the response: "25 minutes." Yipes. Luckily there were a lot of questions.

.@nusco did a really good job explaining the basics of Ruby metaprograming
RT @sudhindraRao Whatever works in #java does not work in #ruby. Even huge classes are maintainable. @rubykaigi
.@nusco's favorite metaprogramming trick is method_missing. I thought he was kidding but he was not. #rubykaigi

When he said method_missing I grabbed the microphone back and responded: "Really?" I couldn't help myself. I tend to avoid method_missing. There's usually a way to do what you want with other programming tricks.

Modules are extremely decoupled and can be tested in isolation so they are very flexible @nusco #rubykaigi

Good point.

Lightning Talks! I'm excited! #rubykaigi

Talking very fast does not lend itself to translation. But still they were very cool. Even better there was a lady in a kimono who would 'gong' you if you ran out of time.

That is awesome.

After the lightning talks a few of us went back to the hotel to drop our things of before the party and we ran smack into this huge festival that happens once a year in Tsukuba. Here are some pics:

The party was super nice. And they had a fantastic spread of wonderful foods and drinks.

Ruby Kaigi 2010 Day 1

2010-08-27T18:19:00.004-05:00

Here I am in Japan at RubyKaigi 2010. Wow. Generally I tweet a lot about the conf live and then publish those tweets here (in italics) and provide slightly more commentary. So lets get it on.

So my flight leaves at noon for #rubykaigi, takes 13 hours, and arrives at 3pm tomorrow... Wait -- that can't be right. #looksitupagain
Before today's trip to Japan the furthest from the U.S.A. I've ever been is: Canada. #howisthatpossible
In cab. It has begun.
Someone once said "If you've never missed a flight you're spending too much time in airports" #atgate2hoursearly

Yep - can you feel the panic in those first couple of tweets? I was totally freaked out. The actual trip turned out to be easy. I met up with Chad Fowler, Yehuda, and Woody at the airport and we took a bus straight to Tsukuba. I went to an exotic foreign land and the first thing I did was take a bus through 100 minutes of strip malls. But I was in Japan. And Yehuda held forth on 'snowmen' and encodings for most of the way so that was interesting.

So it's 7:15 am in Chicago, meaning I've been up for over 24 hours. So that's like 36 old man hours.
My first meal in Japan was near 60 bucks. And I'm not even in Tokyo yet. I'm gonna need a bigger wallet.
Also, I almost got run over by not one, but two bicyclists using cell phones to text.
And 7-11's are everywhere.

So I survived the first day and even had an excellent dinner. It was a bit pricey but worth it. Many thanks to Makoto Inoue for helping organize this get together - it was exactly what I needed.

Donuts are cute in Japan: http://is.gd/eFXCR

Ate second breakfast/lunch at "Mr. Donut" which is really nice here in Japan. Of course you can get noodles.

Good news: #rubykaigi has a non-freezing temperature. I was worried it would be 91 outside and 50 inside. Which tends to make me sick
Some nice low-key techno playing in the #rubykaigi main convention hall.
RT @headius Ruby 1.8.8 will release this year and be the last 1.8 release. Some debate still about whether to backport 1.9.2 features. #rubykaigi

Ah, Charles got to got to the secret Ruby Core meeting so he's in the know. Actually I could have probably gone too but I wasn't sure if just anyone was allowed.

Btw, the advice I received to stay up as long as possible after my flight to Japan was good. Feeling no jet-lag.
I adjusted my Japanese rent-a-phone to display am/pm and now it reads "0:32pm" I guess they really like military time here

It's the little things that are the most endearing.

Getting an introduction to #rubykaigi in japanese with translations coming in IRC which is displayed on side screens. #lag

Next up at #rubykaigi is a panel on Rails 3/ Ruby 1.9.2 (replacing canceled keynote)

Ah, the translations are back. Mostly. #rubykaigi
RT @headius I can't decide if it will be more or less exhausting to attend three days of conference sessions I can't understand :) #RubyKaigi
.@wycats is fearless - he's critiquing Ruby 1.9.2 while sitting 5 feet from matz on stage

So Leonard was doing the translation from Japanese to English and Matz was doing the translation from English to Japanese which lead the a moment where Matz had to translate Yehuda's (nice) criticism of Ruby 1.9.

One of things @tenderlove really likes about 1.9 is using encodings is painless. You have to think about it but it's easy. #rubykaigi
Secret to getting commit rights on Ruby or Rails? Submit patches with tests over a consistent period. #rubykaigi
.@tenderlove doesn't think that ActiveRecord got the same amount of love that ActiveSupport did in Rails 3. #rubykaigi
Specifically @tenderlove doesn't like ActiveRelation's integration in ActiveRecord in Rails 3. "It needs help" #rubykaigi
.@wycats' response: "There's always 3.1" #rubykaigi
.@tenderlove feels less able to bounce around the whole project when developing on Ruby as compared to Rails. #rubykaigi
.@wycats brought up something for Ruby core, saw a lot of discussion referencing his name but he could not participate. #rubykaigi

Ah the perils of trying to develop across (real) languages.

Wait, Sarah's talk is going to be in Japanese? #rubykaigi
Oh, just the first part -- well done. #rubykaigi

She learned a lot of Japanese just for this presentation. Good for her.

Rails wrapping of Javascript is kind of a disaster - mostly because javascript is changing fast. #rubykaigi

Good point - I hadn't really thought about why wrapping SQL works so well while wrapping Javascript works out so poorly.

Step one to writing testable Javascript: Get it out of the view. #rubykaigi
Pivotal uses Jasmine to test their Javascript: Bdd/RSpec like syntax and no dom is required. Can run in browser or headless. #rubykaigi
There's no time between sessions to escape one and go to another. Good thing I don't mind appearing rude. #rubykaigi

Picture a lot of me saying "Excuse me" to people who don't understand english.

OH "Social games are just CMS with bad UI's" #rubykaigi

That's really funny.

MySql 5 only supports 3 bytes for UTF8. huh. #rubykaigi
A 'u' with an umlaut is two code points that represent one character. #rubykaigi
UTF8 and UTF16 are both encodings of unicode. #rubykaigi
In Ruby 1.8 and C a string is just an array of bytes. #rubykaigi
"Corruption is normal" - @wycats #rubykaigi
force_encoding is not the way. If you have to use it you probably have a deeper problem. #rubykaigi
Dear internet: Take all sweeping statements with a grain of salt.

Two sessions in a row on encodings. We all feel like we need to know more about encodings. And then we ignore that feeling until it bites us in the ass.

jugyo has a lot of Growl-themed ideas. #rubykaigi
Had an outbreak of super-sleepiness. Purchased a strange energy drink from an even stranger vending machine and I'm good. #rubykaigi
TermColor can do blink! Now that's progress. #rubykaigi
Cinatra is Sinatra for command line apps. #rubykaigi
"write code like writing blog entries" - jugyo #rubykaigi

Watching Jugyo talk is always entertaining - I loved his lightning talk at last years Ruby Conf.

.@tenderlove has changed into a crappy suit -- It's business time! #rubykaigi
And headgear? #rubykaigi

That, my friends, is awesome. I can't compete in shear crazy and acknowledge my superior.

The number of languages (code) vs. the number of languages (speak) is completely off. #rubykaigi
"People are interpreters with forgiving parsers." - @tenderlove #rubykaigi
.@tenderlove enjoys programming the most at hack nights. Challenging and fun. #rubykaigi
PHP and Ruby living together: Webrick serving up WordPress. You are one weird dude @tenderlove #rubykaigi

Webrick serving up PHP WordPress. Think about that for a moment.

Making some last minute changes to my #rubykaigi presentation: "Speedy Tests" Come see it tomorrow at 13:30 in room 200

Hey, I just found out I have an hour time slot when I had thought I was going to present for 30 minutes. I guess the crowd is going to get some bonus metric_fu coverage. As I'm going to present in 4 hours I better go write some more content. Panic!

Oh yeah, go check out my Ruby Kaigi photo set on Flickr: http://www.flickr.com/photos/jakescruggs/sets/72157624815648014/

Using Git Inside a Git Hook

2010-07-20T15:15:00.002-05:00

Using Git Inside a Git Hook can cause problems. In my previous post: "Signal 13 Problems with Git Hooks" I describe how we are trying to automatically merge certain types of branches into a branch that is designed to hold them all. Anyway, that means we want to run some git commands inside of the git hook. We change dirs into another directory where we have a clone of the repo and start telling git to merge some stuff and we get a bunch of


remote: fatal: Not a git repository: '.'

But if we run the exact same commands as the git user everything works fine. Huh. Eventually we got our linux guru over and he noticed that the environment under which the git user runs is totally different when inside a hook. Gitolite does a bunch of things to the env, but the one that was screwing us up was the setting of the GIT_DIR. After we figured that out, the solution was as easy as:


ENV.delete 'GIT_DIR'

in our ruby script that is triggered by the 'post-receive' hook. And now I must get back to the fun.

Signal 13 Problems with Git Hooks

2010-07-20T12:16:00.005-05:00

Ran into a gotcha in Git today when trying to write a post push hook. We want our designer to have a fast turn around time with clients so we're writing some hooks to merge all of the 'theme' branches he works with to get merged into a special preview branch which is then deployed to the preview site. And all this should happen after he does a 'git push.' Seems like a 'post-receive' hook is just what we want.

Except that every time we tried to create one we got these errors on a push:


error: git-shell died of signal 13
fatal: The remote end hung up unexpectedly
error: error in sideband demultiplexer

If the 'post-receive' file even existed in git_dir/hooks/ on the git repo box, we got this error. We checked permissions, gitolite docs, git docs, google, etc and no help. We finally realized that Git was piping in some information to our 'post-receive' file and since we were not consuming it, that was causing the explosion.

So, I present to you, a stub of 'post-receive' file written in Ruby:


#!/usr/bin/env ruby
STDIN.readlines.each do |line|
rev_old, rev_new, ref = line.split(" ")
# You will get in here as many times as branches were pushed
end

rev_old is the old commit hash, rev_new is the new commit hash, and ref will be something like: "refs/heads/test_branch" Useful information.

Git often passes things into it's hooks, check the git book docs to find out what.

The Road to Ruby Midwest

2010-07-15T22:47:00.004-05:00

Crazy day:
It was the day before Ruby Midwest so I gave my 'Speedy Tests' talk to my workmates at Backstop Solutions during lunch today and they had the nerve to find some things wrong with it -- so I'll be doing some re-tooling before Saturday. Then I was all set to leave 2 hours early for the airport when my manager wants to talk about me helping QA understand our process, new features, and maybe get some automated tests running. Hell to the yeah. So I'm all over this opportunity but our conversation makes me leave the office a bit late to catch my flight. No problem, I build in a lot of buffer. Then the Blue line (Chicago's subway) was down. Uh oh. Some surly dude said there was a free shuttle to somewhere else but I couldn't find it so I took a cab. Crazy traffic puts me at the airport with 30 min until my flight. Panic! But the security lines are longish so I panic while inching forward in a zig-zag line. Then I sprint to terminal C (which is the farthest away and involves going underground) and find my seat minutes before they close the doors.

The lady next to me wanted to make small talk about how I was out of breath. I did not.

Then when I landed I noticed that all the signs say "KCI" when I'm pretty sure I booked a flight to "MCI" so I had a minor heart attack moment where I thought I flew into the wrong city. But no, it turns out that KCI and MCI are the same. That's not confusing at all.

But all was well when I made it too the speakers dinner. Very nice of the Ruby Midwest guys to organize such a dinner and pick up the tab. Good conversations with some locals (Shashank, Luke, and Josh), a 37 signals-er (John), and an Edge Case-r (Adam). Looks like the conference is off to a swimming start.

Rails Conf 2010 Day 3

2010-06-13T13:30:00.002-05:00

A few days late on this because I've been sick all weekend. There just has to be a way to do climate control in conference centers in such a way as to not destroy the planet and, more importantly, Jake's health. I seriously wore black jeans and 2 shirts on hot summer days and yet I was shivering and caught a cold.

But let's get to last day of Rails Conf 2010 and my exciting adventures therein. (tweets are in italics)

Woke to "Relax" by Frankie Goes to Hollywood. Which is good 'cause my heart was pounding. Settled it with some bacon. #railsconf
Yep, I could feel cold coming but I was in denial. I ran into Neal Ford at breakfast and we chit chatted about this and that and then he mentioned that he had read my blog post from yesterday. First thought: I published that sucker at 1am this morning and he's read it already? Second thought: Oh shit, what did I say about his keynote -- I think I called him a jerk at some point. But my brain was foggy and so I wasn't quite sure what I had said. Later I looked and, yes, I had called him a jerk. I was kidding! If only that was the most embarrassing thing I did that day...

Pro tip: Have the hotel watch your bag for the day while your at the last day of the con #railsconf
It's true. This year Rails Conf was nice enough to have a section for people's bags at the conference but they don't always.

Twenty-Five Zeros - Robert Martin

So, as many of you know, I started out at Object Mentor (as an unpaid apprentice/intern) and I'm a bit of an Uncle Bob fanboy. As these next tweets can attest:

Bring it Uncle Bob! #railsconf
Good to see Uncle Bob still starts all his talks with some hard core physics. #railsconf
HOLY SHIT!!! Uncle Bob opens by showcasing some impressive drum chops. #railsconf
RT @duncan: Photo of @unclebobmartin on drums at #RailsConf (ISO 12800 in the dark) http://www.flickr.com/photos/oreillyconf/4688547550/
@spiceee call him (@unclebobmartin) butter cause he's on a roll #railsconf
#railsconf haiku Uncle Bob holds us | We are enraptured by him | Rambling? I think not!

Great speaker, although I had to de-follow him on twitter as I'm not into right wing politics.

If we have computers 10^25 times more powerful than 1960 then why no AI? Always seems 10 years out. #railsconf
Our software has not improved anywhere near 25 orders of magnitude unlike the processors @unclebobmartin #railsconf

Funny how social sciences never really advance as fast as the hard ones. I think of programming as a 'soft' science because the hard part is NOT getting a computer to understand the code it's describing intent to humans.

.@unclebobmartin "What language do you know that still has goto?" guy in crowd: "PHP" #railsconf
A brief tour of programming languages with @unclebobmartin #railsconf
RT "vi? what the hell?" - @unclebobmartin #railsconf

Vi, Emacs, TextMate, etc. all seem like some weird time warp compared to the awesomeness of the language. It always seems like a really good Ruby IDE is just around the corner.

RT the point emerges. multicore processors pass the buck to us. future == parallelism #railsconf
Real innovation in software may be driven by the move to parallel processors @unclebobmartin #railsconf
.@unclebobmartin is hawking #sicp Yeah, yeah, yeah. I'm getting to it. #railsconf
#sicp is a page turner? Really? @unclebobmartin #railsconf

Structure and Interpretation of Computer Programs is so often praised that people in the know just refer to it as sicp. It's available for free here: http://mitpress.mit.edu/sicp/ I'm thinking of working through it in clojure. And who doesn't like LISP? Uh...

But his point about multi-core is important (although it has been made before). A lot of people took his talk to mean that Ruby will lead the way into the future of programming. I think his point was that the people at Rails Conf will lead the way -- but probably not in Ruby. I've seen a few frameworks for handling concurrency in Ruby and they all suck. Why? Because you have to remember to use them all the time. All it takes is one programmer mutating a state somewhere in your code and now you have an intermittent race condition bug. If you're really going to do concurrency you need a language that treats all mutation like a disease that must be contained in explicitly declared blocks. And that language is not Ruby.

But I think we still have a few years left before we all have to get functional, so that's nice.

There's 2 prominent black dudes in the Ruby community. I just gave a hardy hello to one and referred to him by the other's name. #railsconf
Sorry. Sorry. Sorry. Sorry. Sorry. Sorry. Sorry. Sorry. Sorry. Sorry. Sorry. Sorry. Sorry. Sorry. Sorry. Sorry. Sorry. Sorry. #railsconf

In my defense: I'm an idiot.

You May Also Be Interested in: Implementing User Recommendations in Rails - Matthew Deiters

So Amazon makes 25% of its sales based on recommendations. I was talking with a friend from Groupon the night before and he was telling me that just guessing the sex from a person's name (and targeting what they get offered) has led to a significant sales boost. If you're not thinking about how to intelligently recommend to your customers, it's a sure bet your competitors are.

Aaahh the Hoff's crotch is coming at me! #railsconf
Oh it was just an animated gif. #railsconf

I urge you not to click this link: http://www.post-literate.com/gerpunx/archives/2005/01/prepare_to_lose_your_mind.php
I think Deiters was trying to make a point about recursion in SQL being bad, but the image made my brain stop working.

Gem Neo4jr-social is a graph db that uses JRuby but isolates it in a Jetty server so you don't need to use JRuby throughout #railsconf
You can use Neo4jr-social to get friend recommendations and degrees of separation pretty easily. @mdeiters #railsconf
To which Charles Nutter (@headius) responded @jakescruggs Jeez, why not just use JRuby? So much pain could be avoided.
I really want @mdeiters slides for the explanation of all these crazy graph relationship terms #railsconf

Graph db's are amazing for relationship stuff. You can do "what's my degree of separation from X" stuff in milliseconds. And Neo4jr-social looks like a pretty cool way to do that stuff in a readable way.

You have a lot of money lying around in your data - pick it up! @mdeiters #railsconf
RT "This conference is the NoSQL conference. But remember: It's not No SQL. It's Not Only SQL" -- @mdeiters #railsconf

Really good talk. Lots of info in the slides:
http://assets.en.oreilly.com/1/event/40/You%20May%20Also%20Be%20Interested%20in_%20Implementing%20User%20Recommendations%20in%20Rails%20Presentation.pdf

Lapidary: the Art of Gemcutting - Nick Quaranto

RubyGems.org has a nice versioned API @qrush #railsconf
'gem yank' to remove a gem. Didn't know you could do that. There's an undo too. Nice. @qrush #railsconf

Versioned API and an 'undo' -- cool.

gem webhook projects suggestion - gem tarballer @qrush #railsconf
gem webhook projects suggestion - distributed testing service @qrush #railsconf
gem webhook projects suggestion - change log service @qrush #railsconf

All these ideas are great. I command the readers of this blog to go implement them now... Please?

Gemcutter moved from postgres to redis @qrush #railsconf mostly
Lots of Gemcutter clones out there @qrush #railsconf
.@qrush wants to add better indexes, dependency resolution, and support bundler in RubyGems.org #railsconf
.@qrush also wants historical data for every gem with download graphs. @qrush #railsconf
github.com/rubygems/rubygems No longer on svn. @qrush #railsconf

All very cool. Nick is clearly firing on all cylinders -- he deserves a raise.

Yehuda Katz - showed up and thanked Nick for his contributions to the Ruby community. I was in quite a few talks where he made it a point to thank the speaker for things he really liked. Stand up guy that Yehuda.

RT: OH: "Firefox is the new IE." #railsconf (via @glv)
Ouch. But kinda true.

The Cold/Hot/Cold/Hot/Cold/Hot/Cold/Hot of this week has made me weak. Picked up some knockoff airborne at the CVS #railsconf
Getting sicker... Must consume more vitamins and wear more clothes. Covering up crazy shirt now. :( #railsconf
Must. Fight. Sickness. At. The. Airport.

At that point the sickness overtook me so I found a quiet place and laid down. So I missed the last keynote -- which I'm about to watch now.

And I'm back. Good Keynote. Actually, all the keynotes were pretty darn good this year. And so were the sessions. Shortly before I got sick on Thursday I remember thinking "This is probably my favorite Rails Conf."

Gary Vaynerchuk's talk is probably summed up best with this (made up) title:

Relationships, relationships, relationships, (and swearing) - Gary Vaynerchuk

Here's some memorable quotes and thoughts:

"Giving a Fuck is coming on strong"

"Stop using the space (twitter, facebook, etc) just to put out fires"

"If I get a hundred more followers I'll donate $100 to Haiti - Hey Fuck-face just donate $100 to Haiti"

"Everything you're doing is being documented"

"It's just getting real hard to hide"

In 2005-6 everything was free 24/7/365. Now people are being trained to pay for things.

On consulting for huge companies: Its stunning how little most big companies give a crap. Most CEO's want to keep the stock price up for 3-5 years and get out with a huge payday.

Getting people to your site is awesome but "Content is always king"

"What's the ROI (Return On Investment) in social media? Well I don't know fuck-face what's the ROI in having a real relationship? Meanwhile you're paying for billboards..."

Old businesses: "They lived under small town rules" - if you screw someone you're going out of business. Those days are coming back.

Corporations: "We don't want to open this up because people could say our product's bad"

Gary on the future: "I'm all in -- I'm bullish on human beings."

You can find Gary's Keynote along with all the others at: http://www.youtube.com/view_play_list?p=393ECE649BB3813D or on iTunes.

Well that's the end of my Rails Conf 2010 coverage. Thanks for reading.

Rails Conf 2010 Day 2

2010-06-09T23:35:00.004-05:00

Only 40 tweets today (re and otherwise). I must be slowing down in my old age. Tweets are in italics.

Today's morning music was "Bulletproof" by Pop Will Eat Itself. Cram that in your head @ryanbriones
Ryan was complaining yesterday that he had the theme from flash stuck in his head all day because of me. That sounds awesome! I had a Daft Punk song stuck in my head all day today and I was a better person for it.

Trying to work on my Lightning Talk. Lightning talks are usually the best part of any conf so it better be good. 4:25 today #railsconf
Lightning talk is finished. Title: "ActiveMQ and ActiveMessaging: I've Experienced the Pain So You Don't Have To" #railsconf

Yeah, I got up, worked out, ate eggs wrapped in cheese and bacon, showered and then realized I had gotten up an hour early by mistake. Opportunity! So I worked on the lightning talk. To be honest, I was kinda pissed at myself for the boneheaded alarm mishandling but if it doesn't make you money or happy, you need to let it go.

Don't forget to read my sham of a blog post about #RailsConf Day 1: http://is.gd/cIdLZ
So very meta.

iPod at #RailsConf Keynote has been set to "BadPopCountryMusic" wtf mate?
Seriously. Is there anything worse than top 40 wanna be country music?

Creativity & Constraint - Neal Ford

"Some of the 'facts' in this talk may in fact be harmless lies" - disclaimer of @neal4d #railsconf
This disclaimer needs to be in front of all keynotes.

I'm Not Old, I'm seasoned. @neal4d #railsconf
Me too buddy, me too.

"The right community to suggest something really weird to" @neal4d #railsconf
RT I so wanna go to Neal Ford's halloween party! #railsconf
Constraints are liberating: @neal4d excuse for showing cool Halloween photos at #railsconf

His Halloween party pics were off the hook. Did I ever tell you I took Neal to his first day lunch at ThoughtWorks? You'd think that would get me a Halloween party invite but memories are short. Jerk.

What constraints help? Maybe not static typing... @neal4d #railsconf
Double true.

Hemingway and the Ex Pats in Paris wore stupid hats #railsconf
Well they do. Americans in berets look stupid. There -- I said it.

An empty canvas is a daunting task @neal4d #railsconf
The invention of the camera made painters do something different besides making it look real. @neal4d #railsconf
The Camera gave painters freedom from the responsibility of representation. @neal4d #railsconf
Our camera is Java Enterprise Development - they can crank out vanilla apps so we can move on to interpretation @neal4d #railsconf

Interesting. I think he meant no offense to Java devs in specific but much offense to the mindless and complacent.

.@neal4d has a crazy terse 3 line Quicksort implementation #railsconf

Yeah, I had no idea what it did. You'd need a team powerful enough to turn goat piss into moonshine if you're going to check that in.

Non-obfuscating density is a characteristic of art in code @neal4d #railsconf

But how much density is too much? It's a thin line between art and crap.

RT #railsconf @_why : when u dont create things, u become defined by ur tastes rather than ability. ur tastes narrow & exclude ppl. so create

Neal ended up his talk with poignant quote from _why. Pour one out for the homies who couldn't be here.

Sponsored Keynote: Engine Yard's Open Source Love Affair - Evan Phoenix

Dr. Who Theme! Played live by #railsconf ers
Huh?

Did Evan come on as a rock star or a parody of a rock star? you decide. #railsconf
Rim shots at a keynote? #railsconf
Talk show format is... interesting Will it pay off? @evanphx #railsconf

A 'Late Night with X' theme backed up by a live band. And a game of password.

.@evanphx "Perl is the word you were looking for" Dr. Nic: "Never heard of it." #railsconf

Well played Doc.

Enjoyed a pleasant 10 minutes of nothing with @evanphx #railsconf

I really did enjoy it. Better than somebody coming out and saying "hey we bought this slot with our sponsorship and we're awesome."

And then Twitter went down. Again. So I had to type into a document instead of live to the world. It felt weird and dirty.

Ruby on Rails: Tasty Burgers -- Aaron Patterson

First, you have to download his slides just for the introduction: http://tenderlovemaking.com/railsconf2010.pdf

Two samples to entice you:

Btw his outfits put mine to shame. Today's was a deep magenta velvet jacket with a 70's tie and an ironic mustache. Or a sincere but poorly chosen mustache. There's a point when intense silliness becomes serious and upon that point Mr. Tender Love Making is doing an interpretive dance.

Topics he talked about:

sqlite3-ruby and Sqlite

Did you know you can have a virtual file system in Sqlite? Me neither.

The new version of sqlite3-ruby is x1000 faster than before (1.3.0 vs 1.2.5)

Mmap

Mmap is a gem that lets you treat a file like a string. So if you need to pass in a string to something but don't want to read the whole file into memory use Mmap. Pretty useful tool to have in your back pocket.

Rack

Unknown but useful rack trick: Rack will call 'each' and 'close' on the return object. So you can pass it an array or file and things will work out just fine. Handy.

He speculated that Rails could take advantage of this and return html in pieces (as it's rendered) to make things faster. Would be hard to do but awesome.

JSON is a subset of YAML -- I did not know that. Did everyone else? Really? I knew you all were talking behind my back.

He left the stage with a plea for the audience to dig into rails and investigate the 'Tasty Burgers.'

But wait, there was some advanced shenanigans when he called Evan Phoenix to ask him a question. During Evan's talk! Those guys are the kings of wacky.

Redis, Rails, and Resque - Background Job Bliss by Chris Wanstrath

No Slides at all! But still good.

"Redis is all we really wanted it for a long time but we didn't know we wanted it"

Redis is a key value store for data structures @defunkt #railsconf

This doesn't have anything to do with Chris Wanstrath's appearance but comical mustaches are the new black. Except that he has a mustache now. And it looks silly. Not that I can complain about peoples fashions sense.

Chris is totally in love with Redis and Unicorn because their maintainers are awesome, active, and their software is pretty bullet proof (it stays up). Which is kinda important to Github.

He advises you to background everything you can and I agree.

Resque is the queuing system they built on top of Redis. I've used it at Groupon and seen it used on Mad Mimi and it works well. Plus the built in Sinatra web UI provides an amazingly informative look into the live queues.

Github is on Ruby Enterprise Edition now. So is Backstop (my company). And everyone else at Ruby Conf. It just surpassed ironic mustaches as the new black.

Chris is determined to not add any queuing features to Resque. He wants to let the plugins do that and there are more than a few you should check out. I'd Google that for you but I'm tired.

Rocket Fueled Cucumbers - Joseph Wilk

So Joseph's cuke build was getting out of control as it grew. Taking more than 4 hours. He solved this by using EC2 and Testjour (or Hydra) to parallelize the suite across 20 machines and got it down to 11 minutes. But it cost $3K a month. Yipes.

He recommends stubbing out slow services and maybe even caching the db (but didn't specify how).

He's excited about using the cucover gem to do "Lazy coverage-aware running of Cucumber acceptance tests" Kinda like a super smart autotest.

He mentioned this post as required reading that he doesn't necessarily agree with: http://jamesshore.com/Blog/Alternatives-to-Acceptance-Testing.html

Capybara can use envjs gem to run javascript tests in cuke. Which is very cool. Capybara-envjs on github

Dear Lazyweb, how do you get a showoff presentation up on Heroku? I'd like to put my lightning talk up. #railsconf
Posted this before my talk. Still have no answer but if all the cool kids can do it I'm sure I can figure it out.

Lightning Talks

Lightning Talks start now and I'm giving one. Come to Ballroom 1 now! #railsconf
Yep.

RT client_side_validations http://bit.ly/9gEfuL #railsconf
Interesting gem. Way too much live coding for a lightning talk - dude was brave. He barely made it to the end and almost missed giving out the gem name. Live coding seems like a good idea but is to be avoided in presentations. Ignore my advice at your own peril.

Don't hide costs from the client -- it leads to bad relationships #lightningtalks #railsconf
Custom filters based on metadata looks cool in RSpec 2 #lightningtalks #railsconf
gem install surveyor for easy surveys in your rails app #lightningtalks #railsconf
surveyor gem can ask some complicated questions with easy. #lightningtalks #railsconf
RT Mind. Blown. http://bit.ly/90D1ur Now that was a lightning talk by @igrigorik. #railsconf

Yeah the schema free sql database thing was a nice antidote to all this NoSql hype.

http://bit.ly/RwandaOnRails looks like an interesting way to track charity money's effectiveness #lightningtalks #railsconf
"Yes and..." is an important part of improv and pairing. I totally agree. #lightningtalks #railsconf

My lightning talk went well and as soon as I figure out how to get it up on Heroku I'll give you (dear reader) the url.

Update: Here they are: http://active-messaging-pain.heroku.com/

Keynote - Derek Sivers

In 2007 Derek took a lot of heat for his article entitled: "7 reasons I switched back to PHP after 2 years on Rails"
Which can be found here: http://www.oreillynet.com/ruby/blog/2007/09/7_reasons_i_switched_back_to_p_1.html

I remember reading it and thinking he was a moron. He gives a pretty good keynote for a moron so perhaps I should reevaluate.

RT The first follower turns a lone nut into a leader. - @sivers #railsconf
RT derek sivers shows this vid http://bit.ly/guHuz to talk about being a leader and starting a movement #railsconf

The talk was all about how he preferred his own PHP frameworky thing but how that couldn't scale and so he's coming back to Rails. And is excited about Rails 3. Also there was a lot about building community. We use our emotional brain to justify our rational experiences. But sometimes our emotional brain has access to things we don't consciously know (detailed in the book 'Blink'). There was more but it's late and I'm pretty tired so I'm wrapping this blog post up with a few more tweets:

Fogo de Chao ftw #railsconf
Drunk. Drunk on steak. Drunk on chocolate. In fact, I'm like a chocoholic except for alcohol.

Interview Coding Problems

2010-05-18T18:06:00.000-05:00

One of the awesome things about the business we're in is that you can ask people to actually do the thing you're hiring them to do IN THE INTERVIEW! You can't really ask a banker to do some banking during an interview, but it's relatively easy to set up a computer with a coding problem and ask a potential hire to work through it. Having recently gone through a job search, I though I'd discuss my thoughts on coding problems as part of the interview process.

The first time I ever wrote code for an interview was at ThoughtWorks. It was a 3 day at home coding problem. There were always 3 different questions and each could be knocked out in a few hours. The candidate could choose which one they wanted to solve and upon submission, the solution was looked at by at least 2 different developers. The great part about this is that we could filter out a lot of candidates before bringing them in to the intensive day long interview. The downside of a take home test is that you never really know if they got some help. Which is why in this type of interview it's important to go over their solution in person.

One coding interview I did was 2 developers and I working on a Rails plugin. It seemed like a good idea, but I don't create Rails sites from scratch every day so there was a fair bit of fumbling around hooking up the database and trying to remember what generators to use and so on. When we finally got to the writing a plugin part I was a bit flustered and short on time. We made some progress but I don't think either of us were satisfied with the outcome. This brings me to an important point: You will accomplish a lot less in an hour than you think you might. This is because the interviewee is using an unfamiliar machine, text editor, doesn't have his or her custom bash aliases available, and is being watched. And questioned. All this will slow anyone down by 50%. So if it would take you 30 minutes to solve a given problem from scratch then it's about right for an hour long pairing interview.

At Obtiva, Colin and I paired on some truly horrible code that he had inherited. Oh the stories we could tell you about that code base... But I'm not sure how I feel about these "work with me" interviews if they are going to last less than 3-4 hours. Actually, a day would be a better time allotment for this type of interview. The reason is that solving a problem in an existing code base requires a lot of backstory. If you only have an hour or two it's hard to grok the domain and you may just be testing the candidate for familiarity with the subject matter instead of pure programming talent. It would be easy to miss a good candidate or hire someone who just happens to know the domain.

Speaking of domain, you should try as hard as possibly to eliminate it from your coding problem. I once did an interview with a firm where the challenge was to build a highly simplified version of something from their domain. The problem was that it really wasn't simplified enough. After reading a page and a half describing the system my head was spinning with new terms and concepts (I wasn't familiar at all with their field). They had helpfully started me out with some stub classes and some failing cucumber specs but when piled onto a shaky foundational understanding it actually hurt more than it helped. I had to figure out the domain, understand the class stubs (and any hints they may have been providing), describe what I was doing to the interviewer, understand what the cucumber tests were asking for, and navigate an unfamiliar workstation. It turned into a bit of a train wreck. Basically I did what I do when I don't understand something: I went slow and tested everything. But I ran out of time before I could even get one of the cucumber scenarios passing.

You should try to structure the problem so that it has an easily achievable first step. This enables the interviewee to get his or her confidence up to try the trickier parts and also provides an easy hint you can give them if the are struggling. You might say: "Hey, why don't you just try and get X working." X being the easy part. You'll get better results when people are relaxed.

At Backstop we have a couple of coding questions, but the central theme is that the domain is either something everyone is totally familiar with or we can teach them in a few minutes before the coding starts. We also make sure that the candidate pairs with different programmers to see how they interact with different people. Now pairing while interviewing is not like regular pairing, of course. When I run the interview, I approach pairing with an interviewee like I'm a novice developer who is a domain expert. I'll answer any questions they have about the domain but I don't help much with the coding part. I always allow access to any documentation they want to look up and of course I have irb open and ready for experimentation.

For me the pairing interview is just a way to have an engaging conversation about code. If someone doesn't solve the problem but does so in an interesting and thoughtful way then I consider it a success. This also means that an unreadable working solution is considered a failure.

Having actual programing be a part of your interview process will take a some thought, but all that time is well worth the effort. I can't tell you how many times someone has sailed through the talking parts of the process only to crash and burn in the coding section. Hiring someone who is great at talking about code but terrible at actually coding is one of the worst things you can do.

Upcoming Speaking Dates

2010-05-17T20:45:00.000-05:00

I'll be giving an all new talk entitled "The Necessity and Implementation of Speedy Tests" at Ruby Kaigi in Japan! Ruby Kaigi is being held August 27-29th, 2010 in Tsukuba, Ibaraki, Japan. Very exciting!

Want to see the above talk but can't make it to Japan? I'll be giving the same (ish) talk at Ruby Midwest in Kansas City, MO. Ruby Midwest will be held July 16-17, 2010.

Stop on by and say hello -- I'll be the guy in the crazy shirt.

Why I Left Obtiva

2010-05-16T20:20:00.000-05:00

Recently I ended my nearly 2 year relationship with Obtiva and why I left deserves a few words.

The issues I had with Obtiva were mostly just issues I have with consulting:

You only work at places messed up enough that they need to hire consultants
You leave before you get to spend time with code you wrote so it's hard to learn long term lessons
The most stable gigs that pay the bills for a consultancy tend to be the worst programming experiences (Large corporations with way more money than sense)
You have to track every hour of your time
Time off has to be negotiated with two companies instead of one
Learning a whole new bunch of names. personalities, and organizational structures every 6 months

Now don't get me wrong, consulting with ThoughtWorks and then Obtiva made me a much better programmer and was an amazing learning experience but I felt that, in order to grow as a programmer, I needed to work for a product company and learn the lessons that only living with a code base long term can teach you. Consulting is a great drinking-from-a-fire-hose experience that every developer should consider, but it has its limitations. So I joined Backstop Solutions and I write Ruby on Rails sites for hedge funds and the people who use them (note: we sell products to hedge funds, I don't actually work for a hedge fund).

Why Backstop? Well, it's big enough (60ish people) that they've worked out most of the kinks of being a business (direct deposit, 401k, health insurance, etc.) but small enough that you can get to know everyone and (hopefully) help steer the company. Their product is strong and growing, but there's some new products in the works too. They do Agile but would like to do it better (I've been brought on to help with that in addition to my coding responsibilities). But the best part was that while everyone is smart and nice, it's oddly unknown in the Ruby community -- and I can help with that.

Obtiva was a great experience. During my time there I moved from being just a coder to being more of a leader, started speaking at conferences, got to participate in many wonderful geekfests, helped mentor software apprentices (which I once was), and learned a ton from all the programmers I interacted with.

To sum up I'll quote from my company-wide resignation email: "Mommy and daddy still love each other. Except we're going to see other people. And not live together."

Are you Really Doing Agile Development?

2010-05-14T11:15:00.001-05:00

So recently at work I was asked help to make the company more "Agile." Well I'm a developer first and process wonk second so I responded with my usual "How many of the 12 practices are you really following?" Which was met with a lot blank stares. Turns out the classic XP practices are not so easy to find on the internet anymore. It also turns out that the word "Agile" has been so successful that lots of people don't know that XP stands for Extreme Programming. Now I'll be the first to admit that "Extreme Programming" is a colossally stupid name, but what I like about XP and the original 12 practices is that they were controversial and easy to evaluate: Either you were doing them or you weren't. What I don't like about "Agile" is that it's so broad and defined in such a "hand wavy" way that pretty much everyone can fool themselves into thinking that they already are fully Agile.

So to fix this and help my company start measuring their "Agility" better, I've decided to list out classic (They've changed over time -- but not for me. And get off my lawn!) 12 XP processes. And my highly opinionated view of each of them. After that I'll tell you how to score your "Agility."

* Planning Game
Get requirements from customer, form them into (short) stories, estimate the time it will take, do them, measure velocity, examine failed estimates for clues as to how to estimate better, and repeat. A story should be a placeholder for a conversation -- This only works if the customer (or surrogate) is highly available to the developers and the developers take advantage of that availability by checking in frequently during the development process. Note: Stories that take longer than a day significantly undermine many of the other parts of XP.

* Small Releases
Iterations should be 1-2 weeks (depending on how painful/expensive it is to organize an iteration). Releases should happen after every iteration if the application is easy to deploy (web apps).

* Metaphor
Nobody does this. Extra points if you do.

* Simple Design
This means different things at different levels. When kicking off a green field project a week of requirements gathering and design is good. For an individual story, a few minutes to an hour is good. Then you start writing code. Important: Design as you go. Stopping to spend a few hours with a whiteboard is allowed and encouraged. Do the simplest thing that could possible work until it becomes apparent that it will not. Then redesign. The idea is that at the beginning of the project you know the least about it -- spread the design across the life of the project so you can make design decisions with more knowledge.

* Testing
Test first. Coverage above 80% for Java, 90% for Ruby (less exception checking in Ruby means easier to cover). Non-brittle, atomic, fast tests (unit tests should run under 5 min. Actually under 1 minute is best, but most think that's impossible (spoiler: Its possible)).

* Refactoring
Short: Red, Green, Refactor! Long: Write a failing test, make it pass, refactor the code. This also means when you wander into a messy place you leave it a little better than you found it.

* Pair Programming
Above 80% of the time. Seriously. Tasks that seem like you can't pair on them often mean that you aren't using the right pairing techniques. Pairing is a skill: Get good at it.

* Collective Code Ownership
Anyone can change any code. Yes, they should consult whoever is knowledgeable about said code, but there are no parts of the code that only developer X is allowed to touch.

* Continuous Integration
Every check-in to the remote repository triggers a comprehensive suite of unit tests. The developers have to care if the build fails. For example: When the build fails, no one checks in until it is fixed. Also there should be some social ridicule for the build breaker (have to wear a stupid hat for the rest of the day, an obnoxious build breaker trophy placed on their desk, they have to buy donuts for the team, etc.). This all assumes that the build is reliable and that a failure means there is a real problem.

* 40-hour Week
Now called "Sustainable Pace" because "40-hour Week" tends to scare managers and end of project pushes are sometimes necessary even in XP. This should be no more than 2 weeks, have a clearly defined point at which the long hours will stop, and happen no more than twice a year. Actually, if it happens twice a year it means your estimation skills need to get better.

* On-site Customer
Doesn't happen often for non-internal projects. Usually customer surrogates are used. These surrogates need to take pains to talk to real users often (not just the managers of the users) and be highly available to developers to answer questions.

* Coding Standards
Doesn't matter what they are but the developers need to come to a consensus and stick to it. Consensus does not mean everyone nods yes in the meeting but ignores the standards when they feel like it.

So how do I score my Agility? Take the number of practices that you actually do, divide by 12, and then multiple by 100 -- that's your percent Agile. Do not give yourself a point if you don't fully implement a practice. Have Continuous Integration but some of the tests fail randomly? No point. You pair "when necessary" which ends up being 40% of the time. No point! Run iterations but don't estimate your stories? No Point!!

This is take a hard look at yourself in the mirror time, not some hippie love fest.

Lone Star Ruby Conf 2009 Day Two Afternoon Sessions

2009-08-30T18:51:00.004-05:00

More Lighting Talks!

A lot of people skip the lighting talks, but I find that some of the best stuff at any given conference comes out of the these intense 7 minutes sessions.

Jared Ning: "Ruby Without Borders"
Matthew Todd gave a talk last year asking for people to came to Tanzania and help him code in Ruby. Jared did and he loved the experience.
He mentioned a very useful TextMate short-cut:
If you have a bunch of files and folders open in the project drawer and you click the chevron (the triangle thingy) to close a top level folder then when you open it back up TextMate remembers the state of what folders were open and closed underneath. But sometimes you want to close down all the folders underneath. Option clicking on the chevron will do that. Also, option clicking on a closed chevron will open up all sub folders. I've been looking for this.

Check out Ruby Without Borders.

Jeffrey Taylor: "Fast Multi-protocol XML Parsing"
Jeffrey's project had to read a lot of RSS feeds and it was super slow even with the fast ruby xml parsers. So he rolled his own, which can be found here:

http://abluz.dyndns.org/fast_xml_parsing.rb

Jeffrey used the sax model to do super fast xml parsing (10 times faster than nokogiri - he claims). He does admit that the code has a Flog score of over 500 for one method.

Hal Fulton: "Reia: The Next Big Thing"
Reia is an attempt to combine Erlang and Ruby. If you want the cool concurrency of Erlang with Rubyish syntax, take a look.

Yehuda Katz: http://github.com/wycats/bundler
A true dependency resolver
From the README:

Bundler is a tool that manages gem dependencies for your ruby application. It takes a gem manifest file and is able to fetch, download, and install the gems and all child dependencies specified in this manifest. It can manage any update to the gem manifest file and update the bundled gems accordingly. It also lets you run any ruby code in context of the bundled gem environment.

So basically it's a better way to vendor your gems with your project. Check it out if you've had trouble using other systems.

Before Matz's Q and A, Jim Freeze got up and talked about Lone Star Ruby Conf's attendance over it's three year span:
2007 - 200
2008 - 282
2009 - 230ish

Which is right around where they want it. Small enough to be intimate, large enough to be interesting. And that size group fits nicely in the Norris Conference Center.

Btw, the network was outstanding this year. They bought something like 6 wireless routers and although sometimes it was sluggish, it remains some of the best wireless I've had at a conference. Usually the high tech load crashes the network within minutes.

Matz Q and A
The two answers that stood out to me were when he said that the Perl $ variables were the thing he regrets the most about designing Ruby. Also, when I asked him about how often he gets to write in Ruby he said that the largest Ruby program he ever wrote is 2-3000 lines of ruby code. It was a mail client with Emacs front end. Generally he uses it for scripts and such. I find it amazing and a little heartbreaking that Matz doesn't get to program in the language he created and clearly loves.

Encoding Domains - Rich Kilmer

First, there was a lot of buzz about Prezi at the conference. Prezi is a canvas presentation tool that so impressed Rich Kilmer that he learned it the night before and wrote his keynote in it. In Prezi you put all your text, videos, images, sound files, and whatever in one huge canvas and then tell give it a bunch of waypoints and zoom levels to follow. It's pretty bad-ass.

This was Rich's first keynote ever. Which is kinda hard the believe as he's been doing amazing things with Ruby since the dawn of time (which is 2000, btw). In it he discussed how software libraries don't have value, they have potential. What has real value is encoding domains. He told an interesting story about how he encoded the domain of a massive military project mostly by sitting down with some experts in the field and figuring out how they would like the DSL to read. Once he had captured all the information they could enter in to their logistics system in syntactically correct Ruby, he went about making it work. He called it syntax driven development and it worked so well that 5 years later he still gets calls about the prototype he wrote in two months. Apparently they are still trying to create the real product in some other language.

Well, the perfect storm of conferences (Agile 2009, Software Craftsmanship North America, and Lone Star Ruby Conf) is now over and I'm looking forward to actually writing some Ruby tomorrow. I need the rest.