Putting the fun into Funkworks

ruby-prof

2011-11-11T11:34:00.001-08:00

When optimizing for performance, one should always run a profiling tool to see exactly where the real bottlenecks are. The results can be surprising. I had written a ruby program to do a data discrepancy report between 2 systems. It was running pretty long (seems extra long since I was developing/tweaking it and had to run the report each time), so I was of the initial impression that after I got it right, I might have to rewrite it in say Java or C++. Fortunately for me, I profiled it first.

The tool was ruby-prof. Ruby-prof runs better on Ruby 1.8.7, which is fine for me in this instance because the target system runs Ruby 1.8.7, but people have been trying to address the issues for Ruby 1.9.

To install the default gem

gem install ruby-prof

Ruby-prof has many options to run in the your code, but you can just call ruby-prof on your ruby file. I'd recommend the graph_html output (ruby-prof -p graph_html) as the hyperlinks make it easy to traverse the report.

Once you have the report, you can figure out what's taking up a disproportionate amount of time. For my situation it was detect()

It turns out I had 2 large loops that were using detect() in a situation that would be better served with a look up table. Given my development data set was much smaller than the production data set, it performed adequately during development. This was not the case when the data grew that by 1000x. I replaced the detect() calls with look up tables and the run time went from 17 minute to 2.5 minutes. Sweet, no need to convert to another language. Thank goodness for ruby-prof.

Sending function keys on OSX

2011-11-11T11:20:00.001-08:00

Just a quick note since it took me a fair amount of googling to find it. If you need to send functions keys to a terminal application, say byobu, you need to prepend the command key. On an external keyboard, command - function< whatever>. , on macbook (pro), fn + command + function< whatever>.

A Git PSA: Git Bisect

2011-11-11T08:54:00.001-08:00

Ever had a time where there were a slew of commits and you don't know which commit caused the build to fail? Finding the bad commit could be approached w/a binary tree traveral sort of approach: go back and find a good commit, pick a commit halfway between that and HEAD, and based on whether the commit breaks the build or not, pick a commit halfway in the appropriate direction, triangulating on the bad commit. If you use git, then you are fortunate enough that git will do this for you via git bisect.

So I spent several hours being a poor man's git bisect before discovering git bisect, and I thought I'd pass the savings to anyone who would listen in this Public Service Announcement.

(poor man's image from http://boykembara.com/travel-tips/)

Git bisect, in typical git fashion, has many options. Being the lazy sort, here's what worked well for me w/a minimum of commands. It depends on having some sort of program you can run to determine whether the commit is good or bad. For us, it's our test suite.

To set up, run this:

git bisect start HEAD

This sets up the end points of HEAD, where I was when I discovered the problem and I'm going to assume will be a majority of the cases. You can replace this with another start point. The last parameter is some known good point. If you don't have a command that can run to determine good/bad, you can investigate manually and run git bisect good or git bisect bad, and keep going until you find the guilty commit. But, if you do have a command, you run this:

git bisect run

in my case

git bisect run rake spec:controllers

When it's done, you get a result like this

985 examples, 0 failures, 5 pending
ec65d267959dc8e317a4cc535094fd8483b02a14 is the first bad commit
bisect run success

Now you have the guilty commit. To undo the bisect branch do a

git bisect reset

Hope to saved someone some time.

A new level of Microsoft goodness!

2011-10-03T15:06:00.000-07:00

So I was installing a Microsoft update and the installer got to the point where the installer detected if any programs needed to be stopped for the installation to continue. It determined that I'd have to shut down Chrome, Safari and Entourage. I expected the last as that's what the update was for, but not the browsers. Consequently I slide the dialog window aside so I could finish reading the article I was reading before the dialog came up.

A little later, the dialog jumped to the middle of my screen and flashed repeatedly. I'd never seen that before. I guess the installer knows better what I should be doing w/my time. Better than my machine rebooting itself w/o my permission like I used to get on windows (and really hated). I just had to laugh at it, and then compose this post

A little bit of Airbrake goodness

2011-09-22T13:30:00.000-07:00

We use Airbrake (the service formerly known as Hoptoad) for error collection which is quite useful. I just discovered a little bit of "goodness" that I hadn't seen before. I was duplicating an error on development for an error reported on our production system and I saw this

I'm not certain how long that's been there, but it's nice touch to be able to both be able to know an error I'm experiencing has occured in production and having the link the Airbrake (my team can't stop calling it hoptoad, it just falls off the tongue better) right there.

Sweet. Thanks Airbrake.

Part of today expressed in Ruby

2010-12-17T13:51:00.001-08:00

forrest.drive

forrest.hear( flapping_noise )
forrest.look( flat_tire )
flat_tire.found? == false
forrest.look( broken_belt )
broken_belt.found? == false
while( sound_still_happening )
 forrest.worry!
end
sound.stop? == true

(forrest.take( car, mechanic )).to_be_sure
forrest.got( the_scoop )

timer.add( 5.pm, lambda { forrest.money -= 1000 })

money.overrated? == APPARENTLY

2010-09-28T16:19:00.000-07:00

We just had an Agile Process Overview w/the engineers and customer. Post AOSOCAL, I wanted to do this for several purposes

Make explicit the process we are following
Give an assessment where we are on those practices (good, needs improvement, etc.)
Address additional process/practices
Get team buy/ownership/empowerment

I drafted up a Google Doc spreadsheet of 3 parts

the (eXtreme Programming) XP practices in various evolutions and categorizations
A by practice assessment of where we were
An additional practices for things outside of XP

We did this live on Google Docs and iChat (Apple wasn’t lying about the bandwidth improvements to the Snow Leopard — the 3 in the center are actually co-located but we egregiously decided to exploit iChat). I think it went well, and it met my purposes. I’ll say the multi video chat and real time interaction on Google Doc spreadsheets really contributed well.

Our new pair station

2010-09-27T16:27:00.000-07:00

The 1st of our new pair stations for the engineering team at Hedgeye. Just a further evolution on a theme used by Hashrocket, Engine Yard and others, taking advantage of Apple's current layout (we're a Ruby on Rails shop, so life is much nicer on Macs).

Components

27 inch i5 quad core iMac as the pair station
2 15 inch, i7 Macbook Pro's per each developer, each w/22 inch external monitor

We're loving it.

Agile Diary

2009-06-09T22:18:00.000-07:00

Dear Diary:

I had a goal in January to write a series of ongoing blog posts called Agile Diary.

While a lot has been written about various Agile methods, rather than a presenting a methodology or case studies, I thought a regular, diary entries might paint a different picture and provide different insights.

I hope to at least talk about:

The situation
What my initial goals were
The initial plan
how we were able to follow
what we had to adapt to
failures as well as success

Naturally as situations will always change, we will always iterate through the above --- Diary Iterations, per se.

So I'm 5 months behind, I'll try to write some fill in articles and catch up.

Forrest

Creating dates in a time zone w/ActiveSupport

2009-04-02T15:47:00.000-07:00

As easy as 1-2-3

eastern_timezone = ActiveSupport::TimeZone["Eastern Time (US & Canada)"]
mon_before_market_opening = eastern_timezone.parse "3/23/2009 07:59:59 AM"

Rails, "do what I mean"

2009-03-10T16:41:00.000-07:00

A pleasant surprise for me just now. I typed in

script/generate migration add_application_to_content_categories application:string

And the entire migration was generated. The descriptive name I thought of was the right convention to trigger the instant magic. Beautiful.

Jack Bauer 94

2009-03-10T14:44:00.000-07:00

I was reminded for some reason of this video I saw last year to share w/my coworkers. Thought I'd share it here. http://www.collegehumor.com/video:1788161

It's 24 if shot in 1994. Amazing how far we've come since

Instant 3 way bi coastal debugging session

2009-03-10T11:27:00.001-07:00

courtesy via iChat

VirtualBox > VirtualPC

2009-03-04T12:31:00.000-08:00

I have available to me a rather muscular 20inch Vista laptop -- that's right, 20 inches, this is a laptop for King Kong.

Anyways, one of the uses for the laptop is to run a Microsoft provided XP VirtualPC image with Internet Explorer 6, unfortunately still a dominant browser and the "Great Satan" for web developers everywhere. As part of the virtualization wars, Microsoft has graciously made VirtualPC 2007 a free download.

I also planned to setup a CruiseControl.rb instance on a linux VM, and the Vista laptop had plenty of resources to spare (even running Vista). Naturally since I was already running the XP image on VirtualPC it made sense to try and to reuse VirtualPC for a Linux VM.

Some googling, and hoop jumpings later I had a Linux VM running, but it was kind of slow, and unstable. The Linux VM never stayed booted up for more than 2 days. MS does not officially support Linux VM's, go figure ;). Apparently there were linux add ons for VirtualPC 2005, but those didn't support Ubuntu, and no one has been able to get them to run with VirtualPC 2007.

So I decided to give VirtualBox a try after hearing good things about it. The install went very smoothly, and the VM is both stable and quick. The VirtualBox VM runs our unit tests in about 1/3 the time of the VirtualPC.

In general I seem happier with using VirtualBox, though I'll have to come up with a different solution. I will say on VirtualPC's behalf, that XP ran fine under VirtualPC

thru-you.com

2009-03-04T12:21:00.000-08:00

Wow, just caught this from @obie http://thru-you.com/

It's amazing what "undiscovered talent" there is out there. Though I imagine he won't be undiscoved that long. Makes me wanna play again.

Remote collobaration via 4 Way Video Conference with iChat

2009-02-27T11:07:00.000-08:00

Title says it all. Mondo cool.

I stayed home to help watch a sick child, but this came in handy. Normally the 3 on the right are in the same office in Southern California (2 of them are on the call), and the gentlemen at the left is in New Haven, CT.

Pretty nice for free!

Rapid progress: kudos to JRuby

2009-02-12T09:22:00.000-08:00

Courtesy of Facebook

Charles Oliver Nutter JRuby has had over 850 revisions applied since 1.1.6 was branched in December. That's better than 14 commits per day. And 153 issues fixed. about an hour ago

Sweet...

The significance of Red in Red, Green,Refactor

2008-12-22T17:01:00.000-08:00

(image from http://weepeixin.blogspot.com/2008/06/red-light.html)

What a lot of people miss is that the Red part of the TDD mantra "Red, Green, Refactor" serves as a test of the test (or spec for the BDD faithful). In seeing a test fail prior to the addition of code and seeing the test subsequently pass after the addition of code you learn 2 things

The test indeed tested something that actually changed with the addition of code
The code added had something to do with that test changing

One really appreciates this when they've been bitten -- written a test w/o bothering to see it fail, and writing code that supposedly makes it pass, but in fact did not. What can happen, albeit infrequently, is that since the test might've passed w/o the addition of new code--- the code added had nothing to with the code passing, or the test didn't test what you thought it did --- both variations of "accidental programming." Unfortunately, you won't be alerted to this since you never had the red turn into green as a result of the new code. This will break the normally "debugger free" cycle of TDD and reduce your coding momentum. I've been bitten, so I'm particularly aware of this phenomenon. Simply making it habit to always see the red, can avoid this.

I think side effects of not being able to "test the test" occurs more frequently test last, whether you are doing so by design (non TDD coding) or by necessity (adding tests to legacy code, working on code that is not easily TDD'd). In such situations, it's often hard, or not worth the ROI to find a way to make the test fail prior to writing the test. Consequently, writing tests last can often just serve as "documenting how the existing code works", rather than testing it.

(image from http://blogs.technet.com/seanearp/archive/2007/10/02/map-active-directory-in-visio.aspx)

I fell prey to this effect while putting some tests on some LDAP/ActiveDirectory code that I had to explore to get working . Normally, I strive to TDD as much as possible, but I couldn't really see how to do so in this case. I did a spike, exercising the LDAP API until it got the results I needed, then proceeding to refactor and add tests to this "legacy code" (as defined by Michael Feathers).

The test in question itself was wrong -- it didn't test what I thought it did, but since I never saw it fail, then pass, I didn't know it. Fortunately, I had a controller test (Rails app) that touched this code which failed and alerted me to the faulty test which also should've failed (yay multi level test coverage).

I'm reminded of Micah Martin's Ruby Kata and Sparring presentation at RubyConf where Micah did an exercise of live coding w/critiques as a device to improve one's coding. He mostly did a by the book "red, green, refactor" TDD/BDD with Rspec. I had pointed out that he had written a test in which he never saw it fail, but went straight ahead wrote the implementation to see the green. Now it was obvious that Micah had coded this problem before, that he pretty much knew what he was going code here, and that it was probably unlikely that there would be existing code to make this test pass, but none the less it's possible he might've written the wrong test, which would've sent him into a debugging cycle at some later point. When I brought this up, another attendee thought it was a spurious criticism, something along the lines that "we were judging his coding, not his adherence to TDD." I feel that person didn't get the significance of Red in Red, Green,Refactor, but perhaps he's never been bitten.

Kent Beck is a bloody genius, or How I learned to stop worrying about clean books and love the highligther

2008-12-21T22:11:00.000-08:00

I just started reading Kent Beck's Implementation Patterns and have been quite impressed. I've read several of his other books, Extreme Programming Explored, 1st and 2nd Editions as well as Test Driven Development and find him to be remarkably insightful and communicative. The way he's able to express deep thoughts with clarity and brevity, while making it accessible rather than pedantic and dry is a joy to read. While I agree with and have often independently come up with many of the same thoughts he has expressed, Kent expresses them much better than I could. It is definitely useful for me to be able to draw upon his words when trying to express the same thoughts. Kent Beck is a genius.

In the past I've never taken too much writing in my books. I suspect it may stem back partly to the "Don't write in your text books" public school school training. Other factors may be some residual "keep your books pristine" OCD and the fear of marking the books up with something that I might not agree with later. As a compromise, I've annotated books w/sticky notes but doing so was often awkward. Many times, I'll read a section that I've gotten a lot out of, made a mental note to revisit, or write about it, but usually never got around to doing it. Consequentlym I would lose those ideas.

I started reading this book and mentally noting some great point I might want to discuss w/a co worker or write about, and finally broke down and got out a highlighter and a pencil and proceeded to mark it up.

Wow, I think I've enjoyed the parts I've read thus far much deeper than I have in past. I think the effort of highlighting and writing in the blank sections really helped to gel things more than just reading. I'll note that Head First series tries to actively encourage such as an active learning exercise as well.

Anyways, I've had such a good time doing so, I'll have to go back and reread and annotate my other Kent Beck books doing the same.

Done in by truncation

2008-12-10T11:28:00.000-08:00

This was from my LinkedIn Network Updates this morning. Names removed to protect the innocent..

I've highlighted the key part that was "done in by truncation."

Of course, we know that it was supposed to be "She was a huge asset"...

Or do we ;-)

User Experience and Service

2007-06-12T10:53:00.000-07:00

I have a great mechanic. Aas an independent, George doesn't have the huge resources a dealer has, but he beats the dealer in every aspect except for free carwashes with each service. Aside from being an amicable guy, he's honest, only fixes things need fixing -- most repair shops perform services according to mileage whether or not the work needs to be done (blindly following rote process vs doing acting according to context), strives to be preventive rather than reactive, and tries his best to minimize costs for the customer.
(image from agentz.com)

I was supposed to have my car done this morning, except one of George's suppliers didn't come through for the 2nd day in a row. George offered me a rental car at his own expense. I replied that he could just drop me off, but George insisted and I eventually relented. Shortly after, George had to convince another customer, who both just lived up the hill and had an extra car of his own, to accept a rental, as another supplier had failed George wrt that customers service. I was very impressed how much effort George would go through to right a wrong he had no control over. Utlimately, my car was ready 90 minutes after I got home.

In general George goes all out to give the best experience possible. He takes the hit for things out of his control without a second thought. He has my business for as long he runs his business.

Low prices and cost cutting have been squeezing out service/user experience in most businesses It makes sense: people, myself included, like to pay the least amount possible. Systematically cutting costs allows one to lower prices. Low prices brings in the customers, which ought to produce impressive metrics, at least for the short term. What does this mean in the long term? Dell used to be renowned for quality, service and the best prices. This is what shot them to prominence and profitability. Like many, they outsourced their customer support overseas to cut costs and now they're taking the long term hit of having eroded away their reputation for good support.

Ironically, they were only following management "best practices" of the time. Unfortunately, in doing so, they destroyed what made them famous and now HP is beating Dell at the game Dell created. Don't get me wrong, there is strong value in systematically improving quality, and the capability to create such, but it's the blind following of numbers without paying attention to context and what makes a company unique that can lead you down the wrong road, for example 3M adoption of Six Sigma eroded their innovation culture. It's all too easy to be short sighted or have tunnel vision when following metrics. When you reward based on a metric, people will adjust to satisfy that metric w/o context of what is actually supposed to be done. The 3M article points out "Six Sigma green belt projects" being pursued just to look better. Context is important and now 3M is trying to keep the efficiency gains Six Sigma brought them in manufacturing, something it needed, while trying rekindle it's innovation culture in R&D, which is what made them what they are via a dual system approach.

Service/customer experience still lives on where a premium price is paid, i.e. Apple, Lexus, etc. Google stands out as striving to provide premium user experience w/o charging the user a premium price. George stands out as a little guy beating out the big guys (i.e. dealers, large auto shops) in pretty much all areas, mostly by doing the right things. Thing which don't make sense according to mainstream business "best practices" focussed on relentlessly cutting costs.

There are lessons to learn here in software. Ironically, these lessons are the most appropriate for the entreprenurial, who already seem to know these things. The right combination of people (top notch, open minded, flexible), with the right technologies (such as Ruby/Ruby on Rails, Django, Erlang for concurrency, etc.) and process (Agile/Lean), backed by the right culture/philosophies (Google innovation culture, striving to provide "kick butt" user experience ala Creating Passionate Users, etc), can produce better quality, superior user experience products in less time for less cost, far outperforming huge corporations following "best practices." 37 Signals makes for a good example.

So in closing, Rock on George. If you're in need of a mechanic in the Laguna Niguel area, drop me a line.

3 Easy (3EZ)

2007-06-06T14:25:00.000-07:00

At my present job, I work on and/or deal with large, complicated systems, which typically have the following characteristics:

custom hardware
many machines
big iron (major processing power) hardware
millions of lines of code
requires several companies working together for years to produce

To make this level of complexity manageable I've produced a lot of tools to help visualize, control, test, and automate while having a lot of fun in the process. Looking back, there is a common philosophy, 3EZ that drives what I do in these situations. As I've also successly applied this philosophy to much less complicated projects, I think 3EZ has a wide range of applicability.

3EZ

When dealing with a system, the following 3 things should be easy:

It shoud be easy to understand the state of the system, what is currently going on, or what has happened. We're talking quick, easy and effective visualization, to use the word that Ward gave me, both real time and post mortem.
It should be easy control behavior. You be able to pefrom common tasks easily, i.e. 1 click, or a menu entrie. Editing a bunch of config files and restarting does not qualify for easy. You should have a high level control without ever having to restart the system
It should be easy to automate complex behavior and test the system. In the previous point I talk about easy things being easy. Complicated things should also be easy. Once you can automate things in a complex system, you give yourself the abilty for automated self verifying test. When automation and self verified test are easy, they are likely to happen more often.

(updated: cleaned up the wording a bit)

W.O.W Wisdom of Ward

2007-06-06T13:53:00.000-07:00

So I got the chance to spend a little time with my hero Ward Cunningham at RailsConf 07. For those unfamiliar, here's a bit from Jon Tirsen, who's basic sentiment about Ward's accomplishments mirror my own.

Ward has been called what I would call the "master of simplicity." Martin Fowler has described something similar as Wardish. I've come across this sort of description from several people when describing their interactions with Ward. Since I last saw Ward at OOPSLA 2005, I will say I have experienced profound benefits from a number of very subtle things that Ward shared with me.

I'd like to share some entertaining moments that occurred while having lunch with Ward on the last day of RailsConf.

Visualization

Ward had summed up in a single word something that has been a recurrent theme in my work for many years. We were talking about making heads or tails of complicated things. In typical Ward fashion, he said something along the lines of "It's all about visualizing the information."

Visualization. Wow, that's the 1 word I'd been looking for but not found.

Not having the 1 word, I'd use several sentences trying to describe how complicated things could be visualized (didn't necessarily use that word at the time) easily or efficiently. I knew what the problem was, how I would solve it, how the solution would greatly benefit all involved. Try as I might, I could at best reduce it to a sentence for each part. Ultimately, I ended up saying too much, causing folks to miss out on the essence of what I was describing. Now I can tell people how I will allow them to quickly easily visualize the complex activity they're struggling with.

I'm sure you might be thinking, "Duh!", but that's a typical reaction to a Wardish suggestion.

Ward's Secret to Visualization

Are html tables (FIT anyone?). If you can quickly tabularize the data in question, you can quickly and easily display it on any platform.

One of my "secrets to visualization" is quite similar, and involves "real time dynamic workflow diagrams" that show where you are, and where you'll be going. Definitely involves a bit more work than html tables, but has served me well.

Ward did talk about a cool variation of his table approach involving producing swimlanes for test data. It was cool and gave me ideas to use. I hope to write about it when I get the chance to investigate it in more depth. I did suggest to him, that they could use swimlanes as real time status rather than just after the fact analysis, which he thought was a good idea.

Lots of testing going on with Rails

Ward made the comment the huge difference between RailsConf and the Pacific Northwest Software Quality Conference(PNSQC). He contrasted the atmosphere where at RailsConf testing is thoroughly ingrained in the culture. There is lot of testing and excitement about testing. At the PNSQC it's much gloomier with themes of how hard/painful testing is and how you can't get people to test (Should sound familiar to non Agile groups). I believe he summed it up with something like "There more testing here than at PNSQC which is supposed to be about testing."

I know FIT!

Ward and I shared the lunch table w/3 RailConf attendees who didn't know was. Following the previously mentioned testing talk, one of the attendees started to mention success he had with using FIT to make testing more accessible. He asked if we were familiar with FIT.

In true down to earth, low key, Ward-like fashion, Ward said, "I know something about FIT".

Gotta love that about Ward. So down to earth and approachable and never one to toot his own horn.

I felt it appropriate to let them know that Ward was the inventor of FIT.

The artist formerly known as Ward

The immediate followup to the above moment

Attendee: "Oh, I see, so your last name is..."

Ward: "Cunningham, But I like to just go by Ward"

Me: "You know, one name, like Bono, Madonna, Cher, Prince..."

Cool Stuff

So Ward gave us a demo of his swimlane with hyperlinks visualzation.I t was part of the testing for the app that the Eclipse Foundation uses to track the workflow of their process. I'll try to write a bit about this as I get the chance. Really cool stuff that I will certainly borrow ideas from.

Simply having a wonderful RailsConf time

2007-05-20T00:57:00.000-07:00

Apologies to Sir Paul McCartney on raiding his line...

So it's late the 2nd to last night of RailsConf and I figured I should just put in a good word or two. So RailsConf has exploded to 1600 person cutoff (which sold out in very few days) from say 180 people at RubyConf 05, and I have to say, it's kept much of the quality and intimacy of it's RubyConf roots.

Good presentations, interesting entertainment -- a ZeFrank keynote which had the highest humor density of any talk I've ever seen to the "sleazy marching band" they had for lunch, to cool parties in pubs with free as in free beer, beer. Plenty of movers and shakers to be around to talk to, old faces, new faces, it's all good. I don't go to all that many tech conferences but I have to wonder if they're all as good as the Ruby and Rails conferences.

I've been fortunate to be around a number of really great conversations about the most edge stuff, and it's all quite rewarding.

Forrest

Lessig Hardt style for kids

2007-05-15T12:26:00.000-07:00

So I helped put together a presentation for my son's presentation on the Aye-Aye (he's in montessori kindergarten). I naturally took techniques I've been developing for the Lessig/Hardt/Takahashi style that I've been favoring for a while.

It turned out quite well I think. Seems to read like a small kid's book. Seems to me that the same sorts of things you'll do to catch and keep the attention of adults applies to kids, and vice versa. The combination of very few but the right words, copious and appropriate imagery and flow applies to it all.

Forrest