Ruby on Rails Performance Tuning

Cloud tools for rapid development, prototyping and profit!

2011-11-12T17:12:00.001-06:00

Slides from my "Cloud tools for rapid development, prototyping and profit!" given in Eau Claire, Wisconsin at www.chippewavalleycodecamp.com

Cloud tools

View more presentations from John McCaffrey.

Performance Tuning Tutorial Video 2011

2011-11-02T13:57:00.000-05:00

Rails Performance tuning presentation from WindyCityRails 2011 I had a great time giving this presentation and had lots of conversations with the audience members afterward. I even heard back from a few people that they've used the tips I covered to drastically improve the speed of their apps! I have more to cover on this topic, and I've been working it into a workshop format so we can spend more time on each area and do some hands on. Let me know what areas are interesting to you, and maybe I'll run a beta version past you.

Most developers know how to improve initial page load performance. We know how to follow recommendations from tools like yslow and page speed. Still, we struggle with slow performance after pages are loaded.

This talk will review:

* The basic rules from yslow/page speed (and when to break them)
* Tools for finding bottlenecks and slow javascript
* Common javascript performance patterns and how to fix them
* Architectural and design approaches to improve performance
* Considerations for Mobile performance
* Performance features in HTML5 and Rails 3

John McCaffrey has been developing applications since 1998, and is passionate about improving application performance and software development performance. John is an independent consultant focusing on bringing Rails applications to market quickly.

Building a team? Looking to join a team? Visit the ChicagoRuby Job Board: http://jobs.chicagoruby.org.

Here are my slides, please let me know if you have any questions, I will answer them if I can. (there's always something that I wish I could have covered more)

Windycityrails page performance

View more presentations from John McCaffrey.

Recent Web Performance statistics

2011-10-26T13:42:00.001-05:00

Web application Performance impact

Most statistics on web application performance are described in terms of public applications, but the data is still applicable to internal applications. Given some recent statistics, lets think about what this means for an internal application.

While content is increasing, and sites are doing more, users still expect them to be faster. For internal apps, you can think of these graphs as showing when "a user will be annoyed and wish they could leave", "a user's flow is interrupted and they start thinking of something else", or "when they abandon the task".

For retail apps, it easy to think of the impact of performance on 'conversions' in terms of users making fewer purchases, but for internal applications the metrics are different. The general idea though is that if the app is slower, internal users are able to do less work, or if the app is about 'informing their decisions', then we can assume that a slow app probably leaves them less informed.

Improving performance might allow you to:

Handle more customer service calls
Require less staff
Require less hardware
Keep users engaged and productive

If internal users want to leave the site, but can't, maybe they'll just go out for coffee instead, or get a new job!

RailsPerformance.com

2011-10-18T14:54:00.003-05:00

I was finally able to acquire the domain 'RailsPerformance.com', which had formerly pointed to http://yehudakatz.com/.

Along with the new domain, I am focusing my efforts exclusively on Ruby on Rails application development and performance tuning. This site will continue to be a solid resource for Performance related content, as well as a way to contact me for Development services, Application Review, or a Performance analysis.

If you'd like to get in touch, please email me at Railsperformance@gmail.com

Improving application Performance by tracking user Workflow

2011-10-11T11:08:00.000-05:00

Where do I begin?

While it's a common mantra that you don't want to prematurely optimize your application, that doesn't mean that you shouldn't measure where things are at or take the time to know what your current numbers are.

If you aren't measuring, how do you know if you've improved anything?

Monitor your server Log files
A good place to start is to regularly review your log files and get familiar with the average request times. Track the most frequent requests, the slowest requests, and errors, as a baseline for comparison. As you roll out new features and add more users, keep an eye changes to your base numbers. When used in conjunction with page analytics tools like Google Analytics, you can track the relationship between the performance of pages, and your site's goals. You also want to get a sense of what a typical 'session' looks like for your users. How long are they on? What actions do they take? Is there a peak time of day or day of the week that gets the most action? You aren't just looking for the slowest requests, but for all the requests it took for the user to get what they needed.

Track your overall page load time
In addition to server performance and page analytics, its important to have a sense of how long a page takes to load, including images, stylesheets, and javascript. While the server might only take 200ms to return your 'Sign up now!' page, the user might have to wait a full 12sec before all the images and content is fully loaded...and most people won't wait that long! Taking a moment to sketch out the page flow of a new user can help you understand which pages to optimize, and which files can be cached to improve the feel of the app, and the same can be done for returning users.

Ask users about the speed
Most of the time your users will tell you when things are extremely slow, but by then they are already frustrated. Its worth the effort to establish a good relationship with a few 'user advocates' that represent the main user types you have, in order to survey them on new features, ask them about any issues, and basically keep tabs on the pulse of your app. While errors in functionality tend to prevent users from completing a transaction, performance issues prevent them from completing MORE transactions. Ask your users what parts of the application could benefit from being faster, and what that would mean to the user. "If I could see the results and narrow/filter by these core criteria, and see user ratings without leaving the search results page, I could buy more clothes for my daughter on my lunch break"

Optimize the user's workflow
Some performance improvements come straight from a development change, like querying the database more efficiently, loading less data, or caching common results, but there are often significant improvements to be made by optimizing the way the users use the application, by improving the design.

For example:
A common pattern I've seen is to have a simple search and results page which allows users to enter criteria and see the results in a grid. Common features like sorting, filtering, and pagination make the process of finding results easier, and users have come to expect that behavior. The nature of the search input, and how you allow the users to specify criteria, has an impact on how the queries perform, and how easy it is to tune or cache them. At a certain point in a feature like this, users start to complain, and someone has to 'take a look at why the search is getting so slow'. The queries are reviewed, changes are made, a few seconds are shaved off, and that's usually where it ends, but there's more to it than that.

What are you looking for?
When a developer tests a search/results page for performance, they would tend to try a few different criteria, use a wide date range, and click around the sorting and pagination links, but for a real user, the flow doesn't end there. Most of the time that I've seen users complaining about a slow search page, its not because of the initial search, but because of how they use it. A user might fire off a search view the results, sort/filter, then click to see the full details of one record, go back to the results page (which might fire off the full query again), look at a few more individual records, and continually bounce back and forth between the details and results list. To this user, the search might seem slow, not because of how it performs initially, but because of how long it is taking them to find what they are looking for, and the process of going back and forth. Asking them how they use it, and actually watching what they do, you will see opportunities to improve the design and match their flow.

A few common search workflows I have seen, each resulting in different interface features to make things easier:

Collect and Compare: A user may issue different queries to find specific items and compare them
"I'm looking to see if there is a difference for this client at their midwest branch vs. east coast"
Design: Allow the user to 'keep' or 'favorite' particular records, view them in sequence or side by side, or save them in a named 'group' for later review.

Filter and exclude: A user may start broad, but continue to narrow down to a set of records. They may want to 'remove' specific records from the results.
"I need to find all of the accounts that are past due in this region, but not the 3 that are managed by Mary"
Design: Allow a user to 'remove' results from the query, or at least have them hidden from the display.

Routine Checkup: A user may fire the same search at regular intervals to see if anything has changed, top 5 results, list of work to do, etc.
"The rates are updated twice a day, so I come in and fire off my search to see who's in the top 10 in the morning and in the afternoon"
Design: Allow a user to save a query like this, and have it become part of a dashboard or notification email.

Alert! : A user may run a query once a week to look for an infrequent but important business issue.
"I have to check for duplicate orders."
"I have to fax orders with a coupon code to corporate because we can't do split billing yet"
Design: This could be done as an order validation routine. Don't allow orders of this type, or send out the correct email notification immediately upon creation, anything you can do to codify the business logic and allow the user to focus on higher value work.

Sometimes it's the little things
One good way to understand what your user's are doing with your search, without having to ask each one of them, is to allow them to save a search, give it a name, a brief description, and allow them to share it with others of equal permission/role. Letting them save their search will go a long way towards making the app seem faster, as they no longer need to fill in the search criteria, or re-run the search when they forget a checkbox. You may also discover, by analyzing the saved searches, that there is a pattern or grouping of searches that could be improved or cached. You may discover a category of searches that several users are interested in, and instead of having them all log in, fumble through the search input process, and all peg the server at the same time on monday morning, you could run that query once, store the results, show it on the dashboard, fire off the daily emails, and lower the amount of work the system is doing, while increasing its value to the users.

Whenever I do a performance analysis, my workflow tends to be:

Reviewing the log files
Analyzing web page performance
Analyzing application setup and performance
Reviewing the Database setup and tuning the slow queries
Reviewing the performance of the loaded page (javascript, ajax, rendering, etc)
* Looking for design changes that will lead to a faster user experience *

So try to sketch out the full workflow of your users, and look for ways to simplify the steps they take, have the system do more work for them, and let them focus on doing MORE of what you want them to do.

Quick Script: Freshbooks sum of total hours

2011-10-05T11:11:00.000-05:00

I like freshbooks, but for some reason they don't show the total number of hours at the bottom of the invoice, which has been a complaint for awhile.

Here is a quick script I wrote that returns the total for billable and nonbillable hours. Its not perfect, and its likely to break if they change the structure of the page, but its a start.

//take this script and place it in your firebug or javascript console (when you are on the invoice edit page)
//it will segment billable and non-billable (0.00) and total them up (could be improved to group by rate)

var total=0;
function add_them(index, e){
  var num = parseFloat($(e).val());
  if (!isNaN(num)) {
   total += num; 
  }
}

//find trs with td and a non-zero or empty hourly rate
var billable = $('tr').has('td.editable').has('input[name="unit_cost[]"][value!="0.00"][value!=""]');
var total=0; //zero out reusable variable
//within that group, go find the hours and total them up
billable.find('input[name="qty[]"]').each(add_them);
var bill_total = total; //copy value before running the next calc

//find trs with td and a zero value hourly rate
var non_billable = $('tr').has('td.editable').has('input[name="unit_cost[]"][value="0.00"]');
var total=0; //zero out reusable variable
non_billable.find('input[name="qty[]"]').each(add_them);
var nonbill_total = total;

//print it out if console is available
if(this.console){
console.log('Billable hours total: ' + bill_total);
console.log('Non-Billable hours total: ' + nonbill_total);
}

Improving Web Page Performance WindyCityRails 2011

2011-09-17T16:46:00.000-05:00

WindyCityRails 2011

This has been a great conference once again! Its so cool to see how the conference has grown over the last 4 years, and how the Chicago Rails community has grown.

I had a lot of fun giving my talk, and got good feedback on SpeakerRate. I was intimidated speaking after Dr. Nic, and before Tom Preston-Werner from Github, those are some heavyweight Rubyists!

Here are my slides, please let me know if you have any questions, I will answer them if I can. (there's always something that I wish I could have covered more)

Windycityrails page performance

View more presentations from John McCaffrey.

IRB tips and Tricks

2011-08-13T13:29:00.006-05:00

I love me some IRB!

Its such a great tool and something I have open constantly.

I use it for:

Quick syntax checking
File scripting
Regular expression
Checking the 'state of things' in production or staging environments

One small change I should point out is that in my ~/.irbrc, I have this little piece that will load another .irbrc file if it is present in the project, so I can add additional methods to the project, given its ruby/rvm/gems setup.

    load '.irbrc' if File.exists?('.irbrc')

Irb Tips and Tricks

View more presentations from John McCaffrey.

Tons of good Resources:

13 Great Application Performance tuning blogs to follow

2011-08-03T10:53:00.000-05:00

Here's a list of the Performance blogs that I follow:

I put them all together in a bundle, so if you subscribe to this, you'll get new feeds when I add them.
http://bit.ly/perf-feeds

Railsconf 2011 notes, pictures, and review

2011-05-24T12:52:00.004-05:00

RailsConf 2011: Simply Amazing!

Last year was my first Railsconf, and while I learned a ton, I think I made a lot of 'first timer' conference mistakes like:

Not reading up on the sessions, presenters, recent developments and blogs in advance
Not maximizing my networking time (ie. spent too much time with people I already know)
Not leaving a session that wasn't a good fit because I felt awkward about getting up
Not having enough business cards
Not having recent blog or twitter posts to encourage someone to follow me
Not staying out late and meeting more people
Not getting up early and getting to the keynotes on time

So this year I vowed to do a better job, and other than the conflict between staying out late, and getting up early, I felt I tackled all of my concerns pretty well.

I tried to push myself to be more outgoing and less socially awkward than I normally am. Which in some cases meant I just had to walk up to a group of people and ask what they were chatting about, what they found interesting so far, or what they were here to learn. For the first 4mins of any of these 'interrupting intros' it would be rough. I wish I could take a picture of it, the crinkled nose, furrowed brow, 'who the hell are you?' look. I'm very sensitive to facial gestures, and have a hard time ignoring the wealth of feedback I get from their physical output, but I've learned that you just need to get past it, otherwise you'll be sitting in the corner by yourself.

I was very friendly. Gregarious, but not obnoxious, and did my best to add something immediately relevant to whatever they were discussing, but still I got these mean scowls for the first few minutes. I pressed on, ignoring the scowls, determined to make a meaningful connection, and I'm happy to say that it really paid off.

About 10-15mins into almost every one of these conversations we'd get into something interesting and I'd share some advice, perhaps some sites or a blog post they should check out, and then I'd get asked for my business card. I think I gave out 30 business cards to people that directly asked for them, which is more than I've ever done before. And I've already been contacted by 2 of them for potential projects. (I'm not trying to brag, just trying to say that its worth pushing yourself to be more social, walk up to another group and just start a conversation. They are Rails developers after all, you certainly have something you can chat about)

Sunday:

Found my rails peeps at Pratt St. Alehouse

Closed the place down chatting with Steve Smith from OrderedList

Monday:

While in the Rails 3 upgrade, I actually upgraded an existing app from 2.3.5 to 2.3.11, and am finishing up the 3.0.7 upgrade now

I got to chat with some of the team from Heroku at the booth, and gushed about how much I love it. I also pointed them to a great screencast that shows how to monitor heroku under load, and scale up more dynos to get better response time. They added me to the Heroku Beta program, so I can start testing out new features. (but won't be able to blog about them initially)

The Rails best practices tutorial was awesome, and I recommend everyone should go to codeschool.com and sign up for Rails for Zombies and Rails Best Practices. Its very cool, and I expect to see a lot more courses coming out of it. Pure awesomeness! (disclosure: they have a referral program, so I think I get some kind of codeschool dollars or something...I mostly just want to see the tracking part)

Went to Ignite rails conf (gave my extra ticket to Ryan Bates of Railscasts.com...a small token of thanks for all the stuff I've learned from him). The talks were great, and showed a side of our community that goes beyond technical concerns and is passionate about giving, teaching, leadership, and making a difference.

Great party given by EngineYard at Max's Taphouse. Met some really cool people, closed the place down, and grabbed a slice of decent baltimore pizza at 2 in the morning.

Tuesday:

I attended

Credit card processing talk by Living Social

Spree

Geospace your rails app

25 deployment tips

Didn't stay out late, as I wanted to finish up my lightning talk, and also be fresh at least one morning.

Wednesday:

Inside groupon

scaling with friends from zynga

Scalable servers with Event Machine by David Troy

I gave a lightning talk on 'Freelancing on Rails' which seemed pretty well received, even though the majority of the lightning talks were more code and tech focused

Based on the feedback from my lightning talk, I organized a 'Birds of a feather' roundtable discussion on freelancing and managing side projects. It was great because there were people with varying levels of experience and client types, some coming from java/.net, some that do a side project here and there, and others that don't have a full time job, just a full plate of side projects.

I referenced the very interesting video called "F@ck you, pay me", by Mike Monteiro, which lays out some good rules for freelancers.

I then stayed out way too late after the github party, as you can see in the pictures below. (and I regret that I didn't at least drop in on the music jam session. that was a mistake)

Thursday:

missed the first session

Toolkit choices

Emrubyconf (all event machine stuff)

It was a great conference. Very hard to go back to work, and use old stuff when I want to try out all this new stuff, but I love to see how things are moving forward in this space. The Rails community is awesome and I'm so glad to be a part of it.

Here are the pictures I took this year:

What I've learned: Freelancing and side projects on Rails

2011-05-18T16:09:00.001-05:00

I had a blast giving this talk in a short period of time at the Railsconf 2011, along with 16 other people that had to fly through their talk in 5min.

There is a survey of rate info that the attendees filled out.

Freelancing and side-projects on Rails

View more presentations from John McCaffrey

Fixing Rails error format '{{attribute}} {{message}}' caused by i18n and Rubymine RVM issue

2011-05-13T14:07:00.001-05:00

File > Settings > Ruby SDK and Gems

I've been using RubyMine as my main Rails editor since the minute Netbeans discontinued their support of Rails, and I've got to say I've been loving it. I knew that JetBrains made good editors, but I didn't know it would be this good. Its been worth every penny, as it helps me get up to speed on a project faster, and its rvm integration is crucial for ensuring I'm running in exactly the same env as the command line (netbeans did not play with rvm very well)

I recently ran into an issue when running a rails 2.3.5 project. I started getting active record error messages returned in the format "{{count}} errors prohibited this {{model}} from being saved", and "{{attribute}} {{message}}". I had seen this kind of thing before, and figured I must have installed some Rails 3 gems into this rvm (I had just started up a new side project on Rails 3, and must not have been paying attention to which terminal window I was in).

Sure enough there were a few Rail 3 gems, and i18n 0.5.0 in particular was reported to be the main culprit. I removed that version of the i18n gem, fired up the server from the command line, and verified that my error messages were displaying correctly.

But when I launched the app from Rubymine, it was still loading i18n 0.5.0. A little poking around later, and I think that what happened is that I had this project in my default ree 1.8.7 gemset, and ree-1.8.7 was available as the sdk in rubymine, but once I added a new gemset to ree, rubymine no longer displayed the default, or "rubymine loses default rvm gemset" once a named gemset is added.

To fix this, it seemed like I just needed a named gemset for these gems, but I didn't want to install them all again, I just wanted to copy them. According to the documentation, it shouldn't be hard, but I wasn't sure about the default unnamed gemset, because even when calling 'rvm gemset list' it wouldn't show that there was a default, or that I was currently using it.

Here's what worked for me:
rvm gemset copy ree-1.8.7-2010.02 ree-1.8.7-2010.02@proj_gemset

When I hopped back into Rubymine, I saw that ree with that gemset was now available, and set it as the default for that project. (which is available from File > Settings > Ruby SDK and Gems. Which took me forever to find)

Rails Captcha plugins

2011-03-18T15:31:00.064-05:00

Captcha on Rails

Image via Wikipedia

I needed to implement captcha to protect a 'forgot password?' form on a financial site this week, and I went through the various options to see what worked.

The app I'm on is rails 2.3.5, so that's what my testing was under.

When evaluating plugins and tools, I find it easier to start with a clean project, and try it out without any interference from the peculiarities of a full blown app. This allows me to follow the steps in cookie-cutter fashion, see how it works, and push a demo app up to heroku so that I can get some feedback from my clients. Its almost always faster for me to go through the tutorial, follow the basic steps, and push an app up to heroku, than it is for me to implement the tool in my real app, add appropriate tests, wait for the build, deploy to a test env, etc. Deploying to heroku allows me to get the proposed feature in the hands of my clients very quickly, and that feedback can save me a ton of time (particularly when I find out that there are some additional constraints that require me to change direction)

My requirements were:
"add a captcha to the forgot password page"

I looked on ruby-toolbox to see what plugins were being used and had good traction, and narrowed it to these three:

Simple-Captcha
http://expressica.com/simple_captcha

requires rmagick to create the image
requires db:migrate to persist the keyed answer
works well, easy enough to change

reCaptcha (sponsored by google)
http://www.google.com/recaptcha

commonly used, well tested, looks professional
requires outbound http call, (may have to deal with proxies)
can handle ssl
can play audio if text is too hard
expects String.html_safe (rails > 2.3.7), had to monkey patch a do nothing method
has styling options, but I had problems getting it to look right in the form I was using

BrainBuster
https://github.com/rsanheim/brain_buster

doesn't make 3rd party outbound call
doesn't require rmagick
does require a db table to store a set of questions
presents the user with a set of logic questions
by default uses cookies to flag that a user has answered successfully, but that can be turned off to always challenge the user
handles numeric to text conversion when a user answers 'five' or '5'

Getting started
I didn't think my client would want the 3rd party outbound call that reCaptcha uses, so I
started with simple-captcha, and created a quick demo on heroku (15mins reading the docs/code/tests/blogs, 15mins create the app and deploy)

script/plugin install svn://rubyforge.org/var/svn/expressica/plugins/simple_captcha 
  rake simple_captcha:setup  
  rake db:migrate

 #add to routes
   map.simple_captcha '/simple_captcha/:action', :controller => 'simple_captcha'
   
 #add to application_controller
   include SimpleCaptcha::ControllerHelpers

The client checked it out and got back to me that it looked good, and to go ahead and implement it in the app.

Getting it to work in our actual app took longer than I expected.
First, we are using rails_xss, to escape output to the page, and there was an issue with how helpers are registered as 'safe' (but only in development, where classes are reloaded).
A few 'Stack level too deep' and 'segmentation fault's later, and I saw where the problem was. (I needed to register this helper in an initializer, whereas my application helpers are registered in a way that handles the class reloading in development)

The next issue was getting the cucumber tests to pass.
The key was that cucumber has access to the session, and could look up the captcha value via

captcha = SimpleCaptchaData.get_data(session[:simple_captcha]).try(:value)
  And I fill in "#{captcha}" for "captcha"

I was surprised that none of my functional tests were failing, but it turns out that the verification method is set to return true when running in the test env, so I added a few tests to verify the permutations that we don't cover in cucumber.

I checked everything in, the build passed, and I was back to whatever I was doing before that landed in my lap...except, I wasn't quite finished. Turns out that rmagick/imagemagick is a pain to get working on windows, and several of the developers on the team are on windows, so it was time to look at another solution.

I did get a chance to use the git revert command

git revert 'hash of what you want to get rid of'

After some discussion with the team, it seemed like reCaptcha was the next logical tool to try, as its more well known, fully tested, and has the least requirements on the system. (doesn't even require a db table)

The implementation was very quick (though there was an issue with String.html_safe), pushed out an update to heroku to get client feedback, and was working through some slight styling issues as I went into the morning standup.

script/plugin install git://github.com/ambethia/recaptcha.git

# in config/initializers/recaptcha.rb
  
  #monkey patch to fix html_safe method
  class String
    def html_safe
      self
    end
  end

  # get an api key (free) from http://www.google.com/recaptcha
  ENV['RECAPTCHA_PUBLIC_KEY'] = 'youractualpublickey'
  ENV['RECAPTCHA_PRIVATE_KEY'] = 'youractualprivatekey'

# in the view, inside your form

<%= recaptcha_tags %>

# in the controller
 def update
    @user = User.find(params[:id])
    if verify_recaptcha
      if @user.update_attributes(params[:user])
        flash[:notice] = 'User was successfully updated.'
        redirect_to(@user) 
      else
        render :action => "edit"
      end
    else  
      flash[:error] = 'Captcha no good' 
      render :action => "edit"
    end
  end

But as I was mentioning in the standup that reCaptcha makes an outbound call, the team lead recalled that after a recent security review they were cutting back on anything making an outbound call, if possible.

Which brings me to BrainBuster.
It doesn't require rmagick, or make an outbound call, the styling is simple, and it seems to work well.

script/plugin install git://github.com/rsanheim/brain_buster.git
   script/generate brain_buster_migration
   rake db:migrate
 
 # in controller
   before_filter :create_brain_buster, :only => [:edit]
   before_filter :validate_brain_buster, :only => [:update]

 #method for handling the failure
  def render_or_redirect_for_captcha_failure
    render :action => "show"
  end
 
 #in view (path to captcha view in plugins dir is added to controller for you)
  <%= render :partial => '/captcha' %>

I did run into a few things that I needed to change from the default, namely the fact that once you have passed the captcha, a cookie would be set that would lead to that user never being challenged again, which is not what I wanted, but was easy to address.

There is also an issue in the fact that it implements the verification as a before filter which wasn't working for me as the page I'm working on was doing double duty as view and edit, so I stopped using the filters, and make the calls from within my action.

Overall, its great that there are a lot of options for implementing captcha in a rails app. Depending on your requirements, you should be able to find something that works, and hopefully this post might make the decision process a little easier.

Let me know how it works out for you.

Here is the heroku deployment that I created to demo these plugins http://simple-captcha-demo.heroku.com/

I pushed up all the code to github, so you can try it out for yourself.

In the process of pushing up to github, I realized I didn't want my reCaptcha api key to be in the commit history, so after reading this, I ran:

#get rid of the file
  git rm config/initializers/recaptcha.rb
  
  #go through history and rewrite
   git filter-branch --index-filter 'git rm --cached --ignore-unmatch config/initializers/recaptcha.rb' HEAD

  #now push up the rewritten history (dry-run first to see what might go wrong)
   git push --force --verbose --dry-run
   git push --force

  # and then within heroku command line
  heroku config:add RECAPTCHA_PUBLIC_KEY=my_public_key RECAPTCHA_PRIVATE_KEY=my_private_key

Thanks,
-John

Good to know: week #1 (rails logging, postgres iLIKE, rubular.com, and more)

2010-12-14T16:13:00.006-06:00

Image via Wikipedia

I'm not blogging enough. I have a lot of ideas for what I'd like to blog about, but I somehow build up each idea into a series of steps that would take 1-2hrs to complete, and I don't have 2hrs to spare at the moment.

I'm going to try and just write more frequently, and lower the bar on what I'm writing about. I may not blow your mind with some completely unheard of performance tidbit, but hopefully I'll pique your interest and maybe even start a discussion.

So here's a breakdown of stuff I probably should have known, or maybe I did and just forgot, but when I saw it this week, I said "Well, that's good to know".

Rails tips

When invoking rails code in Production, through a rake task, I wasn't getting any logging info. Turns out Rails doesn't autoflush the log in production, so I had to add 'Rails.logger.flush' at the end of my rake task, and voila I could see why a task was failing in prod. (permissions issue, of course!)
(0..5).to_a will give you an array with [0,1,2,3,4,5] (helpful when playing around in irb and you need an array to test with)
how to get a date in a file name from the command line: "/path/to/my_file`date +\%Y\%m\%d`.txt"
It can be handy to use factory_girl in script/console to easily create some test objects to play with, or seed your dev env for testing.
iLIKE john on postgres:

if you do something like this "where upper(first_name) like 'john%'", you won't be using an index on first_name, unless you also created it using upper(first_name), but you can instead just use "first_name iLIKE 'john%'" to use a case insensitive match, and take advantage of your index.
I also came across some code that was doing where upper(ssn) like '%123456789%'
which is not going to use an index because of the upper, and is going to be slower than it needs to be because when there are 9 digits in an ssn, there is no need to do a 'like'

the query I was working on went from 4500ms, to 1.7ms after a simple change to the query. (yes there was other stuff going on)

I Got an ArgumentError (malformed format string - %') when putting a '%' in my like condition, instead of { :conditions => [condition] }, just remove the [ ] and it won't try to format the '%'.
I was copying a bunch of records from a heroku proof of concept app, down to a local copy to use as test data. This query: "insert into other_db.orders_table (select * from test_db.orders_table)" will copy all the data from one table to another, but the columns have to be in the exact same order (duh).
I was doing some page performance improvements, moving the js to the bottom, combining the js files via Rails :cache, and testing in prod mode, to see that the files are combined and everything was working fine. I started with the 2 easiest files, tested it, everything looked good, then added the last file and refreshed the browser to make sure it was picked up. I was about to check in when I saw some duplication that could be cleaned up. I moved some functions around, and cut out a ton of duplicate code from 3 pages, all the while verifying that my highcharts charts were being rendered properly. I noticed a misspelling I had made and was surprised that it didn't cause an error. Then it dawned on me, I was still running locally as 'production' env, and had been using the old file the whole time, and my refactoring of the javascript wasn't being picked up at all! I switched back to dev mode, saw a bunch of errors and then fixed them. Before checking in, I decided to try prod mode again, and make sure the combined files were working. THEY WEREN'T!! A few missing ';' made the javascript file not syntactically correct when it was combined with other files. (hmm, smells like lack of js tests around here)

Interesting sites (Rails, development, misc):

Ruby5 podcast always great stuff

heroku purchased by salesforce I hope this turns out to be a good thing

Groupon says 'No deal' to Google and what it says about Google, and Groupon

http://www.simonecarletti.com/blog/2010/07/the-way-to-rails-3/ its not as hard as it seems to be ready for Rails 3

rubular.com ruby regular expressions tester in your browser, faster and more informative than irb

http://www.arailsdemo.com/ great site that shows how to build up a rails app step by step. I was looking at it to see if there is any quick way to add acts_as_commentable and if there are any updates to that. (seems not much has changed)

http://skribit.com/ - blog plugin that lets people tell you what to write about. Exactly what I need! (please suggest a topic)

Entertainment

fancypants adventure - fun game as google chrome app, shows some nice physics and game play. Works offline.

big buck bunny - very cool mini-movie showing off what the blender engine can do. (the bunny totally reminds me of Chris Farley)

http://www.google.com/nexus/ New Android phone

mashupbreakdown.com 'girl talk' all day music with a visual breakdown of what each sample is

Performance tuning workflow

2010-12-08T16:47:00.001-06:00

Its hard to know where to start when it comes to speeding up your web app, but here's what I've found to work best for me.

Measure twice, cut once.

Its been said, "You can't improve what you can't measure" (Lord Kelvin...who knew a thing or two about measurement), and that's very true for performance tuning. Actually, I've recently run into a few things that I was absolutely positive would make things faster, but in looking at the numbers, it was clear that things were getting worse.

Start with yslow, or page speed in your firefox browser. Measure your main page, even if its simple, and also go through your top 5 pages. Make note of your initial scores, so you can keep track of any improvement, but also make note of the recommendations and timings. In most cases you'll have front-end issues that need to be resolved.

Looking at the waterfall diagram I'd see that the amount of time spent to pull down the html is actually pretty small, and the majority of the time was spent on everything after (js, images, css, ajax, etc).

So we focus on the things that give us the biggest gain:

Make sure keep-alive is on
Turn Gzip on
Combine & minify js files
Combine css files
Optimize images, sprite the ones that make sense

The next round of changes are a bit more complex, but worth it:

Leverage browser caching, set far future expires, and have a strategy for updating the url (which you get for free with Rails)
Either move the js files to the bottom, or use a 'deferred loading' strategy yui, lab.js, head.js, and others
Remove unused code: find unused js and css and get rid of it

At this point I normally see a decent improvement in the rendering, the yslow/page speed scores improve, and things feel a little bit faster.

Looking at a waterfall diagram, I should see less requests (particularly after they've been cached), much less data is downloaded, and the initial html download is a more significant percentage of the overall response time.

From here I'd start profiling the backend:

Use a simple tool like apache bench or httperf to just hit the url and give me some baseline numbers (how many requests can the server handle at a time, and how long does a single response usually take)
Analyze the logs to see which requests are most common, and which requests take the longest (we want to optimize the sweet spot of the most commonly used and slow requests), and confirm that all the numbers are making sense (my initial yslow times, the logs, and how it 'feel's when loading it).
Then I'd pick a few of the requests and see where they spend their time

My approach is usually:

Run rack::bug, or other tools that will reveal the memory footprint, object creation, queries executed, and time spent in rendering.
Try to narrow the slowness down to the main culprit, either using some debugging breakpoints, log statements, or the ruby benchmark utility.

At this point there tends to be a few main culprits:

A single slow query
Lots of object creation
Lots of queries
Cpu intensive work
A long running call to a 3rd party (email, report generation, etc)

The majority of slow queries I encounter when doing performance assessments are actually pretty straight forward to fix. Its usually easy to see what data was expected, and figure out ways to rewrite the query to be faster and still get the right data back.

A lot of poor performing queries are due to lack of indexes, or poorly defined indexes. (even today I fixed a query that was doing 'upper' on a column and was therefore not using the index. The query went from 4500ms to 1.6ms because it no longer needed to scan the whole table)

Beyond indexes, the next problem tends to be loading more data than is needed, followed by loading objects and traversing/filtering in code what could have been done much faster in the database.

After cleaning up the low-hanging fruit, I start to look at more interesting issues:

..which I'll get into next time!

Rails Performance tuning Presentation from WindyCityRails Conference

2010-09-11T15:10:00.004-05:00

Slides

Windy cityrails performance_tuning

View more presentations from John McCaffrey.

Here is the video from the conference

Mysql Performance analysis Improve performance by removing unused indexes from the database

2010-09-10T16:19:00.005-05:00

After focusing on improving the initial page load of an application, the next most critical place to look is at the database, particularly checking the explain plan of the queries, and ensuring that the proper Database Indexes are in place. One thing that might not always show up is when you have indexes that aren't used. They certainly have a cost at insert, and they cause a lookup cost when the query planner is deciding which index to use, but it might not always be obvious.

A key component to how the query planner chooses the index is based on the 'selectivity' of the index. The selectivity is the ratio of distinct values against the total values of the column. The more unique values that are present, the more helpful it is to have an index. Which is also why you would probably not benefit from creating an index on a boolean, or similar elements with poor selectivity**.

This reveals a key point about deciding what should be indexed:

Unique lookup values (foreign keys, ssn, phone, email)
Join values
Commonly searched values (** it would make sense to add an index on a boolean, in order for it to be picked up as a covering index)

And what might not work out so well:

Columns with low selectivity
Large data types
Any columns on tables with only a few entries

Here is a query I just tried out on my mysql db, to see what the selectivity of the indexes are:

The results are specific to the database you ran it against, and the distribution of data within it, so don't make any assumptions or extrapolations based on this data, its merely a way to quickly order and test the indexes that are available, against the data. In fact over at the mysqlperformanceblog, they give a great example of when this technique would actually lead you astray.

To really figure out which indexes should be removed, you have to see which ones are actually being used. Which I'll have to cover in another post.

I think I picked up this query from these slides by Jay Pipes

Ruby On Rails Development Interview: Freelance, Contract and Remote development

2010-09-09T23:56:00.002-05:00

I was interviewed at RailsConf this year, by Avdi Grimm of WideTeams.org, regarding my experience with remote development, Agile Development, effective tools for remote collaboration, and how to be a successful freelance developer.

After the interview I thought I was really rambling, but upon hearing it again, its not as bad as I thought.

http://wideteams.com/interviews/146/

vim notes

2010-08-27T16:38:00.002-05:00

I decided to just throw out a copy of all of the notes I have on Vim, some of these might not be right, as I'm still learning, but maybe you'll find something useful.
vim notes

vi:

basic use

[esc] is the escape key

vimtutor : starts vim editing a copy of a tutorial file -- very good.

i : insert mode (so you can type)

[esc] : so you can navigate and edit commands (stop typing) (stop selecting)

h j k l : move cursor (left, down, up, right)

A : append at end of line (you can't navigate past the next to last character)

u : undo last command

x : delete character under cursor

dd : delete line (also puts it into the clipboard like buffer) (d is delete)

p : pastes what was in the buffer

:wq : write and quit

:w filename : Save a copy of the file you are editing as filename

:q! : quit without saving

:help : display help

[tab] : use tab completion to check what your command is

COPY PASTE (for CUT use d as described above)

v : visual mode -- use to select text

y : use to yank (copy) what was selected above

l w e $ : these are special movements, letter, word, end of word, end of line

notes from going through the vimtutor tutorial (just type 'vimtutor' on the command line)

A to append

d(motion)

dw to delete word

d$ to delete to the end of the line

d^ to delete to the beginng

motions work without a modifier, so just w, e $ and ^ jump around

dd to remove a line , but stores it in the register

2dd will remove 2 lines

u to undo the last command

ctrl-r will redo the last commands

use dd to delete a line, and p to paste it (put cursor above where you want it to go)

r will let you replace the char under the cursor

c is for change (ce, cw, c$)

ctrl-G to find out what line you are on

or :set number

type the number then G to go to that line

there is a matching parenthesis search

ctrl-D to complete a command after typing :

:help or F1 to get to help

ctrl-w ctrl-w to toggle windows

move your cursor to the topic: |quickref| and press ctrl-] to jump into it

ctrl-o to jump back

:Explore to bring up the directory listing

enter to go down a dir

- or ctrl-o to go back

ctrl-o will also jump you back from a directory listing and out of a file

s in directory listing to sort by time, size, name

r will reverse the sort

i will bring in the time and size columns

Modes:

normal, insert and visual, there are others too

[esc] (C-[) takes you back to normal

Enter a number before a command to repeat it, examples:

10w : skip forward 10 words

10dd : delete 10 lines

Useful tips

:set ignorecase : you nearly always want this

:syntax on : colour syntax in Perl,HTML,PHP etc

% : match brackets {}[]()

~ : invert case (upper->lower; lower->upper) of current character

= : (re)indent the text on the current line or on the area selected

Substitution

:%s/fred/joe/igc : general substitute command replace fred with joe, ignorecase

:%s/\r//g : delete DOS Carriage Returns (^M)

:s/$.*$:$.*$/\2 : \1/ : reverse fields separated by : notice the escape for \(

:%s/^$.*$\n\1/\1$/ : delete duplicate lines (can vi sort stuff too?)

global

:g/^\s*$/d : delete all blank lines

:g!/^dd/d : delete lines not containing string

:v/^dd/d : delete lines not containing string

:g/fred/,/joe/d : not line based

:v/./.,/./-1join : compress empty lines

matching

/fred\+/ : matches fred/freddy but not free. notice escaping the \+ (match 1 or more).

/codes$\n\|\s$*where : normal regexp with escaping

/\vcodes(\n|\s)*where : \v for very magic, which cuts down on the escaping

:292 to jump to line 292

:set number to turn line numbers on

Y to yank a line

P to paste at cursor

p to paste below

2Y is 2 lines

y$ is to end of line

dd to delete a line

d3 to delete 3

:set invlist to see the hidden characters

:set ff to see the current os file type

:set ff=unix to convert it to unix, dos, etc

:set paste to paste without getting extra tabs (remember to set lf line endings if copying from windows)

:%s/\r/\r/g To replace every CR with LF (when searching, \r matches CR, but when replacing, \r inserts LF):

:%s/\r$// To delete ^M only when it occurs at the end of a line:

to convert newlines in several files

:args **\*

maybe it should be **/* on windows

:argdo se fileformat=dos|up

LoneStarRuby Conference notes and photos

2010-08-27T13:44:00.005-05:00

General feel

Great people
Organizers are loose and let things just flow
Sponsors are not interfering with the feel of the crowd

Glenn VanderBurg

Amazing presentation, should be required for any software student

Jim Remsik and Robert Pitts: Vim

I really like these presenters, and even got a chance to spend some time with them during the conference. (In discussing their presentation, they both said they were not happy with how it turned out, and we kicked around ideas for how it could have gone better. My notes here are just a summary, not a criticism.)
A late start changed the feel of the presentation, creating a bit of tension and a hurried feeling
Good info, but I think a 'top 5 things you need to know about vim' summary might have helped
I felt like the crowd may not have walked away with the basic knowledge the presenters were hoping for. (A good lesson for me about not trying to cram in too much stuff in a presentation, and be ready to roll with it if the schedule changes)

My additions

There is a lot of material to cover in a talk like this, and I personally really struggle with finding a good balance. It seems some of the best presentations are the ones that focus on just a few core concepts, get the audience excited, and then give them the resources to take the next step. Here are my thoughts on how to take someone at the very intro stage, and give them a place to get started.

The VimCasts series is a great way to get an intro, as well as how to see specific problems solved in vim
Type 'vimtutor' at the command line and go through the tutorial
Use the interactive editor gem to expose vi to IRB (vimcast explaining the details)
All my notes on Vim, as I've learned the basics
Even if you aren't ready to go 'all in' with vi, the main point is to be efficient with the tools you use. Avoid taking your hands off the keyboard with any IDE, or tool. Learn to type without looking at your fingers, agree within your team on the tools that you will use together.

Gregg Pollack: Deciphering Yehuda

Absolutely amazing presentation, anyone that cares about Rails3, and solid ruby development should watch this
Anyone interested in how to put together a dense yet flowing technical presentation should watch this.

Pictures:

Improving Rails Application Performance (ChicagoRuby Presentation)

2010-07-21T13:01:00.002-05:00

It was a good session at the suburban ChicagoRuby meetup on Saturday. The smaller crowd size allows for more discussion and exploring the areas that people are really interested in, instead of just sticking to the slides.

I was glad to find out that most people were pretty familiar with Firebug, but surprised that more people weren't aware of yslow, or the performance tips its based on. (I didn't expect many people to know about Google Page Speed, or Speed tracer)

We took some time to discuss particular issues, like how to decide where to spend your performance time, and a review of the workflow I use when doing an assessment. (because we took the time to go over that, we then skipped over some of the tools)

It was also announced that I will be presenting a session titled "Analyzing and Improving the Performance of your Rails Application" at this year's WindyCityRails conference!

This makes the 3rd year that I've spoken at WindyCityRails (2008: 'Advanced Firebug, and javascript unit testing', 2009: 'Rails PDF Generation')

improving the performance of Rails web Applications

View more presentations from John McCaffrey.

tweaks to rails_indexes plugin to sort by table name and handle errors more gracefully

2010-06-24T17:36:00.002-05:00

One of the first things I do when working on a new project is to install the rails_indexes plugin, and look for missing indexes. Now I don't just take the plugin at its word and create the suggested migration, as it will tell you to add indexes on more stuff than you need (you don't need an index on your primary key as mysql and postgres will do that already), but I do look at the suggestions to learn about the model, where the joins are, and then see if there is an opportunity for improvement. I also use the railroad plugin to model the domain and see what I'm working with)

Today I was using the Rails_indexes plugin on a large project and it blew up with some errors. It turns out there was a bug for how it handled non-model classes, which a few people have patched, but they did it in some specific ways (like only detecting a controller) that I wasn't sure was going to work for me, so I just catch the error, log it and keep on truckin.

The next issue I ran into was that the recommended migration file was huge for this project (tons of models, no indexes!), and it was hard to find what I was looking for, so I changed the code to sort the indexes by table name.

If you haven't used the rails_indexes plugin on your project yet, you should. (I put it in a sep 'profiling' branch where I use all these tools to analyze the code, ensuring that the tools won't break anything)

http://github.com/jmccaffrey/rails_indexes

I still need to get the tests working in my local env, and add some coverage for my changes, and then issue a pull request, but for now this got me what I needed.

Updates to Railroad application modeling tool

2010-06-17T10:57:00.001-05:00

On Monday I started looking into a new project to help diagnose some performance problems in PDF report generation. When I size up a project, I use a varieties of tools and check lists to get my bearings (that might make a good post in itself), and one of the things I've added to my list is to generate a domain model graph, so I can see how the objects are related.

When it comes to performance, its always good to understand the structure of the data, and how its related. While you might find that just a few server tweaks can optimize performance, without forcing you to get tangled up in the code and potentially break something, at some point, you are going to need to understand the domain model, and how it is structured.

I've used the http://github.com/peterhoeg/RailRoad library in the past, and its always worked well. But when I got the current version and tried it out, I ran into a few problems.

I couldn't ignore a pattern of files, only individual files
It couldn't generate the graph for the controllers on a rails 2.3.5 project

I found a fork that solved the rails 2.3.5 problem, but not the exclusion problem. So I forked it and made the tests pass http://github.com/jmccaffrey/RailRoad, then went about adding some tests to prove the file glob pattern functionality I needed.

I run this command to get the graph for a basic Rails project:

railroad -i -l -t -a -m -M | dot -Tsvg | sed 's/font-size:14.00/font-size:11.00/g' > doc/models_graph.svg

The sed part is to deal with an issue in Graphviz tools, which makes the command long, and I can never remember that, so I created an alias:

alias r_graph="railroad -i -l -t -a -m -M | dot -Tsvg | sed 's/font-size:14.00/font-size:11.00/g' > doc/models_graph.svg"

To exclude a pattern just add:
-e app/models/specific_file.rb,app/models/*/*.rb
I didn't realize when playing with the patterns at first, that you don't need quotes, and there can't be any space between the commas. The second entry above will ignore any files in directories under the models dir.

Here is an example from a simple project, using this command to generate a png
railroad -i -l -t -a -m -M | dot -Tpng > doc/models_graph.png

I didn't try it on windows, or with any models that have acts_as_state_machine, though I did move things around, so its possible that I broke something. If you could pull it down, install it and tell me if you run into anything, I'll see if I can fix it. Its a cool tool, and I'd like to make it better if anyone has any ideas.

Mowing the grass: How to get out of the 'commodity software' game

2010-06-13T10:04:00.007-05:00

photo credit: great_sea

A while ago Alice Toth and I had a conversation about how we can better serve our clients, and while we normally delve into project efficiencies like communication, developer training, and good QA practices, this time we both concluded that we need to do a better job of helping our clients reach their goals in the most efficient way possible, and sometimes that means 'talking them down' from the specific implementation idea they have, and finding a faster way to get there.

I said that one of our challenges is that sometimes our clients feel they've hired us to just 'Mow the Grass', not discuss the landscape architecture, so we're not always in a place to be heard when it comes to discussing the fundamentals of a project. Alice wrote a nice post about what 'Just Mow the Grass' meant to her, and the strategy she's devised to find a nice middle ground.

Here's how I see it:

I was thinking about the times when we encounter projects that are looking for what I call 'commodity software development' which is to say they feel they already have all the requirements, and everything is all figured out, and they just need a fixed-bid project to have it built. The reality though is that most of the time the idea is not fully fleshed out, and needs a good deal of work. While there might be some giant spreadsheet of features, usually the core business goals aren't clear, and many fundamental issues have not been covered.

If the customer thinks they have hired us to just 'mow the grass', they aren't in a place to hear any ideas about the big picture, they simply want us to 'build it', and at a consulting company you need projects, and you want to have happy customers, so it seems you can either just build exactly what they asked for, and hope it turns out for the best, or help them confirm the business fundamentals a bit, and verify that the plan will work.

"I didn't hire you to question me"

Its not easy to get into the fundamentals with customers, but the overall success of your work is dependent on knowing the fundamental assumptions that led to its creation, and in my mind that means we need to explore those ideas and test them out.

The other comparison/metaphor I had been kicking around in my head goes like this:

Imagine you are a building contractor. You build buildings to meet the needs of your clients, while adhering to the 'best practices' of your industry, (building codes, rules, regulations and prevalent design patterns). If a client comes to you and wants you to build a new 'asian-fusion' restaurant in an area that you are pretty sure its not going to take off, you could suggest that another location might work better, but in reality, the customer probably isn't interested in what a contractor has to say about the viability of a concept restaurant. If the concept doesn't take off, the contractor still got paid, the building is still standing and can be used for other purposes, and an outsider would not conclude that the restaurant failed because of anything related to the structure of the building itself. And while it might seem that the contractor doesn't care if the concept restaurant succeeds or not, let's suppose that if it had been successful it would lead to a long and profitable relationship where the contractor and the business owner continue to work together on new buildings and projects.

So in this example, the 'building' project has specific and measurable requirements that are independent of the big picture success of the restaurant.

Now comparing that to a software development project that is core to the business. The software product that is created is inextricably linked to the success of the big picture project, and the software is unlikely to stand on its own and be usable for some other business. In fact if the big picture project is not successful often times the software product itself is blamed.

So its pretty clear to me that the success of a custom software project is built on knowing the foundation of the business goals, and that while following Agile Development practices will ensure that you are moving forward and getting work done, those tasks need to be tightly coupled to the business goals to ensure that you are getting the Right work done.

I have seen many projects like this:

Client comes in with a big list of ideas "Guaranteed to change the market", and a small/medium budget
The ideas are turned into a feature list and timeline, the budget is expanded as each time the question "Do you really need that to launch?" is always answered with an unequivocal "Yes!!"
Development begins, real-end user feedback is non-existent, and a single business expert is responsible for deciding how features should be done.
At the midpoint of the original timeline, everyone on the team would say things are going well. The velocity is high, the product is taking shape, some major technical issues have been tackled, and things are moving forward at a good pace. Its not until things are getting ready for "Release" that it starts to fall apart.
A beta/qa release is 'pushed back' as more 'must have' features get added, unknown stakeholders emerge with 'questions' and new must have requirements, "Killer" features are added as a way to justify where the previous time was spent, and the budget is rapidly disappearing.
The Beta release finally happens, customers and stakeholders are underwhelmed, critical features are missing, the budget-eating "killer features" aren't resonating, a competitor goes to market with a simple but 'end-to-end' complete product, the team feels stressed and tensions are high.
The budget is gone, the product is not complete and needs to be 'salvaged'. Accusations of 'over design', and 'over billing' begin. Team members are asked to 'review their timesheets', and make sure every last minute is documented.
At this point the client either "doubles down", coming up with another round of funding to finish the project, or the thing dissolves into a legal battle, and general dissatisfaction on both sides.

(I'm not saying this is how all projects go, and it certainly never happened on a project where I was the project manager, but I have been on some projects like this, and I have seen many play out this way)

I've noticed this fundamental failure to align the proposed features of a product to their expected return more on iPhone projects, and other smaller budget projects. That's not to say that its not present on larger projects, just that its more noticeable on smaller projects.

Ok, so now what?

Where as I had been suggesting that we need to just dive into the fundamentals of the project and balance out the features relative to their expected value, and that its our job to 'Fight' to make sure we're building the right solution to the right problem, Alice's suggestion was that we need to consistently deliver high-quality results on the tasks we've been charged to do, even if it is just 'Mowing the grass', in order to establish trust and credibility, and earn our place at the 'big table' of advice giving.

What I don't like about Alice's idea is that its simple, elegant, logical, and its not mine! (There's nothing worse than someone taking something you said and interpreting it to be more positive and cohesive than you realized)

So with a balance of "just do what they asked", and "let's understand this business plan", we can make progress on the tasks that are laid out, moving forward, but also making sure we're moving in the right direction.

Because everything seems just fine until you go to release and have real users touching the system, it seems pretty clear that the sooner you do that, the sooner you'll be able to correct course.

You may have won this round Alice, but what will you do with customers like these?

RailsConf day #3 photos and videos

2010-06-10T10:32:00.002-05:00

Photos from the day http://picasaweb.google.com/john.mccaffrey/RailsconfDay3

MusicJam photos http://picasaweb.google.com/john.mccaffrey/RailsconfMusicjam

Videos
(you can see the full list, and full length video here on Justin.tv
Message in a bottle

Watch the Video on Justin.tv

The Craziest 'CrazyTrain' I've ever seen

Watch live video from johnny_zebra on Justin.tv

Black Magic Woman

Watch live video from johnny_zebra on Justin.tv

The Who

Watch live video from johnny_zebra on Justin.tv

ZZ Top

Watch live video from johnny_zebra on Justin.tv

Blues Jam

Watch live video from johnny_zebra on Justin.tv