Alex Young's weblog

Rapid Rails: Boot Up Time

Alex Young — Fri, 3 Jul 2009 10:02:38 BST

Rails boot up time really starts to drag when you’re working in a TDD or BDD style. There’s projects out there that create long running distributed processes to ease this, but it’s interesting to look at exactly why Rails might boot up slowly.

I have a project that isn’t massively complicated, but it has suffered from “just add” culture. I’ve been told to “just add” Excel support, PDF generation, and a lot more features that require complex libraries. This naturally increases the memory footprint of my application, and it also increases boot time.

Before I started my analysis, the boot time averaged at 11.6 seconds. After my improvements the boot time is 7 seconds. By comparison, the code for Deadline boots in 4 seconds, but it has few dependencies (it’s a joy to work on by the way).

Before continuing this technical and possibly dull article, I’ve mined my internet meme library to add a soundtrack. Replace all instances of on a boat with on Rails:

And I don’t think we even get SNL in the UK!

Analysing boot up time

Boot up time shouldn’t be related to the number of models and controllers you have. Rails dependency loading should require these files when they’re first referenced—unless you use a plugin like ThinkingSphinx that intentionally loads models on start up.

To prove this, add this to the first line of a model file: puts "I've been required", then start script/console. Unless something is including the model, you shouldn’t see the message until you type the model’s class name and press return.

So what’s increasing my app’s carbon footprint if it’s not my own code? To find out, I wrapped up each major block of code in environment.rb in Benchmark.measure blocks. I found that my library requires were using most of the time. Also, this code took over a second:

ActiveSupport::Inflector.inflections do |inflect|
  inflect.irregular 'cookie', 'cookies'
end

I have that code because even though Rails correctly pluralises it gets the relationship wrong in a has_many (it looks for LoginCooky). But why would that take a second? I have no idea. Seeing as I technically shouldn’t need this definition I ignored it for now.

Requires

The gem requires in environment.rb were the real culprit. If you think about what Ruby does here it’s obvious: the amount of code evaluated for some libraries will be huge.

This project is old and wasn’t using config.gem, so I added calls to that in the Rails::Initializer.run block instead of plain old require. This is cool because Rails will prompt missing library installation, but it’s also cool because of this:

config.gem 'scruffy', :lib => false

Setting the lib to false will make Rails aware you want the library, but not require it. This library is only used in one place, there’s a GraphController that manages dynamic graph generation. I added a require 'scruffy' to that file instead.

This doesn’t always make sense of course: there might be calls all over your code to a particular library. That’s fine, just require it at boot. The amount of times that wasn’t the case, however, reduced my boot time by 4 seconds.

Forensics

I’m not exactly Grissom or Horatio, but I did write a little script to watch the require time of every require in the project. I required it in environment.rb and watched to see which libraries seemed slow. I’ve left my commented out bits where I was trying to figure out how to only measure “top level” requires in environment.rb—I never fully figured this out, does anyone know how to do it?

Lessons

It’s interesting what you learn by looking at older Rails projects. I always used to just require my libraries in the environment file so other developers would see the app break if they hadn’t installed them. The newer way, config.gem, provides a better solution and lets us require files when we actually need them.

Rapid Rails: Mastering Indexes

Alex Young — Thu, 2 Jul 2009 13:48:31 BST

I wrote a series last year called “Rapid Rails”. It was all about making your Rails app and development process faster. I’ve been analysing the performance of some mature Rails projects recently, so rapid rails is back.

If I’m tasked with improving an application’s performance, the first thing I look at is indexes. Indexes are something you’re told are important when you learn about relational databases, but the effort of domain modeling usually makes most people forget all about them. That means you’ll often come across a slow project that can be improved just by adding indexes.

Rails lets you add these in a migration with add_index :table, :row or even add_index :table, [:row1, :row2]. Most of you probably already know this. What you might not know is your RDBMS might be ignoring your indexes anyway!

The following steps will help you detect and avoid this.

#1: Identify a slow query to attack

Look at your Rails development logs to find queries. MySQL has a slow query log as well.

Look at what fields the query joins on, and look at what it sorts on too. Add the index.

#2: Determine if the index is being used

Run script/dbconsole and enter your database’s password to get into a database shell.

If you’re using MySQL or Postgres, EXPLAIN will help identify what indexes are used. Just call a query with it:

EXPLAIN SELECT * FROM `articles`
WHERE
  (articles.published = 1 AND headline = 0 )
  AND ( (`articles`.`type` = 'EquitiesStory' ) )
ORDER BY created_on
DESC LIMIT 1000, 40

This query took up to 5 seconds on some fast hardware with lots of memory. It was really getting on my nerves. Of course, STI wasn’t helping with the application’s large data sets, but it was a case of DRY vs. fully normalised data models.

In this case I added a fat index: index_articles_on_type_and_published_and_headline_and_created_on. However, MySQL said it wasn’t even using my giant index. EXPLAIN said it used:

Using intersect(index_articles_on_published,index_articles_on_headline,index_articles_on_type)

#3: ANALYZE, OPTIMIZE

MySQL store statistics on the way your tables are used. It’s not as fancy as it sounds. If you’d like to rebuild these, check out ANALYZE and OPTIMIZE.

#4: USE INDEX

If you suspect MySQL’s optimiser is letting you down, try running the query with USE INDEX:

SELECT * FROM `articles`
USE INDEX
  (index_articles_on_type_and_published_and_headline_and_created_on)
WHERE
  (articles.published = 1 AND headline = 0 )
  AND ( (`articles`.`type` = 'EquitiesStory' ) )
ORDER BY created_on
DESC LIMIT 1040, 40;

Even if this runs faster MySQL might have cached your previous tests. That’s why I have those LIMITs, I’m trying to get results that haven’t been cached. And EXPLAIN told me that this query uses the index I want.

It would be possible to construct a find in Rails that uses USE INDEX, but I don’t want to. I like keeping things database agnostic if possible. If MySQL’s optimiser is failing to use the right index, what else can we do?

#5: Trick the optimiser

This is bad advice. MySQL’s optimiser changes between versions, so I’m telling you to do something naughty here. After repeated experiments, I found changing my query slightly made the optimiser work harder to find the best index:

SELECT * FROM `articles`
USE INDEX
  (index_articles_on_type_and_published_and_headline_and_created_on)
WHERE
  (articles.published = 1 AND headline != 1 )
  AND ( (`articles`.`type` = 'EquitiesStory' ) )
ORDER BY created_on DESC LIMIT 1080, 40;

Notice the headline != 1? That somehow resulted in MySQL using the index I wanted. This ridiculous modification has resulted in a massive performance gain and a happy client.

#6: Filesort

If you see filesort appear in your EXPLAIN, it’s likely that this will reduce performance. It’ll be caused by an ORDER BY. I found simply forcing MySQL to use my created_on index improved performance for the previous queries, because it went from Using where, filesort to just Using where.

Filesort, as the name implies, is used to sort records when there is an ORDER BY clause in the query. The reason it has the prefix “file” is because temporary files may be used to store intermediate results.

From: Improving Filesort performance in MySQL

#7: Take a break

Don’t be rushing around adding indexes to live apps, and take some time out to surf YouTube.

iPhone 3.0 Development Glitches

Alex Young — Tue, 23 Jun 2009 11:11:44 BST

I had a few glitches in the Helipad iPhone app when 3.0 rolled around. I’ve put the solution here to help those of you searching for the error message.

Helipad syncs to our web app and displays a syncing progress screen. I open this view in a new view controller like this:

SyncController *sync = [[SyncController alloc] initWithNibName:@"SyncController" 
                                                        bundle:nil];
[self presentModalViewController:sync animated:YES];

The parent view displays a UIWebView, and the controller runs the sync process on another thread so the display can be updated with feedback. The problem was that updating UIWebView crashed the app:

bool _WebTryThreadLock(bool): Tried to obtain the web lock from a thread
other than the main thread or the web thread. This may be a result of calling
to UIKit from a secondary thread.

I can only assume that this was caused as part of the improvements to UIWebView’s performance. The crash actually occurred when the sync process called dismissModalViewControllerAnimated:. To resolve this, I called it on the main thread like this:

[self performSelectorOnMainThread:@selector(dismissSyncView) 
                       withObject:nil
                    waitUntilDone:YES];

My method, dismissSyncView, looks like this:

- (void)dismissSyncView {
    [self dismissModalViewControllerAnimated:YES];
    // Other custom code goes here  
}

Experiences Releasing an Open Source Project

Alex Young — Tue, 2 Jun 2009 12:42:14 BST

Note: If you want to read about JsChat have a look at the JsChat blog.

I’ve always dabbled with open source (Linux got me through university), but it’s only since GitHub that I’ve released a lot of stuff: GitHub.com/alexyoung—I only had weird experiments on my site before, and I released some stuff through work too (code.helicoid.net). Nothing much drew any major attention until I publicly announced JsChat.

JsChat is a simple protocol for realtime chat. It’s like IRC, but relies on JSON for message encoding and syntax. I made it so it would be easier to build nice web-based chat clients. I built an Ncurses client, but it’s… kind of dodgy. I use it daily but I recognise that it needs work.

The point is that JsChat is about communication. When you visit the site, you can join a chat room straight away. Therefore, when the project appeared on Ruby news sites, it was inevitable that people would join a chat room and talk to us.

This is what happened:

People complained that we were using polling instead of Juggernaut or other Flash-sockets hacks—they didn’t seem to realise the project was more than the web client, or that I specifically decided not to rely on technologies like Flash
People complained about the language and libraries I used (I had good reasons for using these too)
People joined and made random racist comments
90% of people joined with a name like ‘test’ and wrote random noise. JsChat loads the lastlog automatically in the web client, so it’s obvious that people are having real conversations

On the plus side:

We made some friends through JsChat
Some people are now using it in their sites for professional projects
We got lots of bug reports and useful feedback
Some people said they liked the interface design

Intuitive Code

Alex Young — Fri, 15 May 2009 16:10:34 BST

I’ve read a lot of game reviews where the intuitiveness of a game is described: the control scheme and menu system can be rated by how intuitive they feel. Intuition is a useful concept, and means more than “easy to use”—it imparts a sense of instinct; mental processes found in the subconscious.

Code can be intuitive too. Not intuitive to write, but to read. If this seems confusing just think about pseudo-code: beginners are immediately comfortable with pseudo-code but put off by the arcane constructs and symbols of a real programming language. Some languages appear more like pseudo-code: Ruby can be written clearly with little hand-holding for the interpreter. A few conventions must be learned: symbols and blocks usually confuse beginners, but once the basics are second nature code comprehension edges closer to instinct.

I’ve always argued verbosity over needless truncation: write recurring_reminders not rcrr_rmdrs or just r. Concise code is important but is not created by shortening variable or method names. Verbose names are not enough, however. There are more ways to create intuitive code.

Another more fundamental technique is to follow the conventions of your language. Ruby programmers commonly use two-letter spacing, lower-case names for variables, do ... end blocks rather than curly brackets (unless the code fits on one line), symbols for options (rather than string constants), exclamation marks and question marks where appropriate. Stick to your language’s conventions so other people can follow your code (I learned this the hard way with JavaScript).

Short methods are another good technique: method names more accurately describe a short amount of code, and it’s easier to consider a short amount of code as one chunk, and therefore compose more complex methods.

Tests should also be intuitive, they should follow the same stylistic guidelines. There’s an argument for single assertion tests, which logically follows short method names, but I find this is too restrictive. Here’s an example of one of my tests:

  def test_contains_repeater?
    assert Deadline.contains_repeater?("Kev's birthday every April 9th")
    assert Deadline.contains_repeater?("Simon's birthday each May 9th")
    assert !Deadline.contains_repeater?("Buy every Star Trek DVD on June 9th")
  end

This code is simple and easy to follow, but it doesn’t help the reader understand what a “repeater” is. Rather than adding a comment, a clearer method name would help: Deadline.contains_recurring_event?. I find that TDD or BDD helps me write clear method names from the outset.

On the subject of method naming: I like the concept of Intention Not Algorithm. Methods should be named after what they accomplish, rather than how they work. This could be applied to the test method name above:

  def test_phrases_that_contain_recurring_events
    assert Deadline.contains_recurring_event?("Kev's birthday every April 9th")
    assert Deadline.contains_recurring_event?("Simon's birthday each May 9th")
  end

  def test_phrases_that_do_not_contain_recurring_events
    assert !Deadline.contains_recurring_event?("Buy every biscuit on June 9th")
  end

Without moving on to the obvious question of why contains_recurring_event? is a class method, I hope you can see that considering how intuitive code is for the reader can yield some interesting insights into readability and structure.

It would be great to look at common design patterns to see how intuitive they are in implementation. How many people are quickly confused by code that uses deep-OO jargon to describe simple patterns like Prototype, Decorator and Factories?

Future of Web Design Notes

Alex Young — Fri, 1 May 2009 19:25:59 BST

I almost survived Future of Web Design unscathed. I say almost because I still have a slightly dull head, thanks to the after party and free drinks. I can’t quite remember why, but at one point I thought ordering a load of sambuca shots would be a great idea.

Carsonified’s events have so far been a rare chance for me to meet up with fellow web designers and developers. This FOWD had a particularly friendly atmosphere, and I met lots of interesting people. The Carsonified team were professional and friendly, keeping the event ticking along smoothly.

Web design is a diverse subject. There’s a continuum between design as ergonomics/usability and visual design. Add standards to the mix and web design suddenly looks more like engineering. The talks moved back and forth along this continuum throughout the day.

I don’t think web design currently knows whether it’s design, engineering or science, but combining all the required disciplines keeps enough people interested and inspired to make events like this both fascinating and productive.

Danny Somekh

The first talk was an off the bill presentation by Danny Somekh about a concept he calls Agile Branding. I’m sure he’ll elaborate on the concept at his blog, but a few things stuck with me:

During meetings, only involve people who can add to the creative process
Use visual exploration early on
Involve users/customers
Try to create an environment that fosters co-creation

As a Rails developer and big PragProg fan, I find the idea of agile branding/design fascinating. I found it hard to see how it would be implemented, but so many designers work with such rigid constraints that it could be a big thing in the future.

Jim Coudal

I’ve got a Coudal sticker on my netbook, so I felt like a bit of a fanboy during Jim’s keynote. I’m not really, but I do like their Field Notes books.

He agreed with some of Danny’s points—some early agile design endorsement perhaps? He went on to talk about Booking Bands and the relationship between having a short attention span and the creative process. I liked Jin’s positive analysis of short attention spans, and how switching to new projects/technologies/tools can yield unexpected rewards.

Meagan Fisher

Meagan, from SimpleBits, talked about building mobile web interfaces. Her experiences and design decisions closely follow what I’ve done at Helicoid—I launch all my apps with mobile interfaces like the ones she described, and blog occasionally about the process at mmm.

She really demystified mobile web design for people who were unfamiliar with it, and gave some great book recommendations, like Cameron Moll’s Mobile Web Book. I’ve read it, it’s just about all you need to get going in the mobile web.

Brett Welch

Brett from GoodBarry gave a talk about design and business, with advice on how designers can maintain confidence and relationships with their clients despite market trends.

His talk centred around establishing yourself as an expert, communicating value to clients in ways they can understand, understanding your clients and their strategies. He even gave practical advice on how to conduct meetings with clients:

When starting a meeting, always talk about the client’s business first. Try to establish the business context
Get a feel for their business and set a tone for helping them to become successful
Understand what they want to achieve and manage their expectations
Websites need marketing plans, and ongoing attention
Set business targets: avoid talking about traffic early on
Talk about tools, timeline, what happens after the site has been built

I learned a lot from Brett’s talk, it was useful to hear business advice aimed at us.

Mark Boulton

Mark’s talk was about typography. He discussed the basics, type as interface, the controversial topic of font embedding, and helping Drupal.

I think his work on Drupal sounds like something that could be applied to many open source projects:

Wherever possible, help people make good design decisions
Provide smart defaults: don’t offer every font, just let them choose from two good ones
Styles vs. lots of options: reduce choice
Authors want to be creative, which is why they’ll always experiment with the design options. Create a system that empowers them
Try to make choices higher up the “cascade”

Robin Christpherson

Robin’s talk was about designing for accessibility. He used screen reading software to demonstrate how the web feels for the blind and visually impaired.

Well-structured documents and headings make documents easier to navigate
Documents structured with a logical reading order helps a lot (I think most designers do this anyway since the CSS/standards revolution, but some of his example sites were still hard to navigate)
Consider the amount of info on each page: he brought up a list of the links on a page to navigate, but most of the pages had hundreds of links
Captcha accessibility is generally poor
Sifr/flash: despite attempts at accessibility, more than 4 of these objects on a page can make the reader very slow
Drop down boxes can be skipped entirely so he didn’t have to read each option, unlike lists of links
“Live regions” and screen reader support
Mobile versions of sites are easier because they’re so simple and quick to navigate

Mike Kus

Mike discussed design from the perspective of someone with an extensive background in graphic design. He suggested that we should break trends: designers usually research and draw on a wide range of influences for their work, discovering new combinations, and we should do the same.

His slides were beautiful, and he had a 3 minute video at the end showing how he made them: I expected a video will be posted of his talk at some point, it’ll be worth finding to see this part.

Sabrina Dent

Sabrina talked about “Stalinist web design”. Her presentation was full of humour, but it also had some useful tips:

Learn to say “no”
Eliminate choices: Why give clients 3 mockups? Why not give them your best mockup—you’re the expert, design choices are yours to make
Just do it: Rather than asking a client about possible improvements, just do them
Caveat: You better be good

Folkert Gorter

Folkert’s slides are on a big HTML page here: cargo.superfamous.com/972. He drew parallels between his early work on CDROM design and modern social web applications.

Simon Sankarayya

Simon Sankarayya’s talk reminded me heavily of A Smile in the Mind. He also related his experience from the days of CDROM and Director to the modern web. He mentioned Naoto Fukasawa as an influence.

Molly Holzschlag

Molly’s talk concerned the future of web standards, and had some healthy Microsoft bashing. I noticed she said the next IE will support SVG which is cool, but why couldn’t that have gone into 8?

Ryan Singer Workshop

Alex Young — Tue, 28 Apr 2009 13:00:35 BST

I went to the Ryan Singer Carsonified workshop yesterday. The title of the workshop was “How to design amazing web app user interfaces”. Ryan discussed his influences and processes behind 37signal’s interfaces.

Ryan’s workshop covered:

Least effective difference
Copy
Workflow: Highrise example
Flow
Validation
Rails advice
How MVC helps organise designers and developers

As a web application developer and designer who uses Rails, I found his insights fascinating. Here are my notes from the workshop. I haven’t managed to capture everything we talked about—these are the things that made an impression on me.

Least effective difference

I think one thing most of the attendees will take away from that workshop will be the concept of least effective difference. Ryan cited Edward Tufte for this concept.

Contrast should be used purposefully. For example: adding borders to a table may increase the contrast between the page and the table, making the table’s data less prominent. Certain things cause “vibration”, attracting the eye to things that aren’t important.

Ryan suggested that HTML should have a weak tag. After all, it has strong, so why not? After all, music has piano, mezzo-piano, mezzo-forte, and forte.

Copy

One of Ryan’s major points was the importance of copy in web applications. Generating good quality text is part of his design process. He discussed the following steps, using Highrise’s sidebar as an example:

Write out a list of things the sidebar should do: Add New Person, Add New Company, etc.
Work on the copy without worrying about graphics, buttons, colours—anyone should be able to understand the functionality from the text alone
Start with verbose text
Slowly tighten the language
Iterate this process to refine

I noticed that Ryan seemed to pull out active verbs, strip passive verbs and impersonal pronouns, and try to start the text with verbs. For example:

Would you like to add a person?
You can create a person here
Add a new person

Workflow: Highrise example

Ryan addressed the visual design decisions for a particular Highrise sidebar. At this stage the design was a simple list of links. These are the questions a designer should ask themselves:

Why change the list of links? Does this design do the job?
Links are equally weighted: does this reflect use case?
As a designer, think about why someone is seeing the list, and think about where they want to go next

A daily operation in running a business is “got a new contact, need to add it to Highrise”. That means the “Add new person” link would need to be the most prominent. Ryan evolved the design, applying the least effective difference principle, making certain options grey, others bold and larger, with some spacing and suitable headers.

Ryan also suggested that links generally mean “go somewhere”, whereas buttons represent functionality.

1: List things the feature requires

Add another person to Highrise
Add another company to Highrise
Import contact from vCard, Excel, Basecamp, Outlook
Export data to vCard, Excel

2: Refine the copy

Add a new person
Add a new company
Import (vCard, Excel, Basecamp, Outlook...) 
Export (vCard, Excel)

3: Refine design

Decide what matters
Make that pop out
Make everything else fade back

4: Consider other visual elements

Flow

Always be ready for the next user input or action.

Ryan explained the difference between database-like applications, and ones that consider flow. He also gave examples of web applications that don’t consider user flow.

He explained 37signal’s thinking behind taking the user from the add contact page to the show contact page, rather than back to the list of contacts.

I found this interesting, because it’s sometimes hard to see where the next action should be in a web application. It’s not always clear. In Highrise, people commonly need to append related info to contacts after adding them, so it makes more sense to redirect to the show contact page rather than the list of contacts.

Validation

Rather than being quick to rely on Rails validation errors, Ryan’s designs attempt to accept any user input.

Submitting an empty form could just show the form again, as submitting and showing an ‘add page’ might be more confusing
In Backpack, adding a page with no title will create the page and just name it ‘No title’
Try and avoid states where the user can make a mistake

Rails advice

JavaScript and stylesheets follow Rails controller names
“Body classes” used to manage this
Rarely use helpers to render blocks of HTML, preferring lots of templates instead
Even rjs should use HTML partials

Book recommendations

Edward Tufte: The Visual Display of Quantitative Information
Kent Beck: Smalltalk Best Practice Patterns
Modeling: Christopher Alexander’s notes on synthesis form, Eric Evans domain driven design

Conclusions

If read my blog, you’ll know that my background is programming. As a web developer for the last 10 years I’ve had to learn design as a matter of course, and I’ve taken it more seriously since starting my own consultancy and web application company. The way Ryan works is very similar to me, so yesterday’s workshop galvanised my confidence.

Also, I’d love to work with designers who are familiar with Kent Beck and Edward Tufte’s books, so get reading!

Deadline for Mac (and GitHub hooks)

Alex Young — Tue, 21 Apr 2009 13:39:14 BST

Deadline is a reminder web app I made. It sends out alerts through SMS, email and IM, and now it works with Mac OS too. I spent this weekend building a Mac cocoa app for it that integrates with Growl. You can download it here: DeadlineGrowl

Deadline also works with GitHub: Deadline: Notification API and GitHub support. It uses post-receive hooks to grab commit messages and notify you through your preferred notification method.

I really like the idea of hook URLs. Deadline has a generic notification API (it’s just a GET to a URL) you could use to notify your users about events in your apps. If you’re interested in using this, or using Deadline’s API for a project, let me know and I’ll give you a hand.

Secure Trading Rails Plugin

Alex Young — Wed, 15 Apr 2009 12:05:17 BST

I use Secure Trading for Helicoid’s payment processing. I wrote a payment processor plugin for our apps about two years ago, and it’s been in production since then. A few people have asked me to open source the code, so here it is: securetrading-rails.

I’d like to integrate it with ActiveMerchant because I like their API, but I haven’t had time so far.

Out of all the payment processing companies I’ve dealt with, which is a lot, Secure Trading have been the most reliable with the best customer service. Their XML API is relatively straightforward too.

Beautiful Algorithms

Alex Young — Fri, 10 Apr 2009 11:07:22 BST

I’m starting a new series on Quite Useful called beautiful algorithms. I’m going to talk about the algorithms behind graphical effects, especially those from the demo scene back in the 90s.

I’ve posted some examples to my GitHub account if you want to check out what I’ve been researching so far. They use Ruby Processing, which is a pretty accessible way to try out ideas.

Please contribute improvements to these and I’ll include them. Imagine what the demo scene would have been like if people shared code on GitHub!