I was already quite busy when everything stepped up a couple of gears around February due to going into business with  ProjectX and thus my friend John Clegg.  I’m slowly taking over the back end of the business while he focuses on the front end.  Essentially I get more toys and larger projects to play with :-)

To be realistic I reckon I’ll only get out a blog entry once every couple of weeks at this stage - which will be hard as there’s an increasing amount to blog about.  Top of my list is Nginx with the recently updated Phusion Passenger and Ruby Enterprise Edition which is powering our new Redmine server.

Stay tuned.

No related posts.

So with that in mind I’ve setup a new google group, WellOptimised.  If there is enough interest I’ll organise meetings for those here in Wellington.

If you are interested pop along via the above link and introduce yourself.

Related posts:

1. Performance optimisation through named_scope I recently gave a presentation about using named_scopes to WellRailed...

The plugin is still useful as I didn’t submit it’s session caching changes to Rails - essentially using Rails.cache for session storage rather than establishing a second connection to MemCache.  This also benefits from the local cache changes.

Where next?  In the next week or so I plan on updating the plugin by removing the local cache functionality to prevent any conflicts with Rails 2.3 when it’s released.

After that, I’m planning on working through the list of changes detailed under Future Plans in the plugins readme, also listed below.

• TESTS! (I wrote a bunch for the patch, just need to do that for the remaining session changes)
• Make this a Gem
• Conditional caching
• Multi-Get
• Dynamic key helpers
• Make the code more awesome

Related posts:

1. Spandex Mem Cache Store taken from the plugin’s README Description A enhanced version of...
2. How to reduce your memcache reads Several months back I presented at WellRailed on using Memcache...
3. Starting simple with Rails caching So, you’ve launched your website and increasingly people are actually...

I received two suggestions for the ‘best install method’, firstly MacPorts and then later pre-compiled dmg’s from Kyng Chaos.  I already use ports extensively making that choice easy for me - I like being able to  uninstall things if they don’t work to well and was concerned the dmg option wouldn’t allow me to.

sudo port install postgresql83 postgresql83-doc postgresql83-server postgis

I then added the following to my path in ~/.zshrc so I could actually run things

/opt/local/lib/postgresql83/bin

and started the server.

sudo launchctl load -w /Library/LaunchDaemons/org.macports.postgresql83-server.plist

All seemed to be fine and dandy as I could connect with pgAdmin.  I quickly discovered that I needed to create a user with the same login as my osx login when I tried creating a test db.

Next I tried restoring my client’s db dump and it barfed big time.  As I’ll be working on some mapping software I’d installed the PostGIS package which seems to have failed to link up with Postgres properly.   I was getting the follow missing library error:

could not load library /usr/lib/postgresl/liblwgeom.so.1.3

Obviously it’s looking in the wrong place for a start, so I setup a LIBDIR export in my profile to get over that problem and restarted the server.

export LIBDIR=/opt/local/lib/postgresql83

That still didn’t solve the problem though as I don’t have a file by that name on my system, I do however have liblwgeom.1.3.so which looked to be the same thing.  Pretty annoying, but easily solved with a symlink.

cd /opt/local/lib/postgresql83
sudo ln -snf liblwgeom.1.3.so liblwgeom.so.1.3

Then the backup restored like a chalm.

To get Rails talking to Postgresql I needed to install the postgres adapter gem appropriately called ‘postgres’.  I didn’t expect that to work straight off though as I’d installed postgresql with ports and the gem by default would be looking in the wrong place for it.  I was correct, but I also received an additional error complaining about the architechture not being specified - I needed to tell it I wanted i386.

sudo env ARCHFLAGS="-arch i386" gem install postgres -- --with-pgsql-include-dir=/opt/local/include/postgresql83l --with-pgsql-lib-dir=/opt/local/lib/postgresql83/

I gropped around for a bit trying to figure out the correct way of letting it know and eventually found the above solution on code snippets.

Related posts:

1. Setting up a resumable remote login to ubuntu with vnc I’ve changed my home server from Centos 5.1 (32bit) to...

Since I occasionally have quite a bit to say, like in my previous entry or this one about where business logic should reside, I’m now only showing excerpts on the index page rather than whole entries.  I’ve also gone though and added in a couple of helpful plugins such as auto suggesting related posts and easy social bookmarking.  My next step is to change the source code syntax highlighting css - I’ll probably just use the Textmate bundle in Textmate.

Currently the homepage ySlow score is 79, I can get this to 85 if I concatenate and minify the css as doing this reduces size and number of requests for empty caches.  This hasn’t happened yet as it’ll require a bit of hand tweaking due to not working out of the box.

Overall I’ve gone for a more clean, simple and wider look by using Acamas which is based on the opensource Thematic framework.  It looks great on a Mac, pretty good under Windows and terrible in IE6.  If you experience any problems, in particullar with the rss feed (now thru feedburner.com) please drop me a line.

Update: I’ve gotten the ySlow score up to 90 now as the css is contenanted and minified plus the sidebar images are now being hosted locally allowing proper expires headers.  With the css files I’d simply been adding them together in the wrong order.  There is still a problem with css expressions though, but this will go away when I drop the syntax highlighting plugin - further boosting my yScore.

No related posts.

You’ve noticed that your app is doing a lot of repetitive work - calculating and showing the same data over and over again.  This doesn’t fit with your adherence to DRY principles at all.  If you cached repetitive responses your servers could give you way more bang for you buck - and it would only cost your time.

You are about to embark on an exciting journey fraught with hidden dangers, so let me prepare you - cheesy I know but hey.

First, you have actually done some code optimisation haven’t you?  Stopped using the default mysql config?  Checked your slow query log?  Used pl_analyise to parse your production logs?  Fixed the popular, but slow actions?  Done some eager loading so you get everything in a couple of hits from mysql rather than many many many little hits as your views render?  When you dream you see log tailing in the background?  Minified your javascript and css?  Static assets have expire headers?  Don’t have too many Mongrels running?  Sessions are stored in MemCache? Go do all that if you haven’t and only come back here if this are still too slow.  Oh and try to get your yslow score above 75.

Note: I’m assuming that you have read a bit about caching in Rails already and have had a play, reading fragments from actions etc…  If not try googling as there are a number of great articles on it, the Agile book is probably a good start too.

Start at the end

Instead of just saying ‘do this awesome thing’ I’m gonna explain how I’ve gotten to saying that, as you shouldn’t be making blind decisions about these things.  Most of the time when I implement caching for clients I completely skip page caching and action caching and go straight to line item caching, ie caching of small fragments of html.  Sounds completely counter-intuitive doesn’t it, but it’s not.

First off I need to define a typical client’s website so you see where I’m coming from.  It’s a user driven web 2.0 website - so most of the data is user generated and sometimes social networking elements are included, where a user’s data can be restricted to just their friends instead of being publicly available.  Basically the site lists stuff that people put on it.

Why not page or action caching?  As a starting point it’s too hard to manage due to the continually changing nature of the data being presented.  This changing nature means your cache might not actually get used that much.  Additionally when you are logged in your name is typically on the page with other unique to you info - you have x messages etc…  The latter reason completely rules out caching a whole page for anyone that’s logged in.  Don’t restrict caching based speed increases to joe public as logged in people, ie people that give a damn about your site, shouldn’t be second class citizens.

Don’t start with caching lists

Developers still want to jump straight in and cache lists of items (and pages too) rather than the individual line items - basically as much of the page as they can in a single hit - thinking they’ll get great bang for their buck.  When you deploy with that strategy you’re in for a world of hurt.  This is because they’ve not realised they need to take the following into account.

Each time one of the line items changes you have to expire the cache for the whole listing.  If you have pagination then you’ll likely have to expire ALL cached pages unless you can 100% reliably work out which page the item is on.  So yes you have a separate cache entry for each page of the listing.  If someone adds or edits a line item you have to expire the whole lot - and thats what people should be doing all the time on your site right?.  Additionally, if you allow sorting or a filtering mechanism you have to uniquely cache by page and the additional dimension(s), thus further reducing the frequency the cache will be used.

What’s the goal?

What we are trying to achieve with caching is to DRY up the execution of requests, but in a way that doesn’t add complexity.  The idea is to let your servers focus on the unique stuff like processing credit cards and telling Jill’s friends that based on her network and comment history she really doesn’t like doing things with Sarah unless Joe is there too.

Keep this in mind.  With seeking something to cache, you are basically looking for something that has a high ratio of renders/requests of the exact same html in between changes of that html.  So the more content in the cached fragment, the more frequently you’ll need to expire it.  Putting that another way, the fewer data sources, instances of models and display dimensions like pagination, sorting and filtering used to make up the cached html the better.  The simplest you can go is one data source, ie a line item.  Remember, keep it simple.

Put in business speak, it’s a cost / benefit decision.

It’s all about the keys

When you store fragments in your cache you name them with a unique string based key so you can access them later -  exactly like a hash.  So yes, they are important.  Typically they looks like a url, sort of based on the actual url or controller/action and the dimensions involved to make it unique to the html being cached.  So for the above listing situation the key should look something like this:

‘/products/list/sort=price/status=in-stock/page=1′

This actually presents a completely separate reason as to why you shouldn’t do list caching when you start out on this path.  The problem with this key is that to expire it you have to count the number of products to determine how many pages there are and then manually go and expire all possible keys that could have been generated, one at a time - this is because expiring via regex is not supported.  If additional dimensions are involved the problem will compound even further.

Yes, that’s real ugly and will become completely unmanageable.  There are some clever tricks to help out there such as self expiring keys, but that’s a different topic which I’ll get to at a later date.

When to expire

Wait there’s more!  You have to monitor data changes so you can expire those keys when they become stale as it doesn’t happen magically.  Normally you’d setup a sweeper (observer) and just expire things when ‘after_save’ is triggered.  Not a problem.  If you have data from multiple models involved in your cached html fragments then you need to monitor all the models involved.  You can go as far as only expiring particular keys based on the attributes that are changing etc… now much easier in Rails 2.1 with Dirty Objects.

This is all adding yet another layer of complexity.  Ugh.  It should all be documented so that other developers know about it especially new developers - otherwise further work could mess it up and you might start showing stale data to users.  In my book that’s pretty much on par with errors occurring as you could be showing the wrong price or even deleted data to users.  Unless you are careful about it, it can all become some really hard to manage mess which really isn’t going to give much benefit to your users.

Line item caching

As you are not having to worry about other dimensions such as pagination, sort order and filtering, other line items or even if the user is logged in or not, the line item’s cache key should be something like this:

‘/products/#{product.id}/line_item’

To make sure it expires correctly all you have to do is setup a sweeper watching the line item and simply delete it’s key when it changes - use Rails.cache.delete for this.  If the line item is rendered using data from through relationships, you then have to monitor them too.  I’ve seen people putting a ‘last_active_at’ field on a line item and when anything related to it changes they update that time field.  That works nicely and things are kept simple.

Finally

That should set you off in the right direction, remembering though this is a starting point for your first caching strategy.  Later on you’ll actually start doing multiple layers of caching and covering more complex html, such as lists, if your site continues to grow.

Now hopefully I’ve convinced you to start out with caching the html for a single line item - or something just as simple.  If not post a comment and tell me why not.

Related posts:

1. Spandex Memcache Store is now in Rails 2.3 Just a quick note to say that the local cache...
2. Be careful how you count in Rails I’ve just encountered this problem yet again effecting code I’m...
3. I don’t use Foreign Key Constraints with Rails A couple of months back we had Nigel talking at...

A enhanced version of and replacement for the MemCacheStore shipping with rails.  It uses a per-request local cache to buffer duplicate memcache reads which can result in halving read requests, and it uses a single connection to memcache for both the cache and session stores.

It doesn’t yet have the awesome key creation and maintenance library that I showed as that has evolved quite a bit and needs a big bunch of love first - which I’m working on.

There is now a dedicated page on this site for the plugin, currently it’s holds a copy of the plugin’s README.  My intention is to update the page with examples etc.. as I extend the plugin.

Related posts:

1. Spandex Memcache Store is now in Rails 2.3 Just a quick note to say that the local cache...
2. Spandex Mem Cache Store taken from the plugin’s README Description A enhanced version of...

The problem was this: somewhere along the line attachment_fu was updated to execute the following line of code on classes using the has_attachment directive:

attr_accessible :uploaded_data

This is taking advantage of a security feature in Rails, it instructs the model to only allow :uploaded_data to be set through update_attributes and any like methods such as new and create.  So whitelisting bulk settable attributes.  It doesn’t raise any errors, just warnings in your log like this:

WARNING: Can't mass-assign these protected attributes: <attribute_name>

Due to no errors being raised the problem was presenting itself in other parts of the application as expected data was not being found.  Fortunately I was tailing my dev log and noticed the above warning message.

The problem in this situation was the historic code creating instances of the model by passing a hash through to SomeModel.create.  As this area of code was not utilising data posted to an action it was not a security risk.  Very fustrating, especially as this was common practice throughout the codebase.

As I didn’t want to do a huge bunch of re-keying my initial solution was to comment out the offending line in attachment_fu.  Yes this is a big nasty fix but there was time pressure, other problems to solve and I didn’t want to be held up having caused an unnecessary amount of manual testing.  In case it was forgotten about  and overriden with a future plugin update, a quick test with big comment was added.

This project hadn’t updated it’s gems and plugins for a while, meaning it was a big shock to the system when they where.  This was the biggest headache, all other problems were mainly just gem interfaces changing, so a easy search and replace matter.

I like passing a hash to mass-assign data in Rails as well as the obvious coding benefits it’s visually more readable too, but this is out weighed by my great dislike of having my sites hacked into.  So it’s a shame things have had to go this way.

Related posts:

1. Ngnix Upload Awesomeness pt2 Eek, a week’s gone by without a blog entry.  The...

I’ve done this mainly to stop paying for hosting twice, but also because if I’m going to be talking about performance I can’t possibly have a slow blog and my options were limited with the Dreamhost account.

My immediate problem being that I wanted to move it straight to being served by Ngnix rather than Apache - not something that Wordpress is setup for.  After a bit of googling I found that someone had solved all the problems I was going to have for me!  This post has blow by blow instructions for setting up a wordpress blog on Slicehost and Ubuntu, I couldn’t have found anything better.

It worked exactly as advertised, but me being me I could help tweaking and updated the nginx site config to include expire header for static assets, like so:

server
  {
 
  listen   80;
  server_name www.motionstandingstill.com;
 
  root   <path_to_site>;
 
  access_log <custom_log_path>/access.log;
  error_log <custom_log_path>/error.log;
 
  location ~* i.+\.(css|js|jpg|jpeg|gif|png)$
    {
    expires      7d;
    }
 
  location /
    {
    index  index.php index.html;
 
    # Basic version of Wordpress parameters, supporting nice permalinks.
    # include /etc/nginx/conf/wordpress_params.regular;
    # Advanced version of Wordpress parameters supporting nice permalinks and WP Super Cache plugin
    include /etc/nginx/conf/wordpress_params.super_cache;
    }
 
  # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
  location ~ \.php$
    {
    fastcgi_pass 127.0.0.1:9000;
    fastcgi_index index.php;
    include /etc/nginx/conf/fastcgi_params;
    fastcgi_param SCRIPT_FILENAME <site_path>/public/$fastcgi_script_name;
    }
  } # server

Later I found someone had posted a different conf with easy support for multiple blogs, but I’ve not moved to using it as what I’ve got works.  What it also does is add the expire clause support like I have done above.

When moving this blog to Slicehost I initially set it up on a temp subdomain to make sure it was working sweet first and because I wanted to try out some different themes and play with a couple of other things.

I quickly discovered that as it was running on a different doman than it was configured for, Wordpress would redirect my browser to the correct domain meaning I couldn’t get into the new site.  Ugh.  So I had to whip out my sql skills and manually update the appropriate settings.

UPDATE wp_options SET option_value = 'http://temp.motionstandingstill.com' WHERE option_value = 'http://www.motionstandingstill.com';

I actually used a different domain name and explicitly updated two rows rather than the generic ‘where’ clause I have, but you get the idea.  This allowed me in which is what I wanted.

Then I took the opportunity to tidy up what plugins I’m using as I’d gotten a bit ‘plugin install happy’ when first setting the blog up.  As well as getting rid of unwanted plugins I installed two new plugins I’d recently found, the first a redirection plugin with regex support and the second a security checking plugin.

Once I’d got everything how I wanted it, just sticking with the same theme (the misty tree photo is mine), I grabbed a final db backup from Dreamhost and restored it on Slicehost then changed my DNS A record.

All in all, much easier than I thought it would going to be thanks to some helpful results from google. :-)

Update: Slicehost has been aquired by Rackspace.

Related posts:

1. Nginx upload awesomeness I’ve been a bit busy earning my keep for the...
2. Using Nginx to send files with x-accel-redirect So far I’ve configured Nginx to handle file uploads by...

For me the reasoning is distilled down to my programming style.

Pragmatism

When it comes down to it, what vital benefit does it give you over :dependent => :destroy or model validation?  Will this give you increased quality faster, quicker and cheaper?

Premature Optmisation

Basically don’t waste time optimising something that isn’t slow or is hardly used.  Rails is so slow that some fractional speedup on the DB isn’t really going to be noticeable by the end user.  Instead examine your site with NewRelic or FiveRuns and fix your bottleneck code - way more bang for your buck.

Data is Valuable

So don’t go around deleting it!  If a user wants to delete some data you should want to know why.  Looking at the data can aid in determining why without having to bug the user about it.  Yes there are occasions when this does need to happen, but they should be the exception to the rule and thus explicitly coded for.

Just mark a record as deleted and filter it out automatically, that’s easy enough to do with rails - way easier than any other language I’ve used before.  Better yet, search github and use someone else’s wheel.

Convention Over Configuration

Basically put, you do things the way that everyone else has done it.  Walk the well walked path so to speak’ and you won’t experience any bizzaro problems that someone else hasn’t already solved before you.  As you’ve created your app in a predictable way, someone else can easily come in and contribute to your project, or visa-versa and it makes the whole Rails community more accessible to you.

So in following the conventions and as you are not encouraged to use foreign key constraints, Rails isn’t expecting data to be deleted out from underneath it.

Don’t Repeat Yourself && Model View Controller

DRY is another of the core philosophies behind Rails - partials are a real good example of this especially when you start to think of them as essentially methods.  On top of the obvious ’speed to deliver’ benefits this concept provides, it also causes you to distill your code down to being more readable and accessible.  This ties in nicely with MVC principles as you then tend to group like logic together which is what MVC is all about.

Foreign key constraints are business logic as they describe and implement a behavior - “when you delete this record, also delete these other records over here”.  So why would you go and put some ‘important to know about’ business logic in an isolated and often hard to reach place?

Documentation

The one great thing I learned from Steve McConnell’s Code Complete was how to write comments, they should describe the intention of the code, rather then repeat the code.  This nicely ties in with the DRY principle.

I’m also a great believer in self documenting code, Rails and Ruby help heaps with this, and generally write comments to help simplify complex code.  Instead of my code being littered with redundant comments cluttering everything up, it just contains important ones as they are more likely to be read and updated when the code is next worked on.

If you use foreign key constraints in your DB you’ll likely end up documenting them (ie repeating them) somewhere more accessible, in the models being an obvious place.  That wouldn’t be because the business logic lives there would it?.  Have a think about that.

The Rails Way

Looking at the Convention over configuration, Don’t Repeat Yourself and Model View Controller concepts above - as much as Rails is a framework it’s also about a way of doing things.

Over the years I’ve seen a notable number of people, new to rails and these concepts, trying to make rails work their way - often with less than desirable results.  If you embrace Rails and it’s ‘way’ then it works beautifully and is a pure joy to use, rather than a frustrating experience.

For me

Coding is an art form and as such a creative process.  If something doesn’t look, or even feel right then it’s wrong.  In the end foreign key constraints on a web app just rub me up the wrong way.

Related posts:

1. Starting simple with Rails caching So, you’ve launched your website and increasingly people are actually...
2. Be careful how you count in Rails I’ve just encountered this problem yet again effecting code I’m...
3. Spandex Memcache Store is now in Rails 2.3 Just a quick note to say that the local cache...