Last year I started a personal project to up-skill in some aspects of web development I was lacking in, like Javascript.  I had an idea for a game I’d been bouncing around for several years so I created that.  One thing led to another and now it’s become quite a serious project.

The main page for ESDAO gives an overview of what it is about, you can use this invite link to take the game for a spin.

There is a small base of committed testers enjoying the game – but it’s not really growing much and I’m kinda stumped about the best way to fix that.  I have some ideas but I have little experience in this type of thing and would love to get advice from someone with a successful background doing just that.

The game itself is close to completion and my next major tasks are getting more people playing it and sorting out the look and feel.  I have a designer lined up, but have yet to commission a site design while I work out what to do next.

So, I’m after an advisor who I can meet on a regular basis.  Someone who I can pay for good objective advice.  I’m based in Wellington and would obviously prefer someone local.

If you’d like to know more about me I have a page for that.

I’m best contacted via email – nahum.wild@gmail.com

No related posts.

I was already quite busy when everything stepped up a couple of gears around February due to going into business with  ProjectX and thus my friend John Clegg.  I’m slowly taking over the back end of the business while he focuses on the front end.  Essentially I get more toys and larger projects to play with :-)

To be realistic I reckon I’ll only get out a blog entry once every couple of weeks at this stage – which will be hard as there’s an increasing amount to blog about.  Top of my list is Nginx with the recently updated Phusion Passenger and Ruby Enterprise Edition which is powering our new Redmine server.

Stay tuned.

No related posts.

So with that in mind I’ve setup a new google group, WellOptimised.  If there is enough interest I’ll organise meetings for those here in Wellington.

If you are interested pop along via the above link and introduce yourself.

No related posts.

The plugin is still useful as I didn’t submit it’s session caching changes to Rails – essentially using Rails.cache for session storage rather than establishing a second connection to MemCache.  This also benefits from the local cache changes.

Where next?  In the next week or so I plan on updating the plugin by removing the local cache functionality to prevent any conflicts with Rails 2.3 when it’s released.

After that, I’m planning on working through the list of changes detailed under Future Plans in the plugins readme, also listed below.

• TESTS! (I wrote a bunch for the patch, just need to do that for the remaining session changes)
• Make this a Gem
• Conditional caching
• Multi-Get
• Dynamic key helpers
• Make the code more awesome

No related posts.

I received two suggestions for the ‘best install method’, firstly MacPorts and then later pre-compiled dmg’s from Kyng Chaos.  I already use ports extensively making that choice easy for me – I like being able to  uninstall things if they don’t work to well and was concerned the dmg option wouldn’t allow me to.

sudo port install postgresql83 postgresql83-doc postgresql83-server postgis

I then added the following to my path in ~/.zshrc so I could actually run things

/opt/local/lib/postgresql83/bin

and started the server.

sudo launchctl load -w /Library/LaunchDaemons/org.macports.postgresql83-server.plist

All seemed to be fine and dandy as I could connect with pgAdmin.  I quickly discovered that I needed to create a user with the same login as my osx login when I tried creating a test db.

Next I tried restoring my client’s db dump and it barfed big time.  As I’ll be working on some mapping software I’d installed the PostGIS package which seems to have failed to link up with Postgres properly.   I was getting the follow missing library error:

could not load library /usr/lib/postgresl/liblwgeom.so.1.3

Obviously it’s looking in the wrong place for a start, so I setup a LIBDIR export in my profile to get over that problem and restarted the server.

export LIBDIR=/opt/local/lib/postgresql83

That still didn’t solve the problem though as I don’t have a file by that name on my system, I do however have liblwgeom.1.3.so which looked to be the same thing.  Pretty annoying, but easily solved with a symlink.

cd /opt/local/lib/postgresql83
sudo ln -snf liblwgeom.1.3.so liblwgeom.so.1.3

Then the backup restored like a chalm.

To get Rails talking to Postgresql I needed to install the postgres adapter gem appropriately called ‘postgres’.  I didn’t expect that to work straight off though as I’d installed postgresql with ports and the gem by default would be looking in the wrong place for it.  I was correct, but I also received an additional error complaining about the architechture not being specified – I needed to tell it I wanted i386.

sudo env ARCHFLAGS="-arch i386" gem install postgres -- --with-pgsql-include-dir=/opt/local/include/postgresql83l --with-pgsql-lib-dir=/opt/local/lib/postgresql83/

I gropped around for a bit trying to figure out the correct way of letting it know and eventually found the above solution on code snippets.

No related posts.

Since I occasionally have quite a bit to say, like in my previous entry or this one about where business logic should reside, I’m now only showing excerpts on the index page rather than whole entries.  I’ve also gone though and added in a couple of helpful plugins such as auto suggesting related posts and easy social bookmarking.  My next step is to change the source code syntax highlighting css – I’ll probably just use the Textmate bundle in Textmate.

Currently the homepage ySlow score is 79, I can get this to 85 if I concatenate and minify the css as doing this reduces size and number of requests for empty caches.  This hasn’t happened yet as it’ll require a bit of hand tweaking due to not working out of the box.

Overall I’ve gone for a more clean, simple and wider look by using Acamas which is based on the opensource Thematic framework.  It looks great on a Mac, pretty good under Windows and terrible in IE6.  If you experience any problems, in particullar with the rss feed (now thru feedburner.com) please drop me a line.

Update: I’ve gotten the ySlow score up to 90 now as the css is contenanted and minified plus the sidebar images are now being hosted locally allowing proper expires headers.  With the css files I’d simply been adding them together in the wrong order.  There is still a problem with css expressions though, but this will go away when I drop the syntax highlighting plugin – further boosting my yScore.

No related posts.

You’ve noticed that your app is doing a lot of repetitive work – calculating and showing the same data over and over again.  This doesn’t fit with your adherence to DRY principles at all.  If you cached repetitive responses your servers could give you way more bang for you buck – and it would only cost your time.

You are about to embark on an exciting journey fraught with hidden dangers, so let me prepare you – cheesy I know but hey.

First, you have actually done some code optimisation haven’t you?  Stopped using the default mysql config?  Checked your slow query log?  Used pl_analyise to parse your production logs?  Fixed the popular, but slow actions?  Done some eager loading so you get everything in a couple of hits from mysql rather than many many many little hits as your views render?  When you dream you see log tailing in the background?  Minified your javascript and css?  Static assets have expire headers?  Don’t have too many Mongrels running?  Sessions are stored in MemCache? Go do all that if you haven’t and only come back here if this are still too slow.  Oh and try to get your yslow score above 75.

Note: I’m assuming that you have read a bit about caching in Rails already and have had a play, reading fragments from actions etc…  If not try googling as there are a number of great articles on it, the Agile book is probably a good start too.

Start at the end

Instead of just saying ‘do this awesome thing’ I’m gonna explain how I’ve gotten to saying that, as you shouldn’t be making blind decisions about these things.  Most of the time when I implement caching for clients I completely skip page caching and action caching and go straight to line item caching, ie caching of small fragments of html.  Sounds completely counter-intuitive doesn’t it, but it’s not.

First off I need to define a typical client’s website so you see where I’m coming from.  It’s a user driven web 2.0 website – so most of the data is user generated and sometimes social networking elements are included, where a user’s data can be restricted to just their friends instead of being publicly available.  Basically the site lists stuff that people put on it.

Why not page or action caching?  As a starting point it’s too hard to manage due to the continually changing nature of the data being presented.  This changing nature means your cache might not actually get used that much.  Additionally when you are logged in your name is typically on the page with other unique to you info – you have x messages etc…  The latter reason completely rules out caching a whole page for anyone that’s logged in.  Don’t restrict caching based speed increases to joe public as logged in people, ie people that give a damn about your site, shouldn’t be second class citizens.

Don’t start with caching lists

Developers still want to jump straight in and cache lists of items (and pages too) rather than the individual line items – basically as much of the page as they can in a single hit – thinking they’ll get great bang for their buck.  When you deploy with that strategy you’re in for a world of hurt.  This is because they’ve not realised they need to take the following into account.

Each time one of the line items changes you have to expire the cache for the whole listing.  If you have pagination then you’ll likely have to expire ALL cached pages unless you can 100% reliably work out which page the item is on.  So yes you have a separate cache entry for each page of the listing.  If someone adds or edits a line item you have to expire the whole lot – and thats what people should be doing all the time on your site right?.  Additionally, if you allow sorting or a filtering mechanism you have to uniquely cache by page and the additional dimension(s), thus further reducing the frequency the cache will be used.

What’s the goal?

What we are trying to achieve with caching is to DRY up the execution of requests, but in a way that doesn’t add complexity.  The idea is to let your servers focus on the unique stuff like processing credit cards and telling Jill’s friends that based on her network and comment history she really doesn’t like doing things with Sarah unless Joe is there too.

Keep this in mind.  With seeking something to cache, you are basically looking for something that has a high ratio of renders/requests of the exact same html in between changes of that html.  So the more content in the cached fragment, the more frequently you’ll need to expire it.  Putting that another way, the fewer data sources, instances of models and display dimensions like pagination, sorting and filtering used to make up the cached html the better.  The simplest you can go is one data source, ie a line item.  Remember, keep it simple.

Put in business speak, it’s a cost / benefit decision.

It’s all about the keys

When you store fragments in your cache you name them with a unique string based key so you can access them later -  exactly like a hash.  So yes, they are important.  Typically they looks like a url, sort of based on the actual url or controller/action and the dimensions involved to make it unique to the html being cached.  So for the above listing situation the key should look something like this:

‘/products/list/sort=price/status=in-stock/page=1′

This actually presents a completely separate reason as to why you shouldn’t do list caching when you start out on this path.  The problem with this key is that to expire it you have to count the number of products to determine how many pages there are and then manually go and expire all possible keys that could have been generated, one at a time – this is because expiring via regex is not supported.  If additional dimensions are involved the problem will compound even further.

Yes, that’s real ugly and will become completely unmanageable.  There are some clever tricks to help out there such as self expiring keys, but that’s a different topic which I’ll get to at a later date.

When to expire

Wait there’s more!  You have to monitor data changes so you can expire those keys when they become stale as it doesn’t happen magically.  Normally you’d setup a sweeper (observer) and just expire things when ‘after_save’ is triggered.  Not a problem.  If you have data from multiple models involved in your cached html fragments then you need to monitor all the models involved.  You can go as far as only expiring particular keys based on the attributes that are changing etc… now much easier in Rails 2.1 with Dirty Objects.

This is all adding yet another layer of complexity.  Ugh.  It should all be documented so that other developers know about it especially new developers – otherwise further work could mess it up and you might start showing stale data to users.  In my book that’s pretty much on par with errors occurring as you could be showing the wrong price or even deleted data to users.  Unless you are careful about it, it can all become some really hard to manage mess which really isn’t going to give much benefit to your users.

Line item caching

As you are not having to worry about other dimensions such as pagination, sort order and filtering, other line items or even if the user is logged in or not, the line item’s cache key should be something like this:

‘/products/#{product.id}/line_item’

To make sure it expires correctly all you have to do is setup a sweeper watching the line item and simply delete it’s key when it changes – use Rails.cache.delete for this.  If the line item is rendered using data from through relationships, you then have to monitor them too.  I’ve seen people putting a ‘last_active_at’ field on a line item and when anything related to it changes they update that time field.  That works nicely and things are kept simple.

Finally

That should set you off in the right direction, remembering though this is a starting point for your first caching strategy.  Later on you’ll actually start doing multiple layers of caching and covering more complex html, such as lists, if your site continues to grow.

Now hopefully I’ve convinced you to start out with caching the html for a single line item – or something just as simple.  If not post a comment and tell me why not.

No related posts.

A enhanced version of and replacement for the MemCacheStore shipping with rails.  It uses a per-request local cache to buffer duplicate memcache reads which can result in halving read requests, and it uses a single connection to memcache for both the cache and session stores.

It doesn’t yet have the awesome key creation and maintenance library that I showed as that has evolved quite a bit and needs a big bunch of love first – which I’m working on.

There is now a dedicated page on this site for the plugin, currently it’s holds a copy of the plugin’s README.  My intention is to update the page with examples etc.. as I extend the plugin.

No related posts.

The problem was this: somewhere along the line attachment_fu was updated to execute the following line of code on classes using the has_attachment directive:

attr_accessible :uploaded_data

This is taking advantage of a security feature in Rails, it instructs the model to only allow :uploaded_data to be set through update_attributes and any like methods such as new and create.  So whitelisting bulk settable attributes.  It doesn’t raise any errors, just warnings in your log like this:

WARNING: Can't mass-assign these protected attributes: <attribute_name>

Due to no errors being raised the problem was presenting itself in other parts of the application as expected data was not being found.  Fortunately I was tailing my dev log and noticed the above warning message.

The problem in this situation was the historic code creating instances of the model by passing a hash through to SomeModel.create.  As this area of code was not utilising data posted to an action it was not a security risk.  Very fustrating, especially as this was common practice throughout the codebase.

As I didn’t want to do a huge bunch of re-keying my initial solution was to comment out the offending line in attachment_fu.  Yes this is a big nasty fix but there was time pressure, other problems to solve and I didn’t want to be held up having caused an unnecessary amount of manual testing.  In case it was forgotten about  and overriden with a future plugin update, a quick test with big comment was added.

This project hadn’t updated it’s gems and plugins for a while, meaning it was a big shock to the system when they where.  This was the biggest headache, all other problems were mainly just gem interfaces changing, so a easy search and replace matter.

I like passing a hash to mass-assign data in Rails as well as the obvious coding benefits it’s visually more readable too, but this is out weighed by my great dislike of having my sites hacked into.  So it’s a shame things have had to go this way.

No related posts.

I’ve done this mainly to stop paying for hosting twice, but also because if I’m going to be talking about performance I can’t possibly have a slow blog and my options were limited with the Dreamhost account.

My immediate problem being that I wanted to move it straight to being served by Ngnix rather than Apache – not something that Wordpress is setup for.  After a bit of googling I found that someone had solved all the problems I was going to have for me!  This post has blow by blow instructions for setting up a wordpress blog on Slicehost and Ubuntu, I couldn’t have found anything better.

It worked exactly as advertised, but me being me I could help tweaking and updated the nginx site config to include expire header for static assets, like so:

server
  {
 
  listen   80;
  server_name www.motionstandingstill.com;
 
  root   <path_to_site>;
 
  access_log <custom_log_path>/access.log;
  error_log <custom_log_path>/error.log;
 
  location ~* i.+\.(css|js|jpg|jpeg|gif|png)$
    {
    expires      7d;
    }
 
  location /
    {
    index  index.php index.html;
 
    # Basic version of Wordpress parameters, supporting nice permalinks.
    # include /etc/nginx/conf/wordpress_params.regular;
    # Advanced version of Wordpress parameters supporting nice permalinks and WP Super Cache plugin
    include /etc/nginx/conf/wordpress_params.super_cache;
    }
 
  # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
  location ~ \.php$
    {
    fastcgi_pass 127.0.0.1:9000;
    fastcgi_index index.php;
    include /etc/nginx/conf/fastcgi_params;
    fastcgi_param SCRIPT_FILENAME <site_path>/public/$fastcgi_script_name;
    }
  } # server

Later I found someone had posted a different conf with easy support for multiple blogs, but I’ve not moved to using it as what I’ve got works.  What it also does is add the expire clause support like I have done above.

When moving this blog to Slicehost I initially set it up on a temp subdomain to make sure it was working sweet first and because I wanted to try out some different themes and play with a couple of other things.

I quickly discovered that as it was running on a different doman than it was configured for, Wordpress would redirect my browser to the correct domain meaning I couldn’t get into the new site.  Ugh.  So I had to whip out my sql skills and manually update the appropriate settings.

UPDATE wp_options SET option_value = 'http://temp.motionstandingstill.com' WHERE option_value = 'http://www.motionstandingstill.com';

I actually used a different domain name and explicitly updated two rows rather than the generic ‘where’ clause I have, but you get the idea.  This allowed me in which is what I wanted.

Then I took the opportunity to tidy up what plugins I’m using as I’d gotten a bit ‘plugin install happy’ when first setting the blog up.  As well as getting rid of unwanted plugins I installed two new plugins I’d recently found, the first a redirection plugin with regex support and the second a security checking plugin.

Once I’d got everything how I wanted it, just sticking with the same theme (the misty tree photo is mine), I grabbed a final db backup from Dreamhost and restored it on Slicehost then changed my DNS A record.

All in all, much easier than I thought it would going to be thanks to some helpful results from google. :-)

Update: Slicehost has been aquired by Rackspace.

No related posts.