Nefarious Designs

Eliminating Web Development Waste

2012-05-28T09:02:00+01:00

The concept of waste within web development might seem reasonably clear to most managers and developers, however in my experience it is something that most teams need help to identify. I’ve recently worked with a team who became exceptionally good at identifying waste and, as a result of this alongside a whole swathe of other techniques, were by far the most efficient agile team I’ve ever worked with.

Eliminating waste is the first—and most fundamental—principle of Lean Software Development; something that was introduced to me by an exceptionally talented Project Manager whom it was my pleasure to work with. Lean Software Development is a set of seven fundamental principles that really work hand in hand with agile development to improve software delivery. Those principles are: eliminate waste, amplify learning, decide as late as possible, deliver as fast as possible, empower the team, build integrity in, and see the whole. In this article I’m going to look at how we can improve our ability to identify waste and iteratively remove it from our development chain.

Lean manufacturing

The principles of Lean Software Development are basically reinterpretations of the principles of Lean Manufacturing as designed and tested by Mary and Tom Poppendieck. Those original manufacturing principles—formulated by Sakichi Toyoda, the founder of Toyota Industries, and his son Kiichiro Toyoda, the founder of the Toyota Motor Corporation—were realised as an actual method of production by the Toyota engineer Taiichi Ohno. This later came to be known as both Just In Time Manufacturing and the Toyota Production System.

Lean Manufacturing is a production practice that considers the expenditure of resources for any goal other than the creation of value for the end customer to be wasteful, and thus a target for elimination. Working from the perspective of the customer who consumes a product or service, “value” is defined as any action or process that a customer would be willing to pay for. At its most simple, Lean Manufacturing can be considered a move towards manufacturing to order rather than manufacturing to build stock since stock is essentially waste.

In reading descriptions of American supermarkets, Ohno saw the supermarket as model for what he was trying to accomplish in the factory. A customer in a supermarket takes the desired amount of goods off the shelf and purchases them. The store restocks the shelf with enough new product to fill up the shelf space. Similarly, a Toyota work-center that needed parts would go to the inventory storage point for the particular part and withdraw the quantity it needed. The storage point would then be restocked by the work-center that produced the part, making only enough to replace the inventory that had been withdrawn.

The Toyota Production System

The key objective of the TPS was to design out overburden (muri) and inconsistency (mura), and to eliminate waste (muda). A manufacturing process that removes inconsistency is capable of delivering smoothly. However, that process also needs to be as flexible as possible so as not to introduce overburden which creates waste. Also there is clear value in the elimination of waste in any process.

TPS Reports

Anyone who’s worked in the industry for a while, or has seen the film Office Space, will be familiar with the term “TPS reports” being synonymous to pointless paperwork. TPS reports have nothing to do with the Toyota Production System, the TPS stands for “Testing Procedure Specification”.

That being said, TPS reports as pointless paperwork is entirely relevant as waste that could be removed from your processes. We’ll come back to that later.

Web vs. Software Development

Before we get too involved in investigating the waste element of this post, let’s just examine the difference between general software development and web development:

I consider myself to be a web developer and a web development manager. I am used to working with the internet as part of my development landscape and also as part of my toolkit. This means I am used to dealing with high numbers of users, a vast array of user-agents which interpret my code and therefore need to be tested against, a degree of uncertainty as to the capabilities of the user-agents and the users themselves, and a wide number of technologies. This has been described by Douglas Crockford—the inventor of JSON, and often described as the godfather of JavaScript—as the “most hostile environment imaginable”.

This is not to say that software development is any less hard or varied, just that there are a number of specific complications put on me as a web developer; and there are a number of very specific web-sourced problems that I’m used to solving. It’s simply different because it specifically involves the epic WAN that is the Internet.

Lean Software Development translates very well when dealing specifically with web development since the same problems exist in both development life-cycles. What really varies is how we might implement a solution to those problems. Taking that into consideration, be aware that I’m writing this article as a manager of web developers not as a manager of software developers.

Waste in Web Development

The TPS identifies seven types of manufacturing waste:

Overproduction
Waiting
Transportation
Extra processes
Overstock
Motion
Defects

In relation to software development, however, it’s clear that not all these types of waste translate. Lean Software Development identifies the following types of waste which translate perfectly between software and web development:

Partially complete work
Extra processes
Extra features
Task switching
Waiting
Motion
Defects

Since some of those are not immediately clear as waste, let’s examine them in more detail…

Partially complete work

It’s fair to say that all web development is partially complete at some point in its lifecycle. The priority here is to minimise the amount of partially complete work that is within a project at any given moment.

Partially done web development can quickly become obsolete as it is replaced with a differing method or solution. The only way to be sure what problems exist in a piece of code are to have it fully integrated with the rest of the project. If you are working behind feature branches or feature toggles, or some other method for cloistering partially done work, you are suffering from your isolation as much as you are benefiting from it.

In fact, the only place where you will capitalise on your investment in a feature is when it is in production. Partially done development ties up resources in investments that have yet to yield results. You need to provide integration test coverage, end-to-end test coverage, integration code, and a whole host of other methods of ensuring the minimum of bugs exist when you finally release it from its isolation. If your feature never actually makes it to production, you have some significant investment to write off. Reducing the amount of partially complete work within your project is both a reduction of waste and risk.

Extra processes

We all love our processes and documentation, right? I’d be willing to bet that just about everyone who has worked in an agile organisation of some form has become disillusioned with the amount of meetings or breaks from focus they have to attend as agile “ceremonies”, and the added work dealing with documenting user stories, their requirements, and how they might be tested and accepted. In fact, two of the tennets of the agile manifesto are specifically related to process and documentation:

We have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

At every opportunity you should be assessing whether a particular process or document is entirely necessary for your development. Extra processes and documentation consume resource, slow down response times, and can mask problems elsewhere. What’s more they break focus, which is something I’ll come back to when I explain “motion” as a type of waste.

Many development processes require paperwork of one form or another. Most of the companies I’ve worked for recently have replaced actual hard-copy with wiki documentation these days, but those documents are still there. Documents for release processes, documents for sign-off, documents for traceability, not to mention documentation for security or compliance.

The real test of these documents is whether someone is actually waiting for it as a deliverable, and whether it then adds value to their process. If not, dump it. Otherwise, find ways to simplify that process. Reduce the amount of input required by using templates or pre-designed input forms. Condense the amount of information required. In fact, there should always be an ongoing process of refinement and efficiency evaluation for all documents and processes. Any required process should be as efficient and effective as possible. Rapid yet easy to validate.

In terms of documenting code, do it as you go. Write clean, easily understandable, easy to maintain code and enforce style with static-code analysis and code reviews. Make sure code is commented effectively and you’ll cut down on cost. It’s important to remember that sometimes documentation is required for code when knowledge needs to be shared over a wide base, but if you’ve minimised what other documentation is required this will be the exception to the norm.

Extra features

Adding in extra features to a piece of development is over-engineering; don’t do it. There will be many times that you foresee the need for an extra feature before you actually have the requirement. There are other times that a developer might see the opportunity to add something extra that may save time for others later on. Whilst at first glance this may seem relatively harmless and of benefit in the long run, it is almost always a source of waste.

All code in the system should be tracked, reviewed, tested, and integrated every time it is touched. What’s more it then needs to be maintained for the life of the project. With that in mind, it is important to remember that every bit of code adds complexity and a new possible point of failure. It also introduces something new for all the developers to understand and—worse—misunderstand and misuse.

If there is no requirement for a piece of code now, don’t write it. Putting it in is waste. Resist your urge to over-engineer.

Task switching

The fastest way to complete two projects using the same people is to do it one at a time, sequentially. Trying to run two projects in parallel using the same resources introduces a large amount of waste. Likewise, sharing people between teams is also wasteful.

Much like extra process, task switching breaks focus. Every time a developer, QA, IA, or designer have to switch tasks, there is a fair overhead to pay. These sorts of tasks require enough focus to warrant a significant amount of time to get into “the zone” or “flow”. Thus, switching between tasks incurs waste through time.

Similarly, being a member of two teams introduces interruptions that might be entirely unrelated to the person’s current task. This means even more task switching, and probably at an even more granular level. For this reason alone it is unwise to assign people to multiple teams.

To illustrate how much task switching can affect delivery time, list 6 peoples names. Firstly, time how long it takes to write each name one after the other. Next, time how long it takes to write a letter of each in turn until you’ve written all 6 names completely. You’ll find the latter takes significantly longer than the former. This is only the simplest form of task switching and yet the waste incurred is quite clear.

Back in 2009, Paul Graham wrote a fantastic piece on focus entitled “Maker’s schedule, Manager’s schedule”. In it he examined the difference between time schedules for developers and managers. It’s important that all managers understand that their schedule is significantly different to the developers that report to them.

A developer works in larger units of time than a manager, with much longer between breaks of focus; usually something in the region of a half-day unit. They dive into work as soon as they arrive at the office and will break focus at lunch. Likewise, when they return for lunch they’ll dive back in and only come back to the surface when it’s time to leave the office. Usually they’ll arrange their tasks in chunks that align to this and therefore will complete tasks shortly before lunch or shortly before they’re ready to leave for the evening.

Managers work in shorter units; usually around 1 hour in length. This allows time for meetings and context switches in between. As a manager it’s your job to context switch effectively, although even the greatest manager in the world will struggle to do it for too many subjects. This is why it’s important to make sure you don’t bite off more than you can chew in terms of projects to run. If you have, it’s time to start delegating.

If you’re a manager and you want effective makers, you need to arrange your days to suit them, not you. Allow 30 minutes in the morning for email checking and an added 15 minutes for the daily stand-up meeting. Don’t bother them again until directly before or after the break for lunch or at the end of the day. Trust me; they’ll cope with this much better than continuous interruptions throughout the day.

Likewise, if you’re a developer, be strict about what you do at these key times during the day. Make sure you don’t kick off development before your stand-up or you’ll fail to cope with the interruption. If people can’t agree to arrive at the office at around the same time in the morning, then you’ll be better off as a team scheduling the stand-up around lunchtime.

All this might sound obvious but you’ll be surprised how many teams let this sort of misalignment between team members screw up their daily flow. It’s also worth noting that the larger the team, the harder it is to align this stuff. There’s definitely a sweet spot to be found in team size and I personally believe it’s around 6-8 members.

Waiting

Day to day development is thick with delays. It’s a standard part of our working lives for projects to be delayed at the start, awaiting requirements or documentation; for the actual development to be blocked awaiting testing or deliverables from IA and UX; or to delay delivery through an inefficient release process.

In web development, delays can be a serious form of waste. They prevent the completion of work and they prevent the capitalisation of investment in the development itself. The speed at which we can respond to a customer-driven requirement is directly related to the number of delays in our development life-cycle.

As soon as we can accept that delays are a significant form of waste, we can identify where those delays occur and optimise them out of our processes. One of the seven principles of Lean Development is “decide as late as possible”. Taking that into account, it is better to re-prioritise tasks based on the delays you are likely to experience than suffer significant blockages in your delivery. This might mean putting tasks off until you have more information or free resource, or it might mean rearranging tasks so that developers are working on minimising those delays before delivering their original priority.

The most common delays in web development are when work passes between individuals. There’s nothing worse than finding out you’re prevented from completing a task because the wireframe or design hasn’t been completed or the requirements haven’t been finalised due to poor prioritisation. It is important that, as a team, you don’t agree to work on more than you can complete and similarly, it’s also important that you don’t agree to work on stories or tasks that have outstanding dependencies or requirements. Sometimes taking the time to groom your backlog as a team can mean the difference between wasting time on waiting or a user story or task flowing through your process painlessly.

Further to that, you can often reduce the time waiting for a deliverable by eradicating the deliverable altogether. I’ve had great success in the past by removing the need for wireframes and design visuals from the IAs and designers and getting them interacting with the developers on a regular basis. If the designers and developers maintain a clear style-guide with clear links to the front-end code—and at best a library of corresponding modules of code—they can easily relate new features to this through “pair programming” for an hour or two. Remove the requirement for a deliverable and you’re only left waiting for the developer and designer to have some time together in their calendars. This should be significantly easier to achieve since both parties are no longer required to deliver anything but their expertise during the interaction.

Finally, it’s important to remember that implementation is everything. If you can’t implement quickly following a key decision then you can’t delay that decision either. Empowering your team to deliver quickly and efficiently is your primary concern at all times, whether you are a manager or a team member.

Motion

Agile software development favours colocation for teams. Likewise, the agile manifesto states that we should value customer collaboration; involving them in all discussions and decisions throughout the development process. The point of both these principles is the reduction of “motion” within the team.

It is important to make sure that developers, QAs, IAs, designers, product owners, project managers, and any other involved parties are immediately available to each other. Reducing the amount of time it takes for specific development-related questions to be answered is vital. Having everyone next to each other reduces that overhead.

If you can’t colocate people, the next best thing is to make sure they have easy communications as soon as they need them. Get everyone hanging out in a Jabber or IRC room, and make sure that’s as much a social experience as it is a work experience or they won’t come back. It should really imitate the local working environment so is likely to involve both work-related conversations and more water-cooler-type discussions and jokes. Expect that; nurture it; it’ll bring the team closer together.

It’s also important to remember that it’s good to put a name to a face. If some of your team are remote, make sure you get them over for social occasions once in a while. It’s good to break bread with your colleagues, and it’s easier to foster feelings of camaraderie when you’ve all been out to the pub together.

Similarly to social interactions, artefact interactions can also suffer from waste in motion. Reducing the amount of time it takes to pass a particular artefact of work from team member to team member can significantly improve the efficiency of the whole team. Good examples are the hand-off of design deliverables like style guides, wireframes, and visuals. Placing these in a known shared location using source control means all interested parties can have access to them immediately. Similarly, making sure tests are written and run in the same places regardless of whether they are written by developers or QAs means that there will be a clear vision of coverage for the whole team.

Motion might seem like a fairly inconsequential form of waste with a minimum of impact, but really it combines with task switching to be quite a serious problem. A competent team will work with this at a minimum from the outset, and most teams can address this issue quickly to improve their performance.

Defects

Defects are an obvious form of waste, but really it’s only the undetected defects that cause a project to suffer. That might sound obvious at first, but it’s important to remember that the amount of waste generate by a defect is directly proportional to both the bug’s severity and the amount of time it goes undetected. A critical show-stopping defect that is identified and addressed quickly generates significantly less waste than a small insipid bug that hangs around for months undetected.

Reduce the impact of defects by finding them quickly through thorough testing practices and QA. Make sure the whole team are involved in that; quality is too important to be the responsibility of anyone except the whole team. Practices such as test-driven development and continuous integration can massively improve the time it takes to detect issues. It is important to understand what you are trying to achieve by adopting these practices and implement what you can.

Summary

Understanding waste in your development process can help you improve the efficiency of your delivery. Being able to take the principles of lean development and integrate them into your agile teams are a clear way to improve your life as a manager. In my experience it’s almost impossible to implement agile within an organisation without truly understanding the whys and hows. Further to this, it’s everyone in the team that need to buy-in and understand, not just the managers or the organisation. This rings true for Lean Development as well.

This has really been a study of the types of waste discussed in the Poppendieck’s various books on Lean Software Development. If you’re interested, I recommend you buy at least one of the various volumes. They’re absolutely worth it.

This is the first of a couple of articles I’ve started writing around agile and lean development practices and hopefully it’s at least got you thinking about how you can improve your life as a developer or manager. In the future I want to look at the complications around writing user stories, around implementing good testing practices, and how systems-thinking should be a primary goal of all teams.

Flexible Responsive CSS Grids

2012-03-28T21:47:00+01:00

There are many ways to build CSS column grids; all of which have their pros and cons. In fact CSS3 has begun to implement a built-in model for columns which will be most useful in the future. However, until all the browsers our users surf the interwebs with are capable of this, we’ll have to rely on the good old CSS2 tricks to do it.

For a good while I used defined width floats for my grids. This means you have to calculate the width of each column and fix it (either with pixels or something more flexible like ems or percentages). That method is flawed due to rounding errors and general problems with flow when resizing. A better method is to use a mix of floats and negative margins for positioning. When I need to create a CSS column grid, I use this method.

However, when I was recently asked to share my method—which was outlined to me by a good friend and CSS guru—I had to remind myself how to do it. These are the perils of becoming a manager and developing less. In the interests of remembering next time (and also sharing the knowledge), I’ve decided to document the method in this article.

A word on frameworks

Just a quick word really. There are frameworks out there that will do your grids for you. I know they exist but I do not use them. Often they result in bloated mark-up and esoteric element identifiers that create unmaintainable code. Your mileage, of course, may vary.

I’m capable of creating a grid with the minimum possible code required. Hence, for production purposes, that’s what I do. In a prototyping situation, when I need something quickly and painlessly, I may choose to favour a framework; but it hasn’t happened yet.

If you choose to use a framework to set-up your grid, this article probably isn’t for you.

Source order is important

It’s good to make sure your main content is first in your source order. It’s helpful for accessibility, and it’s (a bit) helpful for SEO. This is something that isn’t always easy to enforce when using traditional grid techniques. The technique I use is actually built around this concept.

Building the mark-up

The simplest possible mark-up we would want for a three column layout would be something like this:

<div>
    Main content
</div>
<div>
    Secondary content
</div>
<div>
    Navigation
</div>

To make it a little easier to target, and to make it a bit more readable, we can add some element identifiers:

<div class="main container">
    Main content
</div>
<div class="related container">
    Secondary content
</div>
<div class="navigation container">
    Navigation
</div>

This gives us three containers for our columns. If we were to use the simplest possible method of floating each column left and fixing their widths, we’d have ourselves a column grid. However, that grid would not respond reliably cross-browser and could pick up wrapping errors. It also requires exact knowledge of the required width of ALL columns. Depending on how we’d like our grid to respond to change, this may be suboptimal.

Floats

A floated element is shifted to either the left or right of its containing box and surrounding content then flows around it. Floating multiple boxes stacks them up either left or right, which is useful behaviour when attempting to simulate columns. However, if there is not enough horizontal room for the float, it is shifted downward until either it fits or there are no more floats present.

Floats are taken out of the normal flow of the document. This means the positioning model is changed somewhat and we can use that to our advantage.

The trick

Applying a negative margin to a floated element allows us to control how it makes use of empty space around it. We can make use of this if we add an extra wrapper to our main content:

<div class="container">
    <div class="main">
        Main content
    </div>
</div>
<div class="related container">
    Secondary content
</div>
<div class="navigation container">
    Navigation
</div>

To make two satellite floats react correctly we need to make sure that our newly added container is also taken out of the normal flow of the document with a float. However, we don’t want surrounding content to flow around it so we should fix its width to 100%.

.container {
    float: left;
    width: 100%;
}

Next we want to put some space around our main content using standard margins:

.main {
    margin-left: 150px;
    margin-right: 200px;
}

Now let’s fix the width of those satellite columns:

.navigation {
    width: 150px;
}
.related {
    width: 200px;
}

This will result in our three containers floating alongside each other. However, because the first container is set to a width of 100%, the two satellites drop down beneath it.

We want the satellites to take up that nice clear space we’ve created with the normal margins on our main content within the first container. To do that, we shift them left across the first container using negative margins:

.navigation {
    width: 150px;
    margin-left: -100%;
}
.related {
    width: 200px;
    margin-left: -200px;
}

Here we’ve shifted the navigation container 100% across the width of the container of our floats (in this case, the body of our page). This puts it flush to the left, filling the space to the left of our first floated container.

We then shift the related container only 200px to the left, matching its width and therefore leaving it flush to the right of the first floated container.

This leaves us with a perfect three-column grid. What’s more, since we’re only fixed the width of our satellites and have left the main content in normal flow, it will respond to the width of its container, as will the position of the satellite floats.

Demo and source

I’ve taken the above and scrubbed it up a bit, to create a nice flexible responsive CSS grid demo page. Further to that, I’ve also stuck the code up on GitHub in a “flexible CSS grid” repo. Feel free to do with it as you wish.

Vagrant Virtualised Dev Environments

2012-03-14T09:22:00+00:00

In case you’ve been living under a rock for the past couple of years, Vagrant is the latest development in easily controlled virtualised environments. In their own words:

The vision of the project is to create a tool to transparently manage all the complex parts of modern development within a virtual environment without affecting the everyday workflow of the developer too much.

Vagrant automates creation and provisioning of virtual machines within VirtualBox via the command line, whilst also allowing for easy distribution and reuse across multiple projects. In this article, I’m going to look at how it can make life considerably easier when dealing with development environments.

Previously

Previously I’ve written a couple of articles on virtualised development environments; one on VMWare and one on VirtualBox. Since then I’ve upgraded my model to use Vagrant, which allows me to control my environment per project, and also fixes me up with painless access to Puppet provisioning.

This has completely altered my workflow for development. To help you to understand why, I’m going to walk through how Vagrant works and then look at setting up a Vagrant-powered development environment.

How it works

The base box

The true power of Vagrant lies within its design. The basis of a Vagrant VM is its “base box” which is simply a VirtualBox VM that represents a base install of your system. This base install has the following basic requirements:

VirtualBox Guest Additions for shared folders, port forwarding, etc.
SSH with key-based auto. support for the vagrant user
Ruby and RubyGems to install Chef and Puppet
Chef and Puppet for provisioning support

Thankfully the creation of base boxes has been done for you a thousand times over, and the subsequent base box files are available for download at various places across the interwebs. However, should you require something more specific, creating a base box is pretty undemanding and there are even programs like VeeWee which will automate this process for you.

For more information on building your own base box, take a look at the Vagrant base box documentation.

If you’d rather download one that someone else has created, take a look here:

http://vagrantbox.es

This is a really useful resource set-up by Gareth Rushgrove, the man behind DevOps weekly and also a passionate advocate of Vagrant. If you can’t find the box you need here, create it, host it somewhere, and let Gareth know so he can list it.

The VagrantFile

Once you’ve got a suitable base box available within Vagrant, you’re ready to start using it as the basis for your development environments. The plural is important here; Vagrant allows you to reuse the base box in generating VMs for multiple projects, each with their own set-up. To best understand that, I’ve drawn a very simple diagram:

Whilst you’ll probably use the same base box for multiple projects–my box of choice at the moment is a base install Ubuntu 11.10–you probably want a very different set-up per project. For instance, you might have a PHP project that requires a box running Apache and mod_php whilst a Python project better suits nginx proxying to gunicorn. This is entirely possible in Vagrant.

For each project you create, you’ll have a base source directory. It’s here that we’ll create a VagrantFile to control the environment for that project. This means you’ll end up with a VagrantFile per project:

/my_application/
    bin/
    etc/
    lib/
    var/
    VagrantFile
    …

/my_site/
    etc/
    public/
    templates/
    var/
    VagrantFile
    …

To create a VagrantFile you simply run the following command in your project directory:

vagrant init

This will create a nicely annotated VagrantFile. I recommend reading through it to understand the various settings available for you to configure.

The VagrantFile will allow you to change settings on the VM that vagrant will generate for you; for instance, you might want your VM to only have 512Mb of RAM, whereas another might need 2Gb of RAM. You can also specify what base box to use, what ports are forwarded to the VM, what IP address it is configured with, how the VM should handle networking (bridged, NAT, Host Only etc.), and many other system settings besides.

Provisioning

The VagrantFile also allows you to specify whether you want to provision your VM using Chef or Puppet. In OpsWorld™ [possibly the worst theme-park in the world] this means setting up the software on your VM. Personally, I prefer Puppet for this, simply because it uses an external DSL for configuration which I find easier to read than the internal Ruby DSL that Chef uses. (For an explanation of Domain Specific Languages, see Martin Fowler’s excellent blog post, book, and guide page.)

Puppet automates the provisioning, patching, and configuration of operating system and application components across multiple servers. Vagrant uses Puppet to do exactly that; to set-up, patch, and configure the base box to suit your project. To do this, you need to specify that configuration in Puppet manifests–which are basically scripts or programs for Puppet (hence the .pp filename).

The core of the Puppet language is the resource declaration, which represents the desired state of one resource. Manifests can also use conditional statements, group resources into collections, generate text with functions, reference code in other manifests, and do many other things, but it all ultimately comes down to making sure the right resources are being managed the right way.

For more on Puppet and its manifests, I recommend reading the Learning Puppet documentation. You can probably even skip straight to the manifests section. Also, the resource types cheat sheet is invaluable.

Another useful resource for Puppet is the Puppet Cookbook, which contains a collection of task oriented solutions in Puppet.

I’ve recently started keeping a git repository of the various manifests I use for Vagrant. This way, I can copy them and share them between Vagrant projects depending on what software or set-up I require. You can find that repo on GitHub:

http://github.com/nefarioustim/puppet-manifests/

Be aware, I tend to use Ubuntu server hence package management is handled via apt. This means these manifests will need a bit of editing should you want to use them on CentOS or some other linux distro.

The Vagrant VM

The Vagrant VM will be created according to your VagrantFile. This means there will be one per project. This granularity affords you a degree of independence with the VM, and allows you to treat it as a really powerful isolated root for your project. If you’ve ever used virtualenv when working with Python, imagine Vagrant to be virtualenv squared. Since you’re not sharing this environment with any other project, you can set it up, alter it, reconfigure it, and destroy it as much as you please; and since it’s entirely controlled by the VagrantFile and the provisioning scripts, those processes are easy and powerful.

Once set-up, your vagrant VM contains a default user of vagrant. That user has full sudo access and is generally the user you’ll use for everything within your development environment. When you think about it, there’s no real need for complex user set-up within this VM since you could be setting it up and tearing it down as much as you’re doing development on it. If you do need complex user set-ups, there’s nothing stopping you doing that with Puppet or Chef during the provisioning stage.

The Vagrant VM itself can be controlled at the command line through a number of commands:

vagrant up

The first time you run this command, it will copy across the base box specified in your VagrantFile, boot it up with the specified networking set-up, and then provision it according to your Puppet manifests or Chef recipes. Once the box has been set up, and as long as it isn’t destroy, this command will boot up the box and confirm the provisioning. You can deactivate provisioning on this command using the --no-provision switch.

vagrant ssh

Logs into your project VM as the vagrant user, using SSH keys. This circumvents the need for complicated SSH set-ups on your host machine and affords you all the power you need.

vagrant destroy

This will remove all trace of your project VM from within VirtualBox. This means, the next time you run vagrant up, Vagrant will need to recopy across the base box and reprovision it. Most of the time you’ll only really want to do this if you’ve b0rked your VM somehow, or you’ve made a significant change to your VagrantFile; like specifying a different base box for example.

vagrant suspend

Takes a snapshot of your VM and suspends it, allowing you to vagrant resume later. This saves you having to perform a length vagrant destroy && vagrant up cycle, but also stops the VM from taking up valuable memory and CPU cycles.

vagrant resume

Resumes a suspended VM from its snapshot state. The vagrant up command will also resume if it detects a suspended VM.

vagrant halt

Forces shutdown of a VM. This is basically a last resort shut down of your VM and the folks over at Vagrant recommend attempting a graceful shutdown via vagrant ssh first.

vagrant status

Shows the current status of the VM related to the VagrantFile in the current directory.

vagrant provision

Runs the provisioning scripts without reloading the entire Vagrant environment. If you’re just tweaking or adding some cookbooks, this command can save you a lot of time.

There are a few other Vagrant commands available to you but the above are pretty much the majority of what you’ll use on a daily basis. For the complete list, see the Vagrant commands documentation.

An example Vagrant environment

Now that I’ve run through the workings hopefully you’ve got a better understanding of the power of Vagrant and how it can improve your development workflow. To further highlight that, let’s set-up a basic example Vagrant environment on a Mac, via the command line.

Installation the best way

With the release of v1.0 of Vagrant comes a new set of installation packages, available from the Vagrant downloads section. This is by far the best way to install Vagrant and will sort you out with all the prerequisites automatically. These are now the recommended and supported method of installation for Vagrant.

When I began writing this article, these packages did not exist and, as such, I covered the slightly more complicated installation with all its prerequisites (for Mac). With these new packages, you should be able to follow along with the majority of this tutorial in Windows (albeit with slightly differing CLI commands).

Installation the old way

Apple Developer Tools

You’ll need to either install Xcode or the newly created Command Line Tools (significantly smaller than the full Xcode install and designed for stuff like this), which you can download from Apple (you’ll need a free Apple ID to log in):

https://developer.apple.com/downloads/

Homebrew

Firstly, I’d recommend you install some package management for which I use Homebrew. Whilst this is not required—you can always install from source—it is lightyears ahead of MacPorts (the alternative package manager for Mac) and is actively supported on GitHub. If you don’t have it already, do this:

$ /usr/bin/ruby -e "$(curl -fsSL https://raw.github.com/gist/323731)"

This is the recommended install for Homebrew. If you’re unsure, read the script first. It installs Homebrew to /usr/local so that you don’t need sudo when you brew install. It is a careful script, it can be run even if you have stuff installed to /usr/local already. It tells you exactly what it will do before it does it too. And you have to confirm everything it will do before it starts.

You can then use Homebrew to update your Ruby and RubyGems installs

Ruby and RubyGems

Vagrant is powered by Ruby and is installed as a RubyGem. This means you require both Ruby and RubyGems on your development VM host machine. Both are preinstalled on Mac OS, so we only need to update them:

$ sudo gem update --system

Install the gem

Once you’ve made sure you’ve installed all the requirements for Vagrant, the actual installation is really simple. Just install the Vagrant RubyGem:

$ gem install vagrant

Installing this gem will also install VirtualBox (if you haven’t already).

Bob’s your uncle, Mary’s your aunt, and Dave’s your vicar’s love-slave. You are now ready to start building Vagrant environments, but first you’ll need at least one base box.

Base box

You now have the option of downloading a base box, or you can build your own in VirtualBox using your operating system of choice. Since this is supposed to be a development environment, I’d recommend choosing an OS that is as close as possible to the one you’re going to use in production. For me that tends to be Debian or Ubuntu (which is really just another flavour of Debian), so I’m working with an Ubuntu base box.

Building your own base box

Creating a base box is straightforward as long as you’re confident building and provisioning servers. If you’re a bit unsure but you’re not scared to give it a go, I heartily recommend trying to build your own because it’s certainly a learning experience.

To begin with, get yourself a disk image of your chosen OS. For me, that’s Ubuntu 11.10 Server AMD64. Once you’ve got your image, you’re ready to start setting up a VM in VirtualBox.

My template for new VMs is:

360Mb RAM
10GB VDI disk, dynamically sized
64 bit
- In General > Basic choose Linux > Ubuntu (64 bit) as your OS
- Set System > Motherboard > Enable IO APIC to true
- Set System > Processor > Enable PAE/NX to true
Disable audio
Disable floppy drive

Once that’s set-up, make sure you’ve “inserted” your OS disk image to the CD/DVD drive on the VM, and fire it up. I generally opt for a standard install for my base box, but if your OS of choice offers a low footprint install intended for virtualisation, you may want to opt to use that instead. If you do, make sure you know the differences; it may use a different kernel and therefore not support some of the motherboard and processor settings we discussed earlier.

When the time comes, make sure you use a default user of:

Fullname: vagrant
Username: vagrant
Password: vagrant

Post install

I’ve written a script to handle post install in a similar method to VeeWee. To use this you’ll need to create a .vbox_version file in your home dir:

$ echo "4.1.8" > ~/.vbox_version

You’ll want to replace the “4.1.8” with the relevant version of your VirtualBox installation because this will be used to work out which Guest Additions to install. Next let’s switch to root for ease:

$ sudo -i

You’ll also want to make sure cUrl is installed:

$ apt-get install -y curl

Once you’ve done that, we can download and run the script (raw gist URL shortened for convenience here):

$ curl -L http://git.io/cSGCgw | bash

If you read through the script you’ll see it does the following:

Creates a log of the date and time we built the box in /etc
Installs aptitude
Updates the package list
Upgrades the installed packages
Installs a bunch of packages that are useful for set-up
Installs SSH
Installs the NFS client
Sets-up the correct permissions in sudoers
Installs Ruby and RubyGems
Installs the Chef and Puppet gems and updates the JSON gem
Removes the the gemspec for an older version of JSON because it’s broken
Installs the insecure Vagrant public keys supplied by Mitchell Hashimoto
Makes sure SSH doesn’t use DNS look-ups
Installs the VirtualBox Guest Additions
Zeros out the free space in the image to make it easier to compress
Removes leftover DHCP and network rules

Once the script completes you should be ready to rock. Better shut down that VM:

$ sudo shutdown -P now

Package up your self-built base box

Assuming the VM you created in VirtualBox is called “ubuntu-11.10”, we package it up using the following command:

$ vagrant package --base ubuntu-11.10 --out ubuntu-11.10.box

The --base parameter refers to the VM in VirtualBox and the --out parameter is what we want our newly packaged .box file to be called. Once you have a .box file, you’re ready to start using it with Vagrant–and you’re also ready to share it with other people should you so wish.

Using a pre-built base box

In true Blue Peter fashion, here’s one I made earlier:

http://timhuegdon.com/vagrant-boxes/ubuntu-11.10.box

For other options, as previously mentioned, have a look at Vagrantbox.es.

Making Vagrant aware of your base box

Assuming you’re in the same directory as your .box file, you can add it to Vagrant using the following command:

$ vagrant box add ubuntu-11.10 ubuntu-11.10.box

The first parameter is the name Vagrant will use to refer to the box in Vagrantfiles, and the last is the path to your .box file.

Configuring your VagrantFile

Now that you’ve downloaded or created a base box, and have made Vagrant aware of it, our next step is to configure a VagrantFile for whatever project we’re working on. It’s here, amongst other things, that we’ll specify which base box we want to use.

Change directory to your project and run vagrant init:

$ cd ~/projects/myproject
$ vagrant init

Vagrant will have now created a VagrantFile at the root of that directory. Let’s open that in the editor of your choice–which for me is Sublime Text 2:

$ subl VagrantFile

If you’ve ever worked with Chef before, or indeed Ruby, you’ll recognise that the VagrantFile is Ruby. You’ll also notice that the folks at Vagrant have made newly created VagrantFiles wonderfully verbose, and if you read through the annotations, all the settings are explained.

For our Vagrant VM we’re going to want to use our base box, so update that line in the Vagrant file to:

config.vm.box = "ubuntu-11.10"

Since we’ve already installed this base box, we don’t need the VagrantFile to be aware of a hosting URL, so leave that line commented.

The next line defines whether you want to run your VM headless (i.e. without firing up the VirtualBox GUI). Generally the only time you’ll want to uncomment this line is if you’re debugging a problem with the VM. Most of the time you won’t want to waste system resources on the VirtualBox GUI. With that in mind, leave it commented.

Now we can choose networking options. For the purposes of this example I’m assuming we’re working on a local development machine and, as such, host-only networking will suit our purposes. If we were working on a separate dev box that may host multiple VMs, we’d probably want to use bridged networking to expose them to the network. For now though, just uncomment the following line:

config.vm.network :hostonly, "33.33.33.10"

This means the VM will be exposed to the host machine on the IP address 33.33.33.10, and will be able to access your network. However, machines other than your host machine will not be able to see the VM.

Since we’ve uncommented this line, leave the bridged networking line commented.

Now we have the opportunity to forward some ports from the host to the guest OS. This can be useful if you want to temporarily expose a host-only network VM to machines on your network, however, for the purposes of this tutorial, please leave this line commented too. Later on we’re going to install a web server and connect to this VM as if it were any other server on our network, so port forwarding is not required here.

Next we need to define some folders to share with the VM, and how to share them. It’s a well recorded fact that when using VirtualBox on a unix-based system it struggles as the number of files within any shared folder increases. For this reason, if you’re running Vagrant on a Mac, it’s better to use NFS to mount your directories—Windows does not suffer the same issue so NFS is not required.

To set up a directory to share on your Vagrant VM, simply declare it as follows:

config.vm.share_folder("v-root", "/vagrant", ".", :nfs => true)

This will create a directory /vagrant at the root of your VM. That folder will be shared with the directory on the host containing your Vagrantfile, implied by the . directory reference. For a Windows host, just set nfs: => false.

Now we can choose how the Vagrant VM is provisioned. As I said earlier, I personally prefer to use Puppet, so uncomment the following lines and update the manifests path. I find “puppet” is more obvious than “manifests”:

config.vm.provision :puppet do |puppet|
  puppet.manifests_path = "puppet"
  puppet.manifest_file  = "base.pp"
end

This will look for our base manifest, “base.pp”, in a directory called “puppet” at the same directory level as our VagrantFile. We’ll set that up later.

Finally, let’s add a last line to our VagrantFile to tell it to set-up a VM with 512Mb of memory:

config.vm.customize ["modifyvm", :id, "--memory", "512"]

You’ll notice this particular command isn’t annotated in the VagrantFile. For more information about customising the specific characteristics of the VM, take a look at the Vagrant docs.

Once you’ve completed these changes, your VagrantFile should look something like this (if you remove all the commented lines):

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant::Config.run do |config|
    config.vm.box = "ubuntu-11.10"

    config.vm.network :hostonly, "33.33.33.10"

    config.vm.share_folder("v-root", "/vagrant", ".", :nfs => true)

    config.vm.provision :puppet do |puppet|
        puppet.manifests_path = "puppet"
        puppet.manifest_file  = "base.pp"
    end

    config.vm.customize ["modifyvm", :id, "--memory", "512"]
end

Our Vagrantfile will now create exactly the VM we require when we run the vagrant up command. However, the VM itself only contains the software installed as part of the base box. Since this VM is serving our project, we probably want some more stuff on there like a web server or a particular version of a language. To set that up, we’ll need to use a provisioner which is where Puppet comes in. (If you want to use Chef, don’t let me stop you. I just won’t be covering Chef here.)

Provisioning with Puppet

Provisioning happens in Vagrant every time we run vagrant up or vagrant provision. This sets up the software on your VM according to a set of manifests (or “recipes” in Chef). Those manifests basically tell Puppet what it should script on the VM. This means you can update anything, including copying across files, updating files already on the system, installing and uninstalling packages, changing permissions, stopping and starting services, and a whole host of other software changes.

To do this, you’ll need to learn Puppet’s manifest language. This is a custom DSL that is quite easy to learn; although the Puppet documentation can be a little difficult to navigate. I’d recommend looking at:

To begin, let’s set up a directory in our project folder to contain our manifests, and another folder within that to contain any files we may want to copy across to our guest OS with Puppet. From the directory on your host that contains your Vagrantfile do:

$ mkdir -p ./puppet/files

Now in the newly created puppet folder, create a file entitled base.pp:

$ cd puppet && touch base.pp

Note that Puppet manifests have the .pp extension. You should be able to find syntax rules for all good editors to deal with this extension and file type.

Open up the newly created base.pp file in your editor of choice. Puppet manifests have a number of constructs, chief of which is the concept of a “resource”. Resources can be packages, files, executed commands, users, groups and services. These resources can then be organised within classes. So let’s start with a class for our dev box:

class devbox {
}

As an example, let’s set-up nginx as a web server. To do that, the first thing we’re going to need to do is install the package:

package { "nginx":
    ensure => latest
}

That was quite simple, right? Puppet already knows to use apt as the default package installer on Ubuntu, so this command checks the apt-cache to see if the latest version of the package is installed, and if it isn’t installs it.

Now we should make sure the nginx service is running:

service { "nginx":
    ensure => running,
    hasrestart => true
}

Equally simple. Here we’ve told Puppet to check the service nginx is running, and we’ve also informed it that the service has a restart function which Puppet can make use of.

I’ve stuck a bunch of hacked together Puppet manifests into a repository on GitHub. Feel free to poke about in them; I’ll be updating them as I do more stuff with Vagrant. For now, at least, take a look at the files directory because we can repurpose some of my set-up for this example. Save the following from my repo to your puppet/files directory which we created earlier:

files/nginx.conf
files/vhost.conf

Now we need to get these files from your host computer to the guest OS in the Vagrant VM. Once again, this is pretty simple:

file { "/etc/nginx/nginx.conf":
    owner  => root,
    group  => root,
    mode   => 644,
    source => "puppet:////vagrant/puppet/files/nginx.conf"
}

file { "/etc/nginx/sites-available/vagrantsite":
    owner  => root,
    group  => root,
    mode   => 644,
    source => "puppet:////vagrant/puppet/files/vhost.conf"
}

Notice the source exists within our /vagrant directory, which we’ve already set-up as a NFS shared directory. This means we’re effectively copying the file from there to the /etc/nginx/ directory on the guest. Also, if the destination directory does not already exist, Puppet will create it. Notice how I’m also setting the owner, group, and mode of the file during the copy.

So far we’re getting along fine, but wouldn’t it be nice if we could let Puppet know we’d like to handle these stages in some semblance of an order? We can do that in the resources using the require parameter:

file { "/etc/nginx/nginx.conf":
    owner  => root,
    group  => root,
    mode   => 644,
    source => "puppet:////vagrant/puppet/files/nginx.conf",
    require => Package["nginx"]
}

Note the reference to the resource we require is capitalised? This is a subtle difference between declaring and reference a resource.

What would also be nice is if we could tell the nginx service that, by changing the configuration files, we’d like it to restart. To do that we use the notify param:

file { "/etc/nginx/nginx.conf":
    owner  => root,
    group  => root,
    mode   => 644,
    source => "puppet:////vagrant/puppet/files/nginx.conf",
    require => Package["nginx"],
    notify => Service["nginx"]
}

We just have a couple more things to do now. Firstly, let’s create a symbolic link to sites-available from within sites-enabled so that our newly copied across vhost is picked up by the nginx configuration:

file { "/etc/nginx/sites-enabled/vagrantsite":
    ensure => symlink,
    target => "/etc/nginx/sites-available/vagrantsite",
    require => Package["nginx"],
    notify => Service["nginx"]
}

Notice I haven’t required the original file for this. That’s because it’s not a requirement for creating the symlink in the first place.

Secondly, let’s remove the default vhost because we don’t need it:

file { "/etc/nginx/sites-enabled/default":
    ensure => absent,
    require => Package["nginx"],
    notify => Service["nginx"]
}

With all that in place, and adding a full set of require and notify parameters, our final class should look like this:

class devbox {
    package { "nginx":
        ensure => latest,
    }

    file { "/etc/nginx/nginx.conf":
        owner  => root,
        group  => root,
        mode   => 644,
        source => "puppet:////vagrant/puppet/files/nginx.conf",
        require => Package["nginx"],
        notify => Service["nginx"]
    }

    file { "/etc/nginx/sites-available/vagrantsite":
        owner  => root,
        group  => root,
        mode   => 644,
        source => "puppet:////vagrant/puppet/files/vhost.conf",
        require => Package["nginx"],
        notify => Service["nginx"]
    }

    file { "/etc/nginx/sites-enabled/vagrantsite":
        ensure => symlink,
        target => "/etc/nginx/sites-available/vagrantsite",
        require => Package["nginx"],
        notify => Service["nginx"]
    }

    file { "/etc/nginx/sites-enabled/default":
        ensure => absent,
        require => Package["nginx"],
        notify => Service["nginx"]
    }

    service { "nginx":
        ensure => running,
        hasrestart => true,
        require => Package["nginx"]
    }
}

The next time we fire up our Vagrant VM, nginx will be installed, set-up, and running, and attempting to serve a website from the /vagrant/static directory on your guest OS. This will be the static directory within the same directory as your Vagrantfile on the host.

On the host, let’s create that directory and stick a very basic file in it:

$ mkdir static && cd $_
$ echo 'Hello world from my Vagrant VM!'' > index.html

Not the most well-formed example, but it’ll do. Now let’s fire up the Vagrant VM:

$ vagrant up

This should copy across the base box we installed, set-up all the things we specified in the Vagrantfile, and begin provisioning from our manifest. You should see messaged that Puppet has successfully installed nginx and the relevant files, and that the service is running. Now point your browser to:

33.33.33.10

Ta da! You have successfully set-up a vagrant development environment with specific software requirements for a specific project. Congratulations! Now go away and hack it.

Summary

So this has been an epic 5000 word look at why Vagrant is now my development environment tool of choice. I hope you can see that it makes working with multiple projects—each with their own requirements for an environment—as simple as possible. With a good understanding of the features of Vagrant, and also of the provisioning tool of your choice, you really do have the power to fire up and destroy environments on a whim.

Apologies for the sheer size of this article but I wanted to make sure I covered as much as possible to help newbies get up and running quickly. It doesn’t take much effort, and it really empowers developers to be able to spin up new environments quickly for hacking around in.

Spring reboot

2012-03-04T17:46:32+00:00

If you’re looking at the site and not the feed you’ve probably noticed some significant changes. After 5 long years of a heavily doctored Wordpress implementation with custom templates I’ve finally gotten around to rebooting the site, bringing it more in line with my requirements.

It’s more of a realignment rather than a redesign. I’ve trimmed back the cruft, and improved what I believe is important—and I’ve also made some more drastic changes which can be reversed should I want to.

So, without further ado, let me just take some time to explain my reasons…

Technology

The site is now statically generated by Jekyll in Ruby instead of dynamically on request via Wordpress in PHP. This has allowed me to rebuild my server and ditch a bunch of technologies I no longer work with. Key amongst those are MySQL and PHP, but I also no longer need Apache. It’s also made the site faster and less in need of a caching solution.

Statically generating the site has also meant I’ve ditched some more dynamic functionality like the contact form and—more importantly—comments. Considering the ratio of signal to noise in the comments posted to my site, and the sheer weight of spam, I’m reasonably confident I won’t miss them. What’s more, with the constant discussion on Twitter and the fact that I can easily bring comments back later with Disqus, I have the option for resurrection should I desire it.

Jekyll has some pretty cool extensions and plug-ins, and if you take a look at some of my more code filled posts you’ll notice the code samples are now styled up with the help of Pygments, a python library that marks-up the code for styling in CSS. It’s the same system used by GitHub.

I’ve used an HTML5 doctype so I can add modern features as they become more widely supported cross-browser. I’ve also used a fair amount of CSS3 so I hope you’re browsing using an up to date browser rather than something as flawed as Internet Explorer.

The styling now requires nothing but CSS and web fonts. I have significantly reduced the number of images required to make the site look pretty. This means the number of HTTP requests made by the page is significantly smaller and overall the site performs better in YSlow and PageSpeed. I have to say, this wasn’t a primary concern but it’s nice to get it without trying too hard.

My new site is now much easier to deploy and update and I’ve made use of all the good stuff I’ve learnt about server administration over the past few years. You could say it’s all much more DevOps-like in approach and makes use of stuff like Puppet, Rake, and SASS.

Design

I wanted to make sure I was designing for my content and not trying to fill space with extra information that nobody really wanted. Hence, the right hand menu has been replaced by much wider blog posts. Hopefully that’ll do away with a few of the nasty horizontal scrollbars that crept into some of my code examples whilst also giving the main content of the site room to breathe.

The typography of the site should now be significantly more predictable cross-browser. I’ve used @font-face before but it’s nice to use it for one’s own devices. Hopefully this new iteration is much easier to read than the last, and makes better use of the space available.

I’ve improved everything for SEO; flattening the site architecture, making sure to include descriptions, moving navigation below the main content, and defining canonical URIs. I’ve also got plans for the category pages but need some more time to plan out the work and felt it wasn’t really a blocker for launch.

The future

So apart from the SEO stuff I’d still like to work in, I’ve still got some work to do on the responsiveness of the site. Currently it’s not looking great for screens less than 480px in width and I need to rethink some positional stuff to make it work.

For the moment I’m reasonably happy with the site, and with some more work on it over the coming months I’m sure I can turn it into something I’m much more comfortable with. The real target was to make the core content more readable and I think I’ve achieved that. I only hope you, as a reader, agree.

User-controlled content density

2012-01-06T22:37:07+00:00

Back in October of last year Google Apps underwent a fairly extensive redesign. The overall intent of this realignment was to improve the user interface design, which has traditionally seemed to be a stumbling point for Google.

Personally I think the new designs are excellent and really appreciate the fact that they address my need to access my Google Apps from a variety of devices. I also like the fact that they allow me my own preference for the visual density of content within the page.

In fact this is such a killer feature that I reverse engineered it to better understand it. I’ll be looking to use this knowledge in some of my future work. Whilst I wouldn’t really refer to this deconstruction as such, plagiarism is the sincerest form of flattery. Unfortunately, due to extensive work commitments, I’ve only just gotten around to writing up my work…

Responsive

It’s my opinion that this technique relates quite closely to the responsive design ethos. The visual content density should respond to both the user’s available screen real-estate, their chosen font-size, and their own preference—which should always override any other external actors.

The majority of responsive design discussions and tutorials seem to talk extensively around how a design responds only to the client width. True responsiveness, however, equates to so much more than this.

Visual density

In this article I am referring to visual content density. This refers to the actual physical space around the content and is not to be confused with keyword density, which is concerned with the proliferation of words and phrases within said content. Balanced visual content density is an important factor towards improved typography, improved legibility, and an improved interface overall. Most designers will wax lyrical on the benefits of plenty of whitespace and how giving content room to breathe can improve the usability of the interface.

The visual density of content within your web-based documents can easily be controlled with an extensive set of CSS properties. However, there are only a few that give entirely predictable results across all browsers:

font-size
line-height
padding
margin
width
height

Importantly, these properties control both content size and the amount of whitespace that surrounds it. Together these two factors can sway the balance between spacious legibility and the efficient use of limited screen real-estate.

A demonstration

Here’s a quick and dirty prototype that I’ve knocked up to demonstrate visual density variance at work. You can vary the density via the buttons at the top, or by resizing your browser window (which will override the former):

http://timhuegdon.com/test-cases/density/

The source code for this example is available here:

http://github.com/nefarioustim/density-demo/

How it works

In my demo I am applying a class to the root element of the document that controls the level of content density selected by the user. I also repeat those styles within media queries for different client widths to allow denser content on smaller screens—try resizing the browser window to watch it respond.

To make this more maintainable in the future I’d probably invoke those styles through some form of SASS or LESS mix-in to cut down on the obvious duplication of styles. However, be aware that the use of CSS preprocessors like SASS and LESS can complicate things when attempting to get browsers that do not support media queries to detect the styles using a JavaScript polyfill like Respond.js.

Summary

So there you have it; a simple technique that has a pretty significant impact to the readability of your content. Hey, and the demo is kinda fun to play with too.

On HTML element identifiers

2011-09-27T10:11:43+01:00

Back in April 2007, I wrote a short article on HTML element identifiers for the .net magazine “Expert Advice” section. I have never republished it online—despite it being 400 words, nicely to the point and suitably succinct—and have long been meaning to reexamine the subject in more detail.

By a total coincidence a discussion in a recent front-end code review and a discussion over the past couple of days on Twitter both relate to the use of HTML element identifiers. The former, a discussion on semantic value and remembering that HTML should not prognosticate styling; the latter, a discussion on the validity of using IDs to target CSS. With both of these considerations I’m going to study the fundamentals of writing good element identifiers within this article.

What’s an element identifier?

HTML has two element identifiers; the id attribute and the class attribute. These attributes can be applied to most of the elements within a modern HTML document.

What do they do?

The id attribute assigns a name to an element. This name must be unique within the document and each element can have only one.

The class attribute assigns a class or set of classes to an element. Any number of elements may be assigned the same class name or names and each element can have many.

In relationship terms this means IDs and elements are a one-to-one relationship, and classes and elements are a many-to-many relationship. IDs are very much an absolute index system and classes are more of a tag-style classification aide.

Make note of those words: “name” and “classification”. I’ll come back to that in a bit.

Mislaid mastery

Put in those simple terms it’s all pretty straightforward, and I think that most front-end developers have a good general grasp of their usage. However, it’s the detail that introduces the issues, and if you take time to step back it’s easy to see why:

Element identifiers are the perfect answer for cross-linking other front-end technologies to your HTML. They are the hooks on which to hang your styling and behaviours, and because of this they take on that personality in your mind. This is where so many developers go wrong.

When front-end developers are first introduced to the id and class attributes, it’s often so the identifiers can be targeted in CSS selectors as a means of styling the developers’ documents. They will also inevitably start using them to hook progressive enhancement into their documents via DOM scripting. These are the two most common uses of element identifiers:

<ul id="site-navigation" class="navigation">
    …
</ul>

#site-navigation {
    border-bottom: 2px solid red;
}
.navigation a:focus,
.navigation a:hover,
.navigation a:active {
    outline: 2px solid yellow;
}

var site-nav = document.getElementById("site-navigation");
site-nav.invokeOarsumNavWidget();

// Use some modern browser native DOM + JS win
var navCollection = document.getElementsByClassName("navigation");
navCollection.forEach(function(element, index, array) {
    // Do something to every element
    // with a class of "navigation"
});

Also, if you’ve ever wanted to make use of the URI fragment identifier in a non-destructive way, you may have used the ID element identifier instead of the old-skool named anchor () approach:

<a href="#navigation">Skip to navigation</a>

In each of the above examples, the element identifiers were added specifically to be used by some other language; for styling it is CSS, for dynamic behaviour it is JavaScript, and for linking it is the URI RFC 3305. However, that association also occurs psychologically in our minds as developers and we often have trouble seeing past it.

Names and classifications

Back to those “name” and “classification” descriptions:

It seems we have carelessly forgotten the fact that element identifiers are a feature of HTML and not the other languages that use them as targets. We should be using them as they were originally intended; to name and classify the content within our mark-up.

We already use HTML elements to describe the content which they contain. This is the very heart of semantic HTML and should result in content like heading text being wrapped in h1-h6 tags; lists in ul, ol, or dl tags; and tabular data in table elements. Since IDs and classes are attributes of these elements, we should use them to improve the semantic value of their hosts and provide further descriptiveness or context.

If we return to our previous list example, the following mark-up is reasonably semantic:

<ul>
    <li><a href="/index.html">Home</a></li>
    <li><a href="/about/">About</a></li>
    <li><a href="/blog/">Blog</a></li>
    <li><a href="/blog/archive/">Archive</a></li>
    <li><a href="/contact/">Contact</a></li>
</ul>

However, we can improve it further by giving it some context with an ID:

<ul id="site-navigation">
    <li><a href="/index.html">Home</a></li>
    <li><a href="/about/">About</a></li>
    <li><a href="/blog/">Blog</a></li>
    <li><a href="/blog/archive/">Archive</a></li>
    <li><a href="/contact/">Contact</a></li>
</ul>

Now it’s more obvious what this list of links actually is without seeing it styled up and rendered in the browser.

We could provide even more information with a single class:

<ul id="site-navigation">
    <li><a href="/index.html" class="current-page">Home</a></li>
    <li><a href="/about/">About</a></li>
    <li><a href="/blog/">Blog</a></li>
    <li><a href="/blog/archive/">Archive</a></li>
    <li><a href="/contact/">Contact</a></li>
</ul>

I’m sure most web developers have done something similar in the past, but it’s a very good example of applying context with a class the right way. The HTML now gives us a very good description of the content it contains. In fact, you now only need glance at this markup to understand its function.

Let’s take it a little further. Imagine we have several types of navigation in the page; we can better describe these elements by classifying them as navigation:

<ul id="site-navigation" class="navigation">
    <li><a href="/index.html" class="current-page">Home</a></li>
    <li><a href="/about/">About</a></li>
    <li><a href="/blog/">Blog</a></li>
    <li><a href="/blog/archive/">Archive</a></li>
    <li><a href="/contact/">Contact</a></li>
</ul>

<p id="breadcrumb" class="navigation">
    <a href="/index.html">Home</a>
    &gt; <a href="/blog/">Blog</a>
    &gt; My Blog Post
</p>

<div id="page-footer">
    <p>© 2011 Oarsum Corp, all rights reserved.</p>
    <ul class="navigation">
        <li><a href="terms.html">Terms and conditions</a></li>
        <li><a href="privacy.html">Privacy policy</a></li>
    </ul>
</div>

Simple, right? We’re using the element identifiers to improve the semantics of the markup and gaining improved readability as a side-effect. What’s more the added context can be targeted by whatever technology you so desire.

Anti-patterns

I’m pretty confident that most web developers will end up with markup similar to this without necessarily understanding my point. To that end, let’s take a look at some common anti-patterns (or misuses) that may go some way towards forwarding that understanding:.

The “clearfix” class

Way back in 2004 [ya rly!], Tony Aslett and Position is Everything published a clever method for containing floats that involved using the pseudo :after selector to insert some content and use that as an anchor for clear: both. It was a clever technique and was quickly adopted by the community, often incorrectly bound to a class entitled “clearfix”:

.clearfix:after {
    content: ".";
    display: block;
    height: 0;
    clear: both;
    visibility: hidden;
}

/* IE targeted property */
.ie .clearfix {
    zoom: 1;
}

Quite apart from the somewhat erroneous name (it should really be “contain-float”) this is absolutely the wrong place to be addressing this sort of styling issue. A styling-specific meta-class such as this is no different to writing a class entitled “red-text” or “big-blue-heading”. In my opinion, the small amount of repetition it saves you in the CSS is not worth the non-semantic clutter in your HTML.

There are several ways to contain floats in CSS and the method you should use absolutely depends on the context. What’s more, some of those methods are a single CSS property declaration. For this reason, declaring a class of “clearfix” and implementing a single float clearing solution within it is actually potentially detrimental to the overall efficiency of your CSS.

It’s worth noting also that providing a meta-class tool like this results in a disassociation between the problem and the “fix”. You’re likely to experience maintenance issues when you change any float behaviours and the containing mechanism is abstracted from the target’s styles. In addition to this you’re likely to find developers over using the solution to address problems they’ve introduced themselves.

The “hide-offscreen” class

When developers are aware of accessibility issues, and are attempting to provide textual description for otherwise potentially inaccessible content, you will often find a class that looks something like this:

.hide-offscreen {
    text-indent: -999em;
    font-size: 0;
}

/* or */

.hide-offscreen {
    position: absolute;
    left: -999em;
    font-size: 0;
}

This moves text off the screen, thus making it invisible for those not using a screenreader. This is preferable to display: none (which you might use to hide content from all users) because the content is only hidden within the browser viewport, but not removed from the complete rendering of the DOM.

However, what we’ve done here is define our CSS first and then apply that classification to our content. We’re supposed to be doing it the other way around. This class is unhelpful in describing the content that it wraps because it describes the behaviour the CSS should apply. This results in it providing little explanation of the developer’s intent, resulting in HTML like:

<ul class="hide-offscreen">
    <li><a href="#navigation">Skip to navigation</a></li>
</ul>

Not a good class at all. This is an improvement:

<ul class="accessibility navigation">
    <li><a href="#navigation">Skip to navigation</a></li>
</ul>

I’ve added the accessibility class here to signal intent: it’s navigation for improving accessibility. I could have been more specific here, but overall I predict a high likelihood I’ll want to signal a similar intent on other features within my document.

However, the double class will cause problems if you need to target both classes at once (e.g. .accessibility.navigation {}) in IE6. That browser will only target the final class in the selector. To fix this, you could combine the classes to create a new one:

<ul class="accessibility-navigation accessibility navigation">
    <li><a href="#navigation">Skip to navigation</a></li>
</ul>

Or a less ugly solution would be to include an ID on the navigation:

<ul id="header-skip-links" class="accessibility navigation">
    <li><a href="#navigation">Skip to navigation</a></li>
</ul>

Or to use a wrapping element and target with the descendant selector:

<div class="accessibility">
    <ul class="navigation">
        <li><a href="#navigation">Skip to navigation</a></li>
    </ul>
</div>

Here we’ve named the content and classified it well. It would be difficult to argue that we haven’t provided any targets for styling or behaviours now too. Unfortunately we’ve had to jump through hoops for IE6 and have broken our rule of changing the HTML for the CSS, but I just wanted to illustrate that there are more options available given a little thought.

Avoiding repetition

It’s entirely fair to argue that both the anti-patterns above avoid repetition within the CSS itself. However, this shouldn’t necessarily be the only reason to adopt one of these types of solution. There are other options available to us to abstract away this kind of anti-DRY maintenance nightmare:

LESS Mix-ins
SASS Mix-ins
Place the CSS higher up the cascade

Indeed most CSS preprocessors will allow you similar functionality as it is clearly a “missing feature” of CSS as a whole. The alternative, of course, is to make your “fix” a part of the relevant cascade, so that all selectors beneath inherit that behaviour.

A nice example

A really nice example of classification is this simple design pattern for pagination:

<p id="paginglabel" class="audible">Pagination</p>
<ul role="navigation" aria-labelledby="paginglabel">
    <li><a href="#"><span class="audible">%TYPE% Page</span>1</a></li>
    <li><a href="#" rel="prev"><span class="prev">Previous<span class="audible">: %TYPE% Page</span></span>2</a></li>
    <li><p><span class="audible">You're currently reading %TYPE% page </span>3</p></li>
    <li><a href="#" rel="next"><span class="next">Next<span class="audible">: %TYPE% Page</span></span>4</a></li>
    <li><a href="#"><span class="audible">%TYPE% Page </span>5</a></li>
</ul>

Here we see the “audible” class describing content that should only be exposed to non-visual devices, such as screen readers and content scrapers (like search engine spiders). This is a good classification of a subset of the content.

It’s also worth noting the good use of WAI-ARIA roles and the rel attributes on the next and previous links. These are two other ways you can improve the semantics of your HTML and abstract away some of the classification to a more suitable annotation system.

Clean Code

Markup languages can often add to the obfuscation of code simply by their structure. If we use the element identifiers as they are intended we should end up reducing the overhead to grok relatively clean code.

Clean code is, amongst other things:

Readable
Understandable
Predictable
Maintainable

Use of element identifiers as semantic descriptors improves each one of these points. This is especially obvious when dealing with somebody else’s code, and in a team environment I personally feel it is imperative to include this in coding standards and best practices.

Good use of names and classification is just as effective as good commenting. When the code is effectively self documenting through its element identifier annotation, the need for commenting is significantly reduced.

On CSS best practice proposals

Now that I’ve covered the “hows” and “whys”, I’d like to highlight why I think certain recommendations for CSS best practice directly contravene the correct use of element identifiers:

There has been some discussion within the front-end community regarding the use of IDs when targeting CSS. The ID attribute has particularly strong specificity in CSS, due to its uniqueness. When used incorrectly, it can cause maintenance problems and specificity bugs. However, the idea that targeting the ID property with CSS should be considered harmful—as seen in CSSLint, for example—is extreme to say the least.

It is reasonably simple to clobber other rules on an element by incorporating an ID in the selector, but this should really be an expected outcome. The use of an ID in a selector should denote the intent that the rule targets only a single occurrence of something within the page. If that is not what the developer intended, then they’ve wildly misunderstood the need for that selector. Certainly it makes sense that targeting an ID should be considered harmful when the developer actually meant to target something less specific, but then that’s just using CSS properly, isn’t it?

The argument that it is less valid to target IDs when working in a truly modular environment is even worse. A truly modular front-end environment will always require instance targeting alongside classification targeting. Sure, it’s likely to be the minority of situations, but there will always be cases in your system where it makes much more sense to target an ID than a class, and since they’re already in place on your HTML by this point [they are, right? We’ve already talked about this. If they’re not, go back and read it again.] you don’t have to worry about your CSS enforcing your HTML.

The developers that make these sorts of recommendations must be aware that they, as leaders in their field, have a responsibility not to mis-sell their solutions. Problems born of misuse do not emphasise an inherent flaw with the technology but perhaps an inherent flaw with the expertise of the user. As an exercise in correlation, it’s worth noting that Chefs do not veto the use of sharp knives because someone once cut their fingertip off with one.

Summary

HTML, CSS, and JavaScript are separate technologies. They are quite tightly related in the world of client-side development, but they should always remain decoupled. The element identifier is an artefact of HTML and should therefore not attempt to predict the hooks required for other technologies; it should simply be a property of effective semantic markup.

Further to this, avoiding the use of an element identifier, simply because it is so often misused by otherwise able front-end developers, is—at the risk of being blunt—ridiculous. Be proud of your HTML, spend as much time perfecting it as you would on any other language, and, most of all, don’t pollute it with the litter produced by a solution, in a related language, that lacks finesse.

Final note

I realise, however, that—despite my banging on about it—there are some developers you just can’t reach. As Jared Spool so eloquently puts it in his blog post, ”Beans and noses”:

No matter how much you try, you can’t stop people from sticking beans up their nose.

Sniff my browser: The Modernizr inadequacy

2011-05-17T13:03:47+01:00

I’m currently involved in a project to write a fairly extensive set of best practices for front-end development. Alongside myself, this project includes input from a fair cross-section of my peers in the front-end development community. These best practices will be implemented alongside a coding standard as standards for development within the organisation I work for, and hopefully many other organisations when they are published.

Of all the standards that a front-end team might want to implement, those that concern the identification and graceful degradation of cross-browser feature sets can be the hardest to define.

With that in mind, I’ve been poking around the front-end community looking for possible solutions. By far the most common approach—and one that gains an astounding level of attention in the community—is to implement Modernizr, a JavaScript feature sniffer created by Faruk Ateş, Paul Irish and Alex Sexton.

Unfortunately, despite my respect for the developers involved, I just can’t advocate Modernizr as a solution. Let me explain why; but first, let’s revisit some concepts that are going to be quite relevant…

The web standards trifle

The separation of structure, presentation, and dynamic behaviour is imperative to web standards. To that end, Andy “Malarkey” Clarke wrote a fantastic post detailing a method he liked to use to explain this fact to clients. He entitled it “the web standards trifle”.

In Andy’s metaphor, the separation of concerns can be imagined as follows:

Sponge

This layer is your content, marked-up with well structured and valid semantic HTML. This HTML provides more information about each piece of content and allows any device reading the code to output as appropriate.
Fruity jelly

This is your presentational layer. This separates all your presentational code into CSS, potentially allowing a user to override it with their own, or for you to serve something different based on context, e.g. a print or mobile stylesheet.
Custard

This is the layer that accommodates any on-page behaviours. In the world of web standards we use this layer to progressively enhance our HTML and CSS layers with more interactive magic. I’ll come back to this later.

The true virtue of this divide is that the presentational (jelly) and behavioural (custard) layers can potentially be removed or overridden without affecting each other, or the content and markup beneath. This is hugely important if we want our websites to maintain client agnosticism and improve accessibility.

Progressive enhancement

Following the publication of this article there have been several suggestions that I explain the difference between graceful degradation and progressive enhancement. Suffice to say, this is something I have covered previously in another article.

Browser vs. Feature Sniffing

Browser “sniffing” is the act of attempting to discern a user’s browser by some means. This “awareness” can then be used in conditional code to control output or interaction for a specific browser. This technique was used a great deal in days of yore, when browsers had very different rendering engines and didn’t all conform to web standards. However, it was only ever really required in JavaScript.

The old school way:

if (document.all) {
    // IE4
    height = document.body.offsetHeight;
} else if (document.layers) {
    // NN4
    height = window.innerHeight;
} else {
    // other
    height = 0;
}

Not a good solution

There are problems with the method used above; for one thing, it is not the most direct method of identifying a browser. However, putting that aside for the moment, there are issues even with the theory of browser sniffing:

It’s quite obvious that browser sniffing does not scale. As new browsers are released, the sniffing code will need updating.

To make matters worse, many manufacturers developed their browsers to identify themselves incorrectly to get around legacy browser sniffing code. This ultimately meant that sometimes browsers were misidentified.

Ultimately browser sniffing is not a good solution. If you adopt it, you will usually end up maintaining separate streams of development whenever you need to add or update features. There are ways to mitigate that pain, but what if there was a better solution?

Feature sniffing

A more scalable approach is to use object detection in JavaScript to discover if a feature is available before you use it. An example of this might be:

if (document.querySelector) {
    element = document.querySelector(selectors);
}

The obvious advantage here is that it is browser agnostic; there are no assumptions as to which browser is involved at all. Using this technique, we overcome the scalability and maintenance headaches incurred through browser sniffing, and we can develop for graceful degradation from the outset.

CSS is a DSL that lacks features

Now, that’s all very well and good when we’re talking about JavaScript, but what happens when we need to affect the presentation layer?

CSS is a very simple domain specific language (DSL) that functions as a series of rules that override each other in a variety of ways (the cascade). There is no mechanism to detect a specific feature or a user-agent; only media types and potentially the odd CSS hack where a user-agent has imperfectly implemented an interpreter. There are also no conditionals; you just have to use the cascade to provide alternatives based on specificity or even just source order.

With these limitations, it is impossible to implement either browser or feature sniffing in CSS. This means we need to find an alternative method to affect the cascade based on the user’s feature set.

The Modernizr method

Step in Modernizr. In the words of the documentation:

Modernizr aims to bring an end to the UA sniffing practice. Using feature detection is a much more reliable mechanic to establish what you can and cannot do in the current browser, and Modernizr makes it convenient for you in a variety of ways:

It tests for over 20 next-generation features, all in a matter of milliseconds.

It creates a JavaScript object (named Modernizr) that contains the results of these tests as boolean properties.

It adds classes to the html element that explain precisely what features are and are not natively supported.

Following along so far? For the most part, that all seems fairly good. What’s more, Modernizr is definitely being pushed as the solution of choice amongst the front-end community:

So with all that in mind, let’s go back to why I will not be recommending Modernizr as a best practice:

Sniffing CSS features with JavaScript

The real issue is that Modernizr uses JavaScript to do the feature sniffing. This method is absolutely fine if we were only looking to test for the features in JavaScript. However, Modernizr is marketed on its ability to detect CSS3 and HTML5 features so that you can write gracefully degrading CSS:

Modernizr is a small and simple JavaScript library that helps you take advantage of emerging web technologies (CSS3, HTML 5) while still maintaining a fine level of control over older browsers that may not yet support these new technologies.

If Modernizr had sold itself as a JavaScript feature sniffing solution that also detected CSS features perhaps I could be more forgiving. In that context, it would be the developers’ responsibility to use Modernizr correctly. However, since the Modernizr site and documentation are selling it specifically as a CSS feature sniffing solution, for primary use within CSS, the error is clearly with the Modernizr team themselves.

It’s all down to the final step outlined in the documentation I quoted earlier:

It adds classes to the html element that explain precisely what features are and are not natively supported.

Modernizr uses JavaScript to add those classes to the html element; one for each of the features detected in the current browser. If we examine the generated source we see the following:

<html lang="en" dir="ltr" id="modernizr-com" class=" js flexbox canvas
canvastext webgl no-touch geolocation postmessage websqldatabase indexeddb
hashchange history draganddrop websockets rgba hsla multiplebgs backgroundsize
borderimage borderradius boxshadow textshadow opacity cssanimations csscolumns
cssgradients cssreflections csstransforms csstransforms3d csstransitions
fontface video audio localstorage sessionstorage webworkers applicationcache
svg inlinesvg smil svgclippaths">

When I first saw that, I was horrified. Not only is it ridiculously cluttered, it also feels decidedly unsemantic and looks like a terminal case of classitis. However, let’s try to be pragmatic here; it’s a means to an end, potentially a minor side-effect to a process that will win us some power in the bigger picture. In fact, it’s one step on from Paul Irish’s proposed work around to the FOUC (flash of unstyled content) which I’ve advocated as a reasonable fallback to marking progressively enhanced objects with an “enhanced” class.

Making use of the classes

The Modernizr method allows you to assign styles based on these new classes on the document’s root element like so (example lifted from the Modernizr docs):

button.glossy {
   background: #ccc url(gloss.png) 50% 50% repeat-x;
}
.cssgradients button.glossy {
   background: #ccc -webkit-gradient(linear, left top, left bottom,
         from(rgba(255,255,255, .4)),
         color-stop(0.5, rgba(255,255,255, .7)),
         color-stop(0.5, rgba(0,0,0, .2)),
         to(rgba(0,0,0, .1)));
}
.cssgradients button.glossy:hover {
   background-color: #fff;
}

Here we can see that button.glossy is first defined with a standard png as its background. The Modernizr-added .cssgradients class is then used to increase specificity and assign a WebKit gradient instead.

What happens if we approach the problem without the .cssgradients class:

CSS allows us to define a property twice in the same rule. It also contains built-in error handling that will ignore a property it doesn’t understand; so browsers that do not understand -webkit-gradient(), for example, will simply ignore a property that uses it. This allows us to progressively enhance our CSS like so:

button.glossy {
   background: #ccc url(gloss.png) 50% 50% repeat-x;
   background: #ccc -webkit-gradient(linear, left top, left bottom,
         from(rgba(255,255,255, .4)),
         color-stop(0.5, rgba(255,255,255, .7)),
         color-stop(0.5, rgba(0,0,0, .2)),
         to(rgba(0,0,0, .1)));
}
button.glossy:hover {
   background-color: #fff;
}

However, because we can no longer target the .cssgradients class specifically, we cannot target the button:hover styles only when CSS gradients are detected. Clearly this is an issue we’ll have to find some way around and is exactly what Modernizr is attempting to solve.

This all seems as if Modernizr is giving us a great deal of power, however, it comes with a significant cost.

Dependency on JavaScript is bad

Fellow web developer and good friend Mike Davies wrote a really great article containing a good summary of obstacles potentially preventing JavaScript from executing back in October 2010. This followed a somewhat presumptuous post by Nicholas Zakas on the YDN regarding the number of users with JavaScript disabled.

In short, there are a large number of obstacles that can affect the execution—or even rewrite the source of—JavaScript included in your pages. Hence, dependancy on JavaScript is bad because it introduces many significant points of failure.

As an example of why dependency on JavaScript can be catastrophic, I recommend you read another of Mike’s posts on Gawker’s JavaScript dependent URIs and how it broke their sites quite considerably. It’s worth noting that the Gawker sites have since seen the error of their ways and are attempting to fix the damage.

Styling dependant on JavaScript

By adding styles that are specific to classes that have been inserted with JavaScript, we are making those styles dependant on JavaScript. This means those styles will fail to apply if JavaScript itself has been disabled or, more commonly, the JavaScript file applying the classes has failed to load or execute successfully due to one of the reasons above. In essence, we may just as well add those styles with JavaScript directly because we’ve already broken the web standards separation of concerns.

Do we really want to remove the majority of CSS items covered by Modernizr if the Modernizr JavaScript include fails for some reason?

It would only be the feature-sniff-dependant CSS rules that would fail, since those classes would now be missing. Potentially this only means the loss of a few nice in-browser rendering optimisations. However, it also means there is a significant loss in the overall structure of specificity within your stylesheets, which could easily cascade unintended effects through to other elements. This could easily result in some content being styled to be unreadable, or worse, incorrectly hidden. This would be very bad indeed.

Alternative solutions

Modernizr is clearly unfit for purpose. If it were simply a JavaScript feature sniffing library for use only in JavaScript, it would be a suitable solution although only really useful for JavaScript progressive enhancement.

With that in mind, what other options are available that allow us to target our CSS?

Server-side browser sniffing

We could browser sniff on the server and add a class to the html element as we generate the HTML that is delivered in the HTTP response. This could result in something like the following:

<html lang="en" dir="ltr" class="firefox">

This means we can target CSS with the .firefox class in much the same way that we used the feature classes that Modernizr provided. There is no way for us to perform feature sniffing on the server; the best we could offer is a feature lookup based on the user-agent string that we’ve used to sniff the browser, but that would ultimately still be browser sniffing.

In fact, the best browser sniffing solution would allow us a little more flexibility on browser versions:

<html lang="en" dir="ltr" class="moz ff ff3-6">

Note I’ve added several classes this time; one for the rendering engine, one for the browser, and one that contains the major and minor versions. This would allow us more specificity.

Update: Matthew Pennell has written an interesting post on server-side browser sniffing with .htaccess and environment variables. Definitely worth a read.

Cache issues

However, server-side browser sniffing means that you’ll run into issues if your pages are publicly cacheable. This server-side solution results in a different HTTP response (the HTML) for each differing browser.

If you utilise any intermediary caching (Squid, Varnish, or a custom origin CDN) for static pages, and you use server-side browser detection, you need to make sure those caches don’t inadvertently send the wrong content to the wrong browser. To do this you’ll need to Vary: User-Agent header in the HTTP response. This instructs any intermediary caches to store multiple copies of the page (one for each User-Agent string that it sees) and to inspect the incoming User-Agent string when looking for cached responses to the current request.

IE targeting with conditional comments

If we’re entirely honest with ourselves as web developers, we can probably admit that the majority of woes we experience in CSS are as a direct result of features that any given version of Internet Explorer has not implemented. In fact, I regularly interview developers who advocate maintaining a separate stylesheet for IE that is included through the use of IE’s conditional comments. Having attempted to use this method in the past, I can confidently say that it is a maintenance nightmare and quickly becomes pretty unmanageable.

A better technique would be to adopt conditional comments to add IE specific classes in much the same way we have with Modernizr or server-side browser sniffing. This method was first proposed by none other than Paul Irish back in 2008 in his article ”Conditional stylesheets vs. CSS hacks? Answer: neither”. With that method, our html element would look like this:

<!--[if IE ]>
<html lang="en" dir="ltr" class="ie">
<![endif]-->
<!--[if !IE]>-->
<html lang="en" dir="ltr">
<!--<![endif]-->

This looks a bit odd, and often (for the true markup perfectionist) takes a bit of getting used to. However, it does give us what we need without requiring JavaScript or multiple cache versions.

We can even make it more specific if we so desire:

<!--[if lt IE 7 ]>
<html lang="en" dir="ltr" class="ie ie6">
<![endif]-->
<!--[if IE 7 ]>
<html lang="en" dir="ltr" class="ie ie7">
<![endif]-->
<!--[if IE 8 ]>
<html lang="en" dir="ltr" class="ie ie8">
<![endif]-->
<!--[if IE 9 ]>
<html lang="en" dir="ltr" class="ie ie9">
<![endif]-->
<!--[if gt IE 9]>
<html lang="en" dir="ltr" class="ie">
<![endif]-->
<!--[if !IE]><!-->
<html lang="en" dir="ltr">
<!--<![endif]-->

Quite clearly this gets ever uglier, but the solution is entirely encompassed in the structural layer; we will always have the correct class, even if CSS or JavaScript are unavailable for some reason. We also have the same markup regardless of user-agent string, which means caching is not a problem.

The only flaw with this solution is that we can only target versions of Internet Explorer. No other browser implements conditional comments.

Use CSS properly and make use of the cascade

As previously stated, CSS is designed to cascade. This means the built-in error handling will ignore properties the interpreter does not understand, and we can override rules based on specificity and source order. In most cases, these key features are more than adequate to develop gracefully degrading stylesheets.

Most capable front end developers are already doing this and coping just fine. Add this to the previous IE targeting approach and you’ll find that you do not need to over-engineer a solution.

In summary

So, in summary then, I absolutely cannot recommend implementing Modernizr as a best practice for front end development. It attempts to solve a problem from the wrong direction, and introduces a new potential point of failure. What’s more it breaks the fundamental ethic of web standards; the separation of concerns.

I continue to recommend well crafted, gracefully degrading CSS, backed up by conditional commented classes on the html element for targeting IE.

Modernizr may still be a worthy solution for JavaScript-only feature sniffing, if only it allowed the developer to disable the injection of classes on the html element.

SEO for Web Developers: Page Construction

2011-04-06T10:29:27+01:00

Following on from “SEO for Web Developers: Keywords and Links”, this next article in my SEO series focuses on page construction. Whilst I’ve previously stated that in-links (i.e. external incoming links) are the fundamental workhorse of good SEO, it is also important to make sure you are constructing your pages in a way that easily exposes your content, and that clearly links it to your identified keywords.

Further to that, it’s important to know what the search bots are looking for when they spider your pages. From URL structure through to semantic markup and page specific metadata, there are a multitude of features you can build-in from the start that will improve the search engines’ insight into your content.

A note on ranking factors

Every two years, SEOmoz conduct a survey across various SEO experts which they use to publish statistical findings in relation to search engine rankings. This data is exceptionally useful in judging what elements to concentrate your efforts on during development.

In 2009 link metrics were worth a whopping 43% of value when calculating rankings. However, in the latest figures, that value has dropped to only 22%, which brings it in line with similar values for domain-level link authority (i.e. your domain is a “trusted” domain for quality). This reduction in size hasn’t resulted in other factors growing in value, but rather new factors have been introduced. These additions are domain-level keyword usage (how the keywords are relevant across the site), domain-level brand metrics (highlighting the importance of “brands” as a whole), and page-level traffic/query metrics.

Understanding the bots

Understanding how the search engine bots evaluate your pages and content is key to learning how page construction affects SEO.

Page vs Site

I have previously written—in my article “Semantics and Structure“—of the importance of remembering that the web is simply a network of single pages. This network has only a vague understanding of the human concept of site; via the comprehension of URLs beneath a fixed domain—and even that doesn’t necessarily translate to a single site, despite being used for authority rankings. Large sections of search bot AI has been dedicated to discerning site structure from the links within your pages.

Whilst we can see from the metrics above that domain-level factors are now more prevalent in the calculations for ranking, it is still important to imagine each page of your site as an independent unit, and structure your content and code accordingly. This means that the page title should be situated in an h1 tag and not the site title. That is unless you’re developing the home page, in which case the site title probably does belong in an h1 element.

Site-wide architecture should be situated within non-pertinent markup (i.e. markup that doesn’t apply any semantic emphasis on the content) and placed accordingly in the source order. I’ve heard very peculiar things about content placed in paragraph tags being more pertinent than content that isn’t. In my experience this is a significant fallacy. For more information about “pertinent markup”, read the “On-page optimisation” section later in this article.

Block-level analysis

Search engines, for the most part, follow an algorithm of block-level analysis. This means they will break a page down into sections (e.g. masthead, navigation, footer, main content, secondary content etc.) as a signal towards ranking the content within.

It is a good idea to make sure that the content you want to rank for is situated within your main content area. This may sound obvious but in a brave new world of modular development, and pages made up of “modules” of content, it is surprisingly easy to confuse the search engines and suffer for it in the rankings.

Source order is important!

It’s also important that you are prioritising your main content in your source order. Search engines will only evaluate the link text of the first use of a URI in a page. If you’re repeating a link with more valuable and relevant link text in your content than you are in your navigation, you’ll need to make sure the navigation comes after the main content. This is often the exact opposite of good UI design which will attempt to place navigation in the most obvious place; across the top, or down the left.

Take the following code as an example:

<ul>
    <li><a href="/about/">About me</a></li>
    <li><a href="/blog/">Blog</a></li>
    <li><a href="/contact/">Contact</a></li>
</ul>

<h1>My site</h1>

<p>Welcome to my site. In here you can find
<a href="/about/">information about me and
my career</a> and <a href="/blog/">my personal
blog about web development and the
internet</a>.</p>

Here the navigation links will be the first things evaluated by the bots, but they’re probably not the most contextually relevant links to the content in question. In fact, the second set of links include good keywords and may add significant value. For this reason the better option would be this:

<h1>My site</h1>

<p>Welcome to my site. In here you can find
<a href="/about/">information about me and
my career</a> and <a href="/blog/">my personal
blog about web development and the
internet</a>.</p>

<ul>
    <li><a href="/about/">About me</a></li>
    <li><a href="/blog/">Blog</a></li>
    <li><a href="/contact/">Contact</a></li>
</ul>

Obviously you can change the visual position of this content with CSS. Whilst the more sophisticated search bots will render CSS in an attempt to detect invisible content (i.e. that which is hidden either through display or visibility rules, or that which is moved offscreen), they are less bothered about the reordering of content visually.

Creating content

It may sound entirely obvious but it’s exceptionally important to write good textual content for search engines. You will always rank higher if you include a good balance of text and links—with a high relevancy to each other—on your pages.

Duplication is bad

Try to avoid duplication of content. By this I mean avoiding the same content on two different URIs. If your content is duplicated then you will be diluting its value by placing it in two places, even if it’s actually the same page served from two different URIs. I’ll cover the method you should use to declare the One True Version™ of your pages later on.

It’s also important to make sure your pages don’t duplicate content that is elsewhere on the internet. Good examples of this sort of repetition would be travel brochure text or product descriptions which are highly likely to be used on a multitude of affiliate sites; especially if that content is included in some sort of feed. Wherever possible try and create your own content; it will always serve you better and will ultimately separate you from the crowd.

Dynamic vs. static content

Dynamic content is that which updates regularly and hardly ever stays the same from day to day. Good examples of this might be a list of blog posts on a blog index page (where the blog items update regularly), news item indexes, feeds from other sites (e.g. RSS, Twitter etc.), and regularly moderated lists of links.

Static content is that which hardly ever changes. Good examples might be “about me” text on your blog, the description on a product detail page, and the article text on a blog article page.

It is very important to find the right balance of both dynamic and static content on your pages. Some pages will suit more static content (e.g. the article page of a blog) whereas others will suit more dynamic content (e.g. the index of that blog). In either case, make sure you’re including some of both types of content. On the article include links to the top articles on the blog, or feeds from Delicious or Twitter; on the index page include some static “about this blog/author” text.

Design your URIs

Since links are the most important part of your SEO strategy, it’s hardly surprising that the design of the URI is a fundamental ranking factor. It’s important that you use just as much care with the URI as you do with the text of those links. Here are the important factors in good SEO-friendly URI design:

Keywords in your domain

If you can, try to get some keywords in your domain. Often the most successful domains will have one or more relevant keywords in their domain name, and the closer to the left the better. A good example of this might be something like travelsupermarket.com, which is currently number one in Google for “travel”.

Rather interestingly, _exact_ keyword match domains seem to perform marginally better than domains with _some_ keywords. By this I mean domains that are entirely formed of keywords in the order for which they are searched (e.g. cheapmajorcaholidays.com). Hyphenated exact match domains (e.g. cheap-majorca-holidays.com) seem to perform slightly worse than those without hyphens, and domains that contain all query terms but are not an exact match are marginally worse again.

Choose the right TLD

Rather notably the .com TLD (top-level domain) appears to perform better than any other TLD. That’s not to say that you can’t easily out rank a .com domain, but they do seem to gain an advantage in a like-for-like ranking test with pretty much every other TLD I tried.

Subdomains

Subdomains do not count as part of the domain. Cross subdomain links, as previously stated, are deemed internal and URIs containing a keyword subdomain appear to rank similarly to URIs with the same keyword as the first path element. Example:

http://games.mygamessite.com/

Ranks the same as:

http://mygamessite.com/games/

Keep paths shallow

Try to keep your URI path to a minimum. Each level you go down the hierarchy, the less value you give to keywords within it. This is another good reason to maintain a flat site architecture. Also, remember that the keywords on the left of each path segment (i.e. each section delimited with /) are the ones with the highest value at that level.

The maximum number of layers (or segments) you should use in a path is about 3. Any more than that and you’ve basically lost any SEO benefit from that section of the URI.

Be strict with your characters

There are a limited subset of characters that are permissible in URI syntax. However, that subset still allows a great variation in the style of your characters. In general it is best to keep your URIs strictly lowercase so as to cut down on the chance of creating or generating duplicates through case sensitivity. Also, try and internationalise the character set in your URIs; it’s always going to be easier to match a UTF-8 search term against a UTF-8 keyword in your URI.

Space separators

There are several characters that can be decoded as a space in URI syntax, but only two that will work successfully as part of your path: “%20”, and “-“. It is best to enforce the use of a hyphen as a word separator in your paths. Google understands this as a space, and it will ensure your URIs remain readable. Underscores (“_”) are not recognised as a word separator and are therefore of little use. Matt Cutts has previously discussed this in his “dashes vs. underscores” blog post.

Branches and leaves

Do not underestimate the value of human readability when designing your URIs. I have personally experienced, through extensive user research and user testing, that users afford a certain level of confidence to a readable URI. What’s more, some SEO consultants whom I have spoken with recommend applying slightly different rules to branch and leaf URIs.

A branch URI is a node that could potentially lead to more branch URIs or leaf URIs. The recommendation is that these URIs should end with a /.

Branches:

http://sportsnews.com/football/
http://sportsnews.com/football/bundesliga/

A leaf URI is one that is the final node in the path tree. The recommendation is that these URIs should not end with a /, and should include some kind of filetype extension (usually .html). I prefer leaving out the extension myself, since it feels a bit old school and almost enforces a file-type assumption on the resource in question, but I accept I may be clouded by a developer’s understanding of HTTP and REST.

Leaves:

http://sportsnews.com/football/international/teams/england.html
http://sportsnews.com/contact-us.html
http://sportsnews.com/terms-and-conditions

These recommendations amount to a more traditional OS directory-style vision, which makes them more familiar to the general non-techie user. It’s worth remembering that the URI is displayed prominently in the SERPs and as such inform the users’ confidence in clicking the item.

On-page optimisation

Now you’ve hosted the pages on good SEO friendly URIs, and you’ve built a good network of links to those pages, it’s about time we looked at improving the way the search bots spider and evaluate your content:

Page title

The page title should sum up the content in as few words as possible. On left-to-right reading pages the left most words are deemed more significant (I’ve no research on right-to-left languages, but one would assume the opposite). A good title will have a healthy sprinkling of keywords whilst remaining human readable:

<title>SEO for Web Developers - Nefarious Designs</title>

Note that, in my titles, I’ve chosen to include the site title following the page title. Firstly, this means the site title will be included in the page title on the SERP, but it also means that the site title is associated as a keyword or keywords.

Semantic content markup

The majority of semantic markup won’t give you a significant boost in rankings. Sorry folks, it’s true; despite the fact that we web devs love some good semantic markup, the search bots are less bothered. Let’s face it, the internet is still full of badly constructed web pages and the bots have to spider, analyse, and rank those too.

However there are some semantic elements (and some presentational elements) that will have a greater influence on denoting keywords in your content to the search engines. These are:

h1-h6: Well structured headings, that are relevant to your content (and not used as way-finders such as “navigation”) are highly important. The h1 on your page is arguably the second most important element short of your page title and should contain relevant keywords accordingly.
em, strong, i, b: Emphasising textual content with these tags will infer the content has some degree of added value. However, it’s important not to saturate your content with this as it will inevitably be flagged as keyword spam. Overall my tests were inconclusive in quantifying the effect of emphasised keywords, but there is clearly value in it.
a: I’ve previously waxed lyrical on the importance of links within your content, and the textual content of those links is just as important. Linking to similar pages with text that involves your keywords, or is simply related, can significantly boost your pages’ performance in the SERPs.

As a side note, I’ve been told by so-called SEO “consultants” that all internal links should be fully qualified (i.e. be absolute links including the domain) to prevent scrapers from duplicating your content elsewhere on the internet. This is absolute nonsense. There’s really no reason that a scraper won’t be able to remove the domain from any links it identifies as internal to your site (i.e. on the same domain as the page it’s scraping).

Meta elements

There are plenty of articles misrepresenting the value of meta elements in terms of SEO. For pretty much the last ten years, they have been mostly irrelevant for SEO. In fact, at the moment, this very site isn’t using them (more through laziness than intention—the site templates are old and I intend on migrating away from them sooner or later).

Keywords

The keywords meta element is all but abandoned by todays’ search engines. You will see little effect in rankings if you remove it altogether. I have noticed some smaller search engines using it, and it’s conceivable that other search engines may use links from these search engines as authority on your content. In short, you might see a small amount of benefit by proxy, but possibly not enough to warrant making the effort to keep the keywords list up to date and matching your content.

Description

The description meta element is used as the first port of call for the description of your page in the search engine. There is also a minor ranking effect from including keywords in this description but as ever, it’s important not to spam them here.

Link element

The link element is a sneaky little devil. Most web developers use it solely for linking stylesheets to their pages, but it has so many other uses that can help the search engines understand your site architecture better.

The link element allows you to specify another URI that is linked to your page, and also what relationship that URI has with the current one. To specify the relationship, you declare it in the rel attribute. The rel attribute has many possible values, but only a few that are relevant to SEO:

Canonical

At a minimum it’s important to understand the canonical link element. This allows you to declare the One True Version™ of any page on your site. This is probably easier to explain through example. Consider the following two URIs:

http://sportsnews.com/football/

http://sportsnews.com/football/?ads=false

Assuming these are actually the same page, the search engine will consider these to be unique URIs and could penalise you for duplication of your content. To avoid this, it’s important to declare one version as the canonical version. To do this you add the following to the page:

<link rel="canonical" href="http://sportsnews.com/football/">

This method is also true of something more subtle like “topic” style pages. Imagine the following URIs:

http://sportsnews.com/teams/tottenham-hotspur/peter-crouch

http://sportsnews.com/teams/england/peter-crouch

By adding a canonical link to the first item we can declare that, although the pages are exactly the same, the second is actually the same page on a slightly different URI.

Others

There are a whole host of available options to use in the rel attribute of the link element that may improve the search engines’ understanding of your pages and site structure. I really can be bothered listing them all out here, but you can find an interesting explanation of SEO relevant rel values here.

Search bots as knowledge-based systems

Modern search engine bots are incredibly complex programs. Considering the amount of information they gather on each pass of your pages, they should certainly be considered to be knowledge-based systems. As the web changes, so do the methods the bots adopt to navigate your pages in as close to a humanised way as possible.

To this end, both Googlebot and Bingbot are capable of understanding fairly complex navigation systems. What’s more, they are very good at evaluating navigation links within your pages to ascertain different routes to your content. Through this understanding they are able to apply value calculations to specific URIs.

Breadcrumbs

Google announced at the end of 2009 that they were attempting to display site hierarchies as an alternative to the URL in the SERPs where they could surmise it from on-page breadcrumbs. For more information on this, see “New site hierarchies display in search” on the Googleblog.

To do this, Googlebot simply analyses any constructs that look like a breadcrumb within your pages. This can be either a list of links identified by “breadcrumb” as an ID or class, or a set of links separated by the “>” character. In my tests both of these methods have been successfully picked up by Googlebot and displayed as site hierarchies on the Google SERP.

There are other methods of marking up breadcrumbs using “rich snippets” which I will talk about later.

Learning new interaction paradigms

Googlebot has been designed to understand modern in-page navigation paradigms, such as tab, carousel, and accordion widgets. It does this by searching for the presence of known markup configurations and included scripts.

For this reason, it’s sometimes better to use commonly used scripts such as popular jQuery or jQuery UI widgets which are likely to have a large user base across the web. Sadly this often means that badly written front-end interaction scripts may perform better than more bespoke options.

You can also improve Googlebot’s understanding of your pages by adopting common element identifiers on your JavaScript enhanced elements. This means using classnames such as “open”, “closed”, “active”, or “selected” etc.

It is worth noting, however, that in my tests rankings seemed better when pages used prebuilt widgets (i.e. those already in its knowledge base) than with bespoke techniques using predictable classes or IDs.

Rich snippets

These days Google is capable of rendering a large number of “rich snippets”. These are generally pieces of code specifically designed for marking-up data of a particular type, e.g. microformats, microdata, and RDFa. Often these snippets simply involve a range of predictable element identifiers or tags.

When displaying search results, Google will try and handle these rich snippets in the best way it sees fit. For example, reviews will appear as a 5 star rating widget just below the title of the search item. Recipes, on the other hand, are awarded their very own search utility:

Google Recipe search for “Chicken Madras”

Currently, Google supports the following types of rich snippet:

Reviews
People
Products
Businesses and organisations
Recipes
Events

For more information on implementing these snippets for your content, take a look at Google’s excellent rich snippet explanation page.

JavaScript

Googlebot currently runs a headless browser as it navigates pages. A headless browser is basically a browser with no user interface and, in this context, it means Googlebot constructs the correct DOM, and attempts to render all CSS and some JavaScript. It also means that many DOM events are fired as it follows links through your site, potentially exposing functionality it wouldn’t normally find.

The JavaScript related tests I have run to analyse the bots’ paths through my sites have largely been quite inconclusive; it appears that sometimes Googlebot simply evaluates embedded script in the page (i.e. code inside ). It also appears to find URLs manipulated into links on click events. However, it is definitely finding links in embedded script tags more regularly than those in external script files.

So far I haven’t managed to get Googlebot to spider content or links added during a DOMready or window.onload event. My tests are ongoing and I shall endeavour to document my findings on this blog once I have a clearer picture.

Summary

Apologies for the length of this article, it just kept growing! There was so much to cover that I really didn’t want to have to split it out into several sub-articles; especially considering the fact that it’s already part of a series.

Hopefully you’ve now got a head-start when it comes to building your pages for SEO. Tie this to the link building and keyword targeting you’ve been doing following my first article and you’ll be ranking well in the SERPs.

In the following article I’ll be looking at what you can do to help the search engine robots specifically by providing metadata for them, and how proper handling of HTTP can improve crawlability. I’ll also cover some basic tools that can aid your monitoring and improvement of SEO.

SEO for Web Developers: Keywords and Links

2011-03-22T22:47:12+00:00

Since the dawn of the web, developers and content editors have sought insight and enlightenment into the arcane art of search engine optimisation (SEO). This relentless battle to rise above the competition and feature at the top of the search engine result page (SERP), for a chosen keyword or search term, has forced a constant clandestine evolution upon the search engines themselves whilst they adapt to out-trick the trickster.

What’s more the proliferation of false knowledge, the equally charlatan-populated worlds of SEO consultancy and web development, and the sheer abundance of SEO false positives have meant that snippets of true SEO wisdom are drowning in a sea of irrelevant balderdash.

It is no wonder then that your average web developer is not armed with the erudition required to fight the good fight; knowledge that will allow them to articulate the message of their content to the search engines and therefore be exposed to a wider audience.

Fear ye not, weary web devs, for here is your codex of power. I hereby commit to blog my personal experiences and knowledge of Getting SEO To Actually Work™. Everything I discuss in this series of articles has been learned through experimentation, careful analysis, and search-bot honey-pot test sites. Oh, and mummyfrakkin’ science, bitches.

Keywords

One term you will hear more than others when researching or improving SEO is “keyword”.

A keyword is a word or phrase that contains high relevance to your content’s subject matter. A keyword is best imagined as a term a user might enter into a search engine in the hope of finding content like yours. For example, on a travel website generalised keywords might include “travel”, “vacation”, and “holiday”, whereas more specific keywords might be “5 star holidays” or “Greek holidays”.

As the search engines crawl and index the content on your pages they keep track of those pages in keyword-based indices. Thus, rather than storing 25 billion web pages all in one database, the engines maintain a vast array of smaller databases, each centred on a particular keyword term or phrase. This denormalisation makes data retrieval much faster, reducing it to mere fractions of a second.

Use of keywords

The search engines analyse the embedded keywords within your content to ascertain their relevance to the subject matter. This means a good way to optimise your content is to make sure the keywords are situated in key content structures, such as URLs, page titles, headings, and areas of emphasis (<strong>, or <b>; your non-presentational mark-up bears little relevance here). I’ll cover this in more detail when I talk about on-page optimisation later in this series.

It is also of note that a generalised keyword—such as “car insurance”, for example—will have a broader use, and therefore considerably more competition in terms of rankings. Something more specific like “women’s small car insurance” will limit your competition but might not have such a large search density. There is a clear balance to be found in your market and it’s important to identify the keywords you wish to target for a given page as early as possible.

A word on keyword density

The term “keyword density” describes the density (i.e. occurrence) of a keyword within your content. The term is a veritable minefield of misinformation and should set off alarm bells as soon as it is mentioned.

The density of your keywords bears no relation to the quality, semantics, and relevancy of your content; and it is these characteristics that glean the highest return. Do not be tempted into thinking that a page saturated with keywords will perform better in search rankings than a page that seats fewer keywords in higher quality, higher relevancy content.

Keyword targeting

As I said earlier, it’s important to identify the keywords you wish to target before you do anything else. To be effective, SEO requires that keywords are associated with and contained in your content at as many levels as possible.

Most SEO consultancies will invest significant amounts of time researching keywords for a given market, and also the competition against those keywords. What’s more, that analysis is ongoing, as the value of your chosen keywords will fluctuate wildly due to changes in the market, target saturation (i.e. too many people using it), and other such external factors (including black-hat devaluation which I will cover later).

Links

Let’s make sure we’re not under any false illusions before we get too far in; the two most important things about any page—in terms of SEO—are the links into it, and the keywords associated with those links. You could serve up the most hideous pile of non-web-standards, non-semantic, tables-for-layout, text-in-images content, and if the links into it are plentiful, and also contain the right keywords and/or link juice (more on that later), it will perform well in the rankings. Links are how the search engines will begin to calculate the relevancy of your page. Once they have a relevancy rating, it will then be affected by the analysis of the content of that page.

That’s not to say that there isn’t some artistry in the placement, design, and variance of those links, but since hyperlinking is the very essence of the world wide web, it is unsurprising that it is also the very essence of SEO. A high quality link will be featured on a page containing high relevance to the link’s subject matter, will be situated in an area of highly relevant content, and will have link text containing relevant keywords. It’s all about relevance.

Link Juice (aka PageRank)

Search engines apply a numeric score to a spidered page known as PageRank. This is often called “Link Juice” in SEO circles and is a calculated number based on the links—and the quality of those links—into, and out of, your page. This score is then used as a factor in calculating the ranking of your page against keyword relevancy.

As the search-bots spider the links within a page they pass a proportion of the host page’s score to each link. The page at the other end of the link is therefore afforded a total based on the score flowing into it from links elsewhere and how it passes that score onward. The best way of imagining this is as a complex network of pipes and sluice gates (links), with the score being the liquid within. High quality pages will have higher page rank due to incoming links with a higher score.

Bear in mind, however, that “link juice” is calculated differently based on “internal” and “external” links. Most search engines appear to deem links within a domain (internal) of less importance than those from a different domain entirely (external). For the most part, my linking tests seems to infer that cross-sub-domain links are also deemed internal. However that sort of testing is fairly difficult to quantify, so I’d recommend investigating for yourself if you’re intending doing something like that.

As an aside, it’s worth noting that a high link score (juice, PageRank, call it what you will) does not necessarily mean high SERP rank.

The “`nofollow`” `rel` attribute value

Following the dawn of the “blogosphere” and other such UGC (user-generated content), such as forums and guestbooks, there was an explosion of link-spam. These are links that have been placed on as many different sites as possible—either manually, or through automated processes known as spam-bots—with little relevance to the intended page content or conversation. This link spam is a black-hat method of rustling link juice into a given target page.

To combat this, the search engines implemented a scheme that allows a developer to mark a link as non-pertinent in the flow of link juice. To do this, you apply a rel attribute of nofollow:

<a href="http://somewhere.com/" rel="nofollow">Link</a>

`nofollow` sculpting

The rel="nofollow" attribute used to allow you to sculpt the score that would pass between links on your site, by preventing score flowing through no-followed links. This is no longer the case.

Before Google changed their algorithm, rel="nofollow" used to be handled like this:

Link 1 - Link score/4
Link 2 - Link score/4
Link 3 - Link score/4
Link 4 - Link score/4

Link 1 - Link score/3
Link 2 - No follow
Link 3 - Link score/3
Link 4 - Link score/3

Now it works like this:

Link 1 - Link score/4
Link 2 - No follow
Link 3 - Link score/4
Link 4 - Link score/4

This means that you get significantly less benefit from no-following a link than you used to. However, all the search engines (with Ask being an exception) state that they do take rel="nofollow" into account when weighting rankings:

Google, Yahoo!, and Bing say no-followed links are treated as plain text and therefore have no added emphasis when calculating keyword relevancy. Y! and Bing, however, do state that they use no-followed links for discovery and may follow them without affecting your rankings.

It’s all very complicated but suffice to say, it’s probably better to mark unimportant links on your pages as rel="nofollow" and let the search engines sort out amongst themselves what they want to do with it.

Improving “crawlability”

To improve the efficiency of spidering (i.e. how the search bots crawl your site) it is advisable to maintain a flat site architecture. In this context, site architecture is simply a means to describe the link depth of your pages. “Link depth” is the number of links it takes to navigate from the home page of your site to any given page within it. The shorter the maximum link depth, the better the “crawlability”.

For the most part, search engines understand a “site” to be a confined graph of internal links as mapped by the search bot when it spiders your site. Each pass through the site will give the bot a better understanding of your site-map. There are several ways to inform this process, such as sitemap XML files and the robots.txt file. I’ll cover those later in this series.

Maximum links per page

Most search engines recommend hosting less than 100 links on any page. This is primarily because, above a certain number, link score will deteriorate due to a search engine defined degenerative quality equation. This will result eventually in links that are not spidered at all. Secondly, a large number of links on any page is a signal of diminished content quality and will directly affect SERP rankings as a result.

The arbitrary value of 100 is misleading though as a high quality landing page, such as your homepage, can easily maintain 250-300 good quality links without penalty. The pages at the next level can then host ~200 quality links and so on. With this architecture, a site with a link depth of 4 or 5 can easily maintain millions of pages within its structure with little or no penalty.

Link building

Link building is the art of obtaining external links back to your content from high quality (i.e. high relevance to your content subject matter) pages elsewhere on the web. There is a fine line to walk between link building and creating link spam; it is the difference between the relevance of the surrounding content on the link hosting page.

Many SEO companies employ link building specialists who will inundate the web with links, either posted manually or accrued through link sharing contracts, registration with pay-to-link directories, and other such socially manufactured solutions.

Third-party linking

To build effective links it also a good idea to offset the effort. In some cases, placing a link on a site with high traffic, but that isn’t particularly high quality—or isn’t even spidered effectively—can work in your favour. A good example of this is Facebook which hides a large majority of its content from search engine spiders. A link placed on Facebook has a high likelihood of being reposted elsewhere, where the link quality will be vastly improved.

In today’s internet company you will find many Social Media “Experts” who’s sole responsibility is to use social networking to market the brand, the company, and, more importantly, links to the website. Whilst the quality of links on social networks is debatable, the benefit of reposts elsewhere cannot be ignored, regardless of quality.

Link bait

Another technique when link building is to create content that can be classed as “link bait”. This is content that has a high likelihood of being reposted elsewhere and is therefore subject to significant third-party linking.

In my personal opinion the concept of link bait is very hypothetical; one man’s link bait is another man’s quality content. In web development particularly there are any number of cynical parties waiting in the wings to troll a link as link bait. This may, however, have something to do with the inherent cynicism in the development world in general.

Summary

Understanding the concept of keywords, and then applying that concept to link building and your internal link structure, are really the two base concepts of SEO. Once you have mastered them you should be well on the way to improving your performance in search engine rankings.

The next article in this series will cover SEO in page construction, including information about on-page optimisation, designing your URLs, correct use of HTTP, and other such technical information.

Stalking friends on GitHub

2011-02-20T23:42:26+00:00

A while back, I used Yahoo! Pipes to filter my private GitHub RSS feed, so that it only shows information that may be interesting for me to track. The resultant feed allows me to see what my friends have started watching or have forked, or repos they have recently created, from the comfort of my feed reader.

Following a small refactor—guided by some more Pipes-knowledgable friends of mine (cheers Carlo and Dirk)—I privatised my token, and allowed customisation of the GitHub account used.

So, without further ado, here’s the pipe; feel free to use it to stalk your friends on GitHub:

GitHub Friend Stalker

On coding standards

2011-01-31T22:00:16+00:00

In my time as a web developer, I have been involved in the definition, implementation, and maintenance of several different coding standards, across various web-based languages. In my experience, this process is not as straightforward as it first seems, and can lead to a great deal of headaches if not handled in a very specific manner.

As the front-end development team where I currently work are going through a definition process, I have been reminded of a few things I’ve learnt over the course of my career. With that in mind, I thought I’d type up some of that knowledge in the hope that it might benefit other developers out there.

Some things to remember

It’s important to remember that a developer takes great pride in his or her work, and as a result has a significant emotional attachment to the code they output. When that code is submitted for a code review—where the implementation of any coding standards is going to be assessed by the developer’s peers (obviously alongside other things)—it is important not to tear the code apart without respecting that emotional attachment.

Further to this, it is important that the reasoning behind the rules in the coding standard are rigorously explained and well documented, along with good and bad examples. If the developers understand why they should be doing something, you’re more likely to get them doing it as second nature.

Finally, always check to see if there is a more general coding standard you can utilise, such as Python’s PEP-8 standard or the PHP PEAR standard. It makes far more sense to start with one of these and then build your own standards upon it.

Why have a coding standard?

There is really only one reason to adopt a coding standard; to maintain a predictable base of easily maintainable, understandable, and legible code. Ultimately, you are aiming to develop code that can be picked up by any developer—either in an internal team, or some other, wider community—and easily read, understood, and either fixed or appended.

Further to that, it is not the opportunity to enforce your own particular coding style on others; in fact, you should probably expect to change your own style as there will invariably be some decisions made that will alter your usual syntax or method. This should be embraced as a good thing, as you may learn something.

Creating

When creating a new coding standard, it’s important not to dictate rules to the wider team or community. The best standards are defined in agreement, not by dictation. It’s important to make the definition of team standards a team task.

When a standard—and the reasoning behind that standard—are offered to the team, discussion will invariable ensue. That discussion needs to be controlled and constructive, without dragging on. Nobody wants to sit and listen to two or three developers arguing; sooner or later someone will need to step in and make a decision, despite potentially disagreeing (and therefore upsetting) multiple parties involved.

A good way to avoid massive discussion on mailing lists, or in team meetings, is to make sure your recommendation and acceptance processes are not too complex. As soon as you instigate a complicated system of recommendation, confirmation, voting etc. you will instantly regret it. Developers are adverse to change, and you are potentially changing the very fundamentals of their day to day work. You just can’t expect the process to be quick and painless unless you enforce quick and painless at every possible stage.

The absolute best way to implement and maintain a coding standard is based on the outcome of code reviews. These are the perfect forum for discussion of style and method, and a good follow-up on that discussion is to update the coding standard.

The last thing you should bear in mind when developing a coding standard is that it will usually work best if it’s fairly language agnostic. Similar rules should be applied universally across languages; if you break this rule, you’ll quickly find yourself in a heated discussion about some minutia of syntax that is entirely deconstructive. Having said that, there are always going to be particular features and conventions of a specific language that should be applied to aid legibility or maintenance.

Comments

I hate badly commented code. I also hate writing comments. Ergo, I hate my own code.

Comment as you go

Commenting code is usually way outside the modus operandi of most developers. If we feel a piece of code should be commented, we’ll go back and do it afterwards, which means we’ve automatically given ourselves a mind-numbingly dull job that will, without fail, fall by the wayside upon release. To comment effectively, it’s important to do it as you go, even if you find yourself changing the comments as you refactor your code.

Comment sensibly

Comments should be added in distinct blocks, rather than commenting every single line. It’s important to remember what is important in a block of code and explain that. You’re documenting it for maintenance, so at least afford the next developer a modicum of intelligence. Of course, if you come back to it 6 months down the line, that next developer could very well be you!

When I say “distinct blocks”, I mean functions, methods, objects, classes, event callbacks, blocks of CSS rules, and anything else that requires explanation e.g. particularly complicated or confusing sections of functionality that might not otherwise be entirely obvious. Also, be sure to include references to other sections / files that may be relevant.

Pick your audience

As I’ve said previously, comments are there for the code maintainer, so if you can hide them from the end user, you may as well do so. Comments should be added in the template language rather than the HTML; likewise, any other code generation should keep the comments in the generator not the generated file (unless it is specifically pertinent to do so). If you comment JavaScript and CSS, make sure the comments are removed as part of your build process/combo handler, alongside minification.

Commenting standards

Commenting standards, such as JSDoc, are a good thing. You’re already trying to implement a coding standard, you might as well leverage [manager++] someone else’s.

Naming conventions

Language constructs may allow you the freedom of your given language, but please adopt a little restraint. It’s important to make them readable and decipherable by the next developer along.

Style

Try and adopt a naming convention that suits the style of the language. Here are some examples:

HTML and CSS — Use hyphenated lowercase IDs and classes:

<div id="live-scores" class="tab-module">
    …
</div>

JavaScript — Use camelCase variable naming, with constructors in UpperCamelCase:

var myVar = true,
    myFunction = function() {
        // Wicked awesome codez here.
    },
    MyConstructor = function(id) {
        this.id = id;
    };

PHP, Python — Use lowercase with underscore separation, with UpperCamelCase for classes:

$dev_name = "Timmah";


function get_foo() {
    return "foo";
}


class SimpleClass {
    // property declaration
    public $var = 'a default value';
    // method declaration
    public function displayVar() {
        echo $this->var;
        }
}

Grace

It is important to keep names concise yet explanatory. There’s no need to write a variable name that’s fifty characters long, if you could do it in ten. Likewise, it’s important you aren’t just writing a nonsensical string of characters that ends up looking like some kind of demented Countdown Conundrum.

Bad:

var boolean_flag_for_portrait_view = true;

Also bad:

var bl_flg_pview = true;

Good:

var is_portrait = true;

Prefixing / suffixing type

Type prefixes and suffixes are a bad idea; loosely typed languages allow the developer the ability to change things up, so you’d better expect it once in a while. Further to that, any intelligent developer can work out what type a variable is if they absolutely have to, so why waste the characters?

Final note: If type prefixes and suffixes are a bad idea, Hungarian notation is a very very bad idea. Don’t go there. SRSLY.

Style

As I’ve stated previously, a coding standard is not a way for you to enforce your own coding style on others. Try and keep any such rules to a minimum.

Tabs vs. spaces

One of the first conversations you’re going to have, when defining a coding standard, is whether you should be using tabs or soft-tabs (spaces) for indentation. In a modern development environment, this is irrelevant—even in languages where whitespace indentation is important, such as Python.

A developer should be able to set-up their IDE to use either tabs or spaces when indenting, thus they shouldn’t have to worry about it. The only thing that is important in this area is that it remains uniform in any given file. There is nothing worse than opening a file to find some things indented by 4 spaces, and others indented by 3 [deviants!].

Legibility

It’s important to write code that is legible; the more so, the better. To aid comprehension of blocks of code, it is useful to separate them with new lines.

It’s also important to try to keep lines of code to a reasonable length. Many developers and coding standards recommend a length of around 80 characters, with some even enforcing it. Python’s PEP-8 certainly falls under this heading. Personally, I’d prefer going over that length if the line remains legible; often breaking a line with carriage returns can have a negative effect on legibility.

Similarly, Lines that are too long are often difficult to read. If this is the case, then you must attempt to find the most legible form. Legibility is always the most important factor.

Whitespace

Liberal use of whitespace in your syntax can mean the difference between readable and unreadable code. Having said that, over use of whitespace can be entirely detrimental. Like my previous notes on style, it is important to find a balance when wrapping parentheses, expressions, assignments, and other such code with whitespace:

Readable:

var myVar = 'foo';

for ( var i = 0, j = myArray.length; i < j; i++ ) {
    …
}

Less readable:

var myVar='foo';

for(var i=0,j=myArray.length;i<j;i++){
    …
}

Accepted standards

There are many accepted standards used across multiple languages. It is important to make sure you don’t break these with your coding standard, even if you don’t specifically declare them anywhere. Good examples might be:

Use of i, j, k for simple iterators.

for ( var i = 0, j = myArray.length; i < j; i++ ) {
    …
}

Use of _ to denote private members.

var myConstructor = function() {
    this._private = 'something private';
};

Use of UPPERCASE to denote constants.
```
var VERSION = 'v1.1';
```

Expressions

Expressions are often where the complex logic within your code can be found. As such, to aid assimilation, you may like to break them down to their constituent parts.

A complex expression:

if ( myArray.length < 4 || ( myString.length > 2 && myString !== 'foobar' ) ) {
    …
}

Might become:

var isArrayShort = ( myArray.length < 4 ),
    isStringCorrectLength = ( myString.length > 2 ),
    isStringMatched = ( myString !== 'foobar' ),
    hasCorrectString = ( isStringCorrectLength && isStringMatched ),
    isPassed = ( isArrayShort || hasCorrectString );

if ( isPassed ) {
    …
}

If we clean it up further:

var isArrayShort = myArray.length < 4,
    isStringCorrectLength = myString.length > 2,
    isStringMatched = myString !== 'foobar',
    hasCorrectString = isStringCorrectLength
                       && isStringMatched,
    isPassed = isArrayShort
               || hasCorrectString;

if ( isPassed ) {
    …
}

Note the fact that I have broken the logical operators onto their own lines. This is another technique that can sometimes aid the readability of an expression, should it adopt multiple ANDs and ORs.

This last example may look more complex to begin with, but should a second developer read your original code, they will be left confused. The second example breaks the expression down into constituent boolean values to aid legibility.

Now imagine we have a similar example, but adopted using ternary notation:

var output = ( myArray.length < 4 || ( myString.length > 2 &&; myString !== 'foobar' ) ) ? 'win' : 'fail';

This would become:

var isArrayShort = myArray.length < 4,
    isStringCorrectLength = myString.length > 2,
    isStringMatched = myString !== 'foobar',
    hasCorrectString = isStringCorrectLength
                       &&; isStringMatched,
    isPassed = isArrayShort
               || hasCorrectString,
    output = isPassed ? 'win' : 'fail';

Ternary becomes much easier to read by a second party when used in this way; particularly when dealing with nested ternary statements. Of course, a good rule of thumb is to just avoid nesting ternary statements altogether.

Concerns over file size

In some languages, the size of the source code files you are creating can be pertinent to the performance of the final software. The obvious example of this is JavaScript files being served to a client.

In this instance, it is important not to sacrifice maintainability for optimisation. Should your files need optimising for final consumption, factor that optimisation away into your build or delivery mechanism. With JavaScript, this means minifying your code using an algorithm that can understand how to refactor it. Google’s Closure Compiler is a good example of one such tool; however, it still has limitations, and you may find it beneficial to write your own based around your coding standards.

Enforcing

There is really only one way to enforce a coding standard, and that is exactly the same way you should be developing your coding standard; through code reviews.

Summary

As my final word, I’d just like to point out that your mileage with this may vary wildly. This article is really only based on my own experience. However, whether you agree with everything here or not, I would hope you agree that easily maintainable code is the most important outcome of any good coding standard.

Object-oriented JavaScript follow up part 2: Technical

2010-10-28T15:12:52+01:00

Following on from part 1 of my follow-up to “Object-oriented JavaScript”, part 2 provides a technical update to some of the theories and examples.

Without further a-do, let’s jump in with the technical stuff:

A quick note on variable declaration

The var declaration allows you to chain definitions in a comma separated list; what’s more that list can reference variables declared earlier in the same list.

Take this code as an example:

var el = document.getElementById(“my-element”);
var elWidth = el.offsetWidth;
var elHeight = el.offsetHeight;
var elArea = elWidth * elHeight;

This could alternatively be written like so:

var el = document.getElementById(“my-element”),
    elWidth = el.offsetWidth,
    elHeight = el.offsetHeight,
    elArea = elWidth * elHeight;

Personally, I prefer the latter method simply because it keeps my var declarations in one place (where possible) without endless repetition of the var keyword. However, this is simply a minor note on coding style, rather than something that can improve the operation of your code.

Understanding object-oriented scope

As any experienced JavaScript coder knows, JavaScript is functionally scoped; this means any variables created within an object or function are only available within that object or function.

Simple, right?

Well no, there are actually a couple of complications to this behaviour that you should really understand:

Passing by reference

When passing arguments in JavaScript, it’s not immediately clear how those arguments are being handled internally. In fact, it is entirely dependant on the data-type of those arguments.

When passing arguments of type Number, String, or Boolean, JavaScript will pass them in by value. This means that the value stored in the argument will actually be a copy of the original value:

function accidentalFall(count) {
    count = count - 1; // Remove one
}

var greenBottles = 10;

alert(greenBottles); // greenBottles is 10

accidentalFall(greenBottles); // Pass by value

alert(greenBottles); // greenBottles still 10

Passing in an object, however, passes the argument by reference. This means that, within the function, the argument is not a copy, but a reference—or link, if you like—to the original object. This means any members of that object are available within our function, and any changes to the object passed as an argument will be reflected outside of the scope of the function.

Actually, the way JavaScript handles references is another blog article in itself. Watch this space.

Here’s our example again:

function accidentalFall(obj) {
    obj.count = obj.count - 1; // Remove one
}

var greenBottles = {
    count: 10
};

alert(greenBottles.count); // 10

accidentalFall(greenBottles); // Pass by reference

alert(greenBottles.count); // 9

This is a subtle difference, but one, I think you’ll agree, that is important to understand.

The `this` special operator

The this operator can be used to obtain a reference to the current context object and allows properties and methods within that execution context to be referenced. The context object can be considered a “hidden” parameter that is passed to any function.

There are four ways the context is made available to a function:

Implicitly with a method

var myObject.method = function(arg1, arg2…) {
    // Context here is myObject, and has been passed
    // implicitly.
}

Implicitly with `new`

var myConstructor = function(arg1, arg2…) {
    // Here we have a new anonymous context
    // understandable as the instance…
}
// …so the context here will be myObj
var myObj = new myConstructor(arg1, arg2…);

Explicitly with `Function.call`

var myObject.method = function(arg1, arg2…) {
    // Implicit context is myObject
}
// Explicitly force context to anotherObject
myObject.method.call(anotherObject, arg1, arg2…);

Explicitly with `Function.apply`

var myObject.method = function(arg1, arg2…) {
    // Implicit context is myObject
}
// Explicitly force context to anotherObject
myObject.method.apply(anotherObject, [arg1, arg2…]);

In general, JavaScript developers tend to rely on implicit context passing rather than explicit. However, there is one situation where this reliance falls down…

A common problem

When registering event handlers against DOM nodes in JavaScript, the this context no longer references the method’s parent object, but rather the DOM node to which the event handler is registered.

The easiest way around this is to make sure the reference to the context object originally stored in this is maintained before defining the event handler:

var widget.init = function () {
    var myButton = document.getElementById(“my-button”),
        counter = 0,
        that = this;

    myButton.onClick = function(event) {
        that.counter++;
        this.innerHTML = “Clicked!”;
    }
}

Further to this, according to Douglas Crockford, there is an error in the ECMAScript spec. that causes this to be set incorrectly for inner functions; something that he discusses in his article Private Members in JavaScript:

function Container(param) {
    function dec() {
        if (secret > 0) {
            secret -= 1;
            return true;
        } else {
            return false;
        }
    }

    this.member = param;
    var secret = 3;
    var that = this;

    this.service = function () {
        if (dec()) {
            return that.member;
        } else {
            return null;
        }
    };
}

Now, the ”var that = this” technique is certainly a bone of contention amongst members of the JavaScript community, but I’m firmly on the fence in the whole discussion. Certainly my second example could be solved with a complex network of Function.call and Function.apply instead of that.member, but my first example would be somewhat more difficult.

Personally, I feel if Doug Crockford—a developer with many years more experience and infinitely more JS knowledge than myself—says it’s ok, then I’m fine with it.

Garbage collection

As JavaScript is a scripting language, it implicitly allocates memory for objects, strings, variables, and so on as it runs. Garbage collection is the process by which the JavaScript engine detects when those pieces of memory are no longer reachable—that is, they could not possibly ever be used again—and reclaims the memory.

The ECMAScript specification (of which JavaScript is an implementation) doesn’t include a definition on how a JS engine should handle garbage collection, so although each JavaScript engine includes garbage collection—memory usage would quickly snowball if they didn’t—they all handle it somewhat differently.

Some engines are better than others; Google’s V8 is exceptionally good, where as the IE JScript engine is notoriously bad (often resulting in memory leaks).

Nested functions and closures

JavaScript allows the nesting of functions, creating nested scope blocks that inherit scope from their parents all the way up to the global scope.

Take this code for example (there are much better ways of doing this; this method is only in the interests of illustration):

function appendList(list, itemPrefix) {
    var getListItem = function(x) {
        return document.getElementById(itemPrefix + x);
    }

    for (var i = 0, j = list.length; i < j; i++) {
        getListItem(i).innerHtml += ‘ [done]’;
    }
}

In the above example getListItem is a nested function within appendList. This means it inherits everything within the scope of appendList. As a result of this, the inner function is able to make use of the itemPrefix parameter that is passed to its parent without it needing to be passed in as another parameter on getListItem.

When JS automatically garbage collects the appendList function, it will find no external references to the getListItem function and will garbage collect that as well.

This is a useful feature of JavaScript, and allows for fairly advanced lambda functions (anonymous functions), as well as providing a useful tool for namespacing library functions.

However, if, upon garbage collection, JavaScript finds an external reference to a nested function, it creates a closure of scope; thus maintaining access to all the variables required within the scope of that function.

This is known as a closure.

Common usage of closures

The most common occurrence of a closure in JavaScript is when declaring event callbacks. This is simply due to the fact that the callback function is referenced within the event listener once registered.

However, it’s important to understand that many modern libraries, plugin architectures, and plugins themselves also make use of closures, and understanding how they use them can mean the difference between controlled and uncontrolled memory usage.

Invisible pitfalls are invisible

It’s obvious when you think about it, but all closures have a potentially high memory imprint as they are maintaining much more than just their constituent parts. What’s more, because of their inherent avoidance of garbage collection, they are maintained until they are manually destroyed either by the code, or by a page refresh.

Prototypal inheritance

I did discuss prototypal inheritance in my original Object-Oriented Javascript article, however I just want to go into a little more detail here.

In a prototypal system, objects are supposed to inherit from objects; unlike in a classical inheritance system where classes inherit from classes and instantiate objects. JavaScript has no class (pun intended).

Prototypal inheritance is actually simpler than classical inheritance once you get your head around it. You don’t need to define classification, so your code is smaller and less redundant since objects inherit from other more general objects. It is a model of differential inheritance; i.e. each level of inheritance only adds the differences with its parent.

Unfortunately, JavaScript is a bit confused about its prototypal nature, and wants to fit in with the other scripting languages by pretending to like The Beatles and by making itself more attractive to classically trained programmers by introducing the new operator.

This means that:

new myConstructor();

produces an object that inherits from myConstructor.prototype.

This indirection means that JavaScript actually has a pretty ugly constructor pattern and most new JavaScript developers never really get to grips with the inheritance model at all.

However, all that has changed with the all singing, all dancing advent of JavaScript 1.8.5 (The New Shit™), which introduces the Object.create method:

var human = {
    legs: 2,
    arms: 2
};

var tim = Object.create(human, {
    tall: true,
    fat: true,
    overInflatedSenseOfSelfImportance: true
});

Mmm tasty, but since JS 1.8.5 only seems to be implemented in Firefox 4 (still in beta), don’t get all excited just yet.

If you want something similar in your own code, you can implement the following for a similar approach (and still kiss goodbye to ever having to use the new operator again):

if (typeof Object.inherits !== 'function') {
    Object.inherits = function(parent, child) {
        function temp() {};
        temp.prototype = parent.prototype;
        child.super_ = parent.prototype;
        child.prototype = new temp();
        child.prototype.constructor = child;
    };
}

Object.inherits(parentConstructor, childConstructor);

What’s more, this DIY version will allow you access to any superclass’ implementations of a particular method like so:

childConstructor.prototype.foo = function(a) {
    // Here’s some of that scope correction we discussed
    // earlier…
    childConstructor.super_.foo.call(this, a);

    // Other code
};

This code is a mix of the goog.inherits function—included in the Closure library—and the Object.create outline by Crockford in his article on prototypal inheritance.

JavaScript design patterns

Taking all this into account, we can start developing some incredibly useful design patterns that make use of prototypal inheritance, closures, and JavaScripts nuances of scope.

In fact, several have already been developed within the JavaScript community:

JavaScript namespacing

To avoid the perils of globally scoped variables and functions, it is advisable to create a single global object for the purposes of namespacing the rest of your code. This is relatively simple to achieve:

var NEF = window.NEF || {};

This command tests for the existence of a NEF object at the global scope, and if evaluated to true, returns that object. Otherwise, it creates a new object using the object literal syntax.

You may now assign any further variables or modules to your namespace:

NEF.myAttribute = ‘foo’;
NEF.myFunction = function() {
    …
};

Module pattern

The JavaScript module pattern was first proposed by Doug Crockford as a means of enforcing public and private object members through the use of closures.

My preferred variation on the pattern is such:

NEF.Module = function() {
    // Private members
    var privateAttribute;

    function privateMethod() {
        alert('Private method called.');
    }

    var pub = {
        // Public members
        publicAttribute: true,

        publicMethod: function() {
            // Private members are in scope here

            privateMethod();

            // Public members are addressable through the 'pub' object.

            pub.publicAttribute = false;
        }
    }

    return pub;
}();

This method simple uses a closure to maintain private variables in the scope of the returned pub object. The trick of this technique is the use of the (), right at the end of the code, after the parent function definition, which causes the function to run immediately and return the pub object.

jQuery plugins

The jQuery plugin architecture also makes use of closures to do something quite similar to the Module Pattern above. However, jQuery plugins are specifically namespaced to the jQuery object itself:

(function($) {
    // Private members
    var debugMode = false;

    function debug(msg) {
        if (!debugMode) { return; }
        if (window.console && window.console.log){
            window.console.log(msg);
        } else {
            alert(msg);
        }
    }

    $.fn.extend({
        myPlugIn: function(config) {
            var defaults = {
                …
            };

            if (config) {
                $.extend(defaults, config);
            }

            this.each(function() {

            });

            return this;
        },

        publicMethod: function() {
            …
        }
    });
})(jQuery);

Alternatively, should you wish to only include a single public method, you can avoid jQuery.fn.extend by instead making use of jQuery.fn for definition:

(function($) {
    // Private members
    var debugMode = false;

    function debug(msg) {
        if(debugMode && window.console && window.console.log){
            window.console.log(msg);
        } else {
            alert(msg);
        }
    }

    $.fn.myPlugIn = function(config) {
        var defaults = {
            …
        };

        if (config) {
            $.extend(defaults, config);
        }

        this.each(function() {

        });

        return this;
    };
})(jQuery);

In this code we can see that jQuery is passed to an anonymous function, that is run immediately, as the parameter $. The anonymous function provides a scope block for the plug-in code, thus preventing potential headaches from variable naming clashes across shared code. The plug-in functions themselves are then created as closures so that they have access to the private variables and functions within that outer anonymous function. These plug-in variables are then bound to the passed in jQuery instance $ using jQuery’s own built in extension functions.

This is actually quite a clever use of closures, although it does have the potential for memory leaks and excessive memory usage if variables and references aren’t kept in check. With that in mind, it’s often a very good idea to profile your jQuery plugins to understand what might need tidying up.

Custom events

Most modern JavaScript libraries include a mechanism for defining your own custom events. A custom event is merely a bespoke event, defined in your code, that other event-handler functions may be bound to. This custom event may then be triggered by you at any given point of execution in the code.

The custom event architecture is an implementation of the Observer Pattern, in which an object maintains a list of dependant “observers” which it notifies automatically of any state changes. In JavaScript, the object maintaining the list is our event, and the “observers” are the event handlers.

Using custom events in your own code provides a useful binding for other developers’ code. The event itself is loosely coupled to the event handlers, thus, additional event handlers can be added or removed by third-party code.

For more information on custom events, I’d recommend having a look at the documentation for the library of your choice. Here are few links:

Summary

So that’s the technical addendum to my 2006 post. Hopefully, that should serve to bring the article up to date, and correct a few of my previous errors. As always, comments and corrections are welcome via the comment form below.

Originally I had intended posting this article in two parts, however, it has continued to grow even as I write it. For this reason, the third and final part of this follow-up series will look at what’s on the horizon for JavaScript development, including how GoogleBot deals with JavaScript, the importance of supporting a core experience (where JavaScript is unavailable), server-side JavaScript, and architecting JavaScript applications.

Object-Oriented JavaScript Follow-Up Part 1: Method

2010-10-08T10:43:08+01:00

Back in May, 2006, I wrote an article entitled Object Oriented JavaScript. It was an exercise in both writing a useful tutorial, and a brain-dump of various things I had learned about writing object-oriented code on the client side. It quickly became apparent—as the article was linked to from a multitude of my favourite web standards development sources—that the tutorial itself was helpful; the comments burgeoned, and the traffic rocketed.

When I read that article now, however, it bothers me. The tutorial is still good although the information within is somewhat dated; and my personal experience, knowledge, and general approach to writing JavaScript has changed quite drastically. The fact that the article still receives a consistently high number of unique visitors on a daily basis, when I no longer agree with all the content within it, bothers me even more. With that in mind, I have decided to write an update that looks at OOJS as I would currently use it.

The resulting post has become quite extensive, so I’ve decided to split it into two separate articles. This first one will cover some development theory, whilst the second will dive into JavaScript coding specifics.

A note on the name

A friend of mine recently quoted this little gem in his Twitter stream:

Java is to JavaScript as Ham is to Hamster.

Enough said.

Is it object-oriented?

In my original article I stated that JavaScript was not an object-oriented language, but rather an object-based language. This was entirely unfair on JavaScript, and I’d like to fix that:

JavaScript is an object-oriented language. However, it breaks away from more conventional object-oriented languages by implementing prototypal inheritance, instead of classical inheritance. This is a Very Good Thing™ (although that seems to be a matter of opinion). More on that later.

First principles for JavaScript programming

Before I start, let me be clear that there are first principles of programming that are language agnostic. We’re going to look at a couple of those, but I’d also like to discuss a few of the more JavaScript specific principles first; those that relate to the cross-browser web standards environment. Finally, I’m aware that I’m really only scratching the surface of some of these topics. I don’t really want to muddy the water here by waxing lyrical on good development practices, so if you’re after more depth, I’d recommend researching them via the search engine of your choice:

Proper separation of the behavioural layer

Above everything else, it’s important to keep your content and markup (HTML), presentation (CSS), and behavioural (JavaScript) layers separate. Why is that important? Well if one or more of those layers are missing or broken (with the obvious exception of the core HTML), the others should still function as intended. Ultimately this makes presentation and behaviour optional, and a user might turn them off—via their browser settings—should they so desire. Alternatively, the user may include custom presentation or behaviour to aid a particular use-case.

In JavaScript, proper separation requires making use of the DOM to bind behaviour to HTML elements via element identifiers (i.e. the id or class attribute) in your source code. The alternative would be to write JS directly into the HTML, via event attributes such as onClick=”” or via the javascript: pseudo-protocol in the href attribute. This is not a good practice and I heartily recommend against it.

Graceful degradation

Graceful degradation is a well known software engineering concept which involves writing your code to take advantage of awesome feature X, unless awesome feature X is unavailable, in which case degrade to use slightly-less-awesome feature Y (assuming that too is available).

In web development terms this might mean making use of an amazing JavaScript date picker widget on a text-based input box, and just degrading to server-side date validation if JS is not turned on in the browser. It could also mean making use of the latest CSS3 styling techniques but degrading to use CSS2 techniques in less capable browsers.

In short; build everything in the most up to date browser, making use of all it’s wonderful modern features, and then go back and make sure the site is still usable when any of those features are not.

The problem with approaching the issue from this direction, however, is that you are saying to your users: “You (or your browser) are incapable of handling the full experience we want to present, so here’s a cut-down version.” This pretty much flies in the face of user-centric design, and good user experience. With that in mind, we’d be better using…

Progressive enhancement

Progressive enhancement involves approaching the problem from the other direction. You should design for the user-agent with the least capabilities and add functionality for more capable browsers afterwards.

In web terms this means developing and designing with nothing but the base features of HTML in mind, and then enhancing those features (or the features of the browser interpreting that HTML) by attaching CSS and JS to the HTML via element identifiers like IDs and classes.

The final outcome of these two approaches might be very similar, but the subtle difference is that you are giving more attention to the less capable user-agent when you progressively enhance.

The most common use-case for progressive enhancement is the support of a non-JS experience.

DRY (Don’t Repeat Yourself)

Every piece of knowledge must have a single, unambiguous, authoritative representation within a system. This means abstracting code into shared functions and objects whenever you find you are repeating yourself, and attempting to maintain a single source of truth in every aspect of your system.

DRY is a well known paradigm in software engineering, and as such, should be applied in every detail of your JavaScript code.

Refactor early, refactor often

Further to the principle of DRY, it is important that if you spot a potential issue in your JavaScript code you fix it. The more small refactors you do, the less time-consuming big refactors you are likely to need to do later on.

Once you’re refactoring often, you’ll be able to…

Make it work, make it right, make it fast

The first iteration of your code should only look to reach the level of functionality defined in the functional specification. Once this has been achieved, refactor to improve the way it works—so the code is “right” in your opinion. This may very well mean a complete overhaul of architecture, but you will be able to concentrate much more specifically on that since you already have it working. The final stage in this development cycle is to optimise your code for performance and efficiency; premature optimisation is a fundamental mistake in development.

Useful resources

When it comes to general development, there are a veritable plethora of resources out there, but my recommended starting point is the book The Pragmatic Programmer, by Andrew Hunt and David Thomas. This book outlines quite a few paradigms that should get you thinking about the way you develop.

Important things to understand

Pure JavaScript

In this age of JavaScript libraries and frameworks, it’s important to make sure you understand core JavaScript functions, objects, and methods. Although your library of choice may clean up any cross-browser issues you may encounter, it is always important to understand exactly how you are interacting with JavaScript as a language.

Useful resources:

The Mozilla Developer Centre section on JavaScript is an invaluable reference resource when writing pure JavaScript. Also, make sure you own a copy of JavaScript: The Definitive Guide by David Flanagan.

The DOM

The Document Object Model is an entirely separate beast to JavaScript, but is implemented natively within it. There are also DOM libraries for other languages that implement the W3 DOM specification when dealing with XML and SGML files.

Understanding the DOM in JavaScript is essential when designing and developing interactions with your HTML and CSS. Thankfully all JavaScript libraries abstract out the pitfalls of DOM scripting so that you don’t have to, but as before, it’s still important for you to understand how so that you can maximise your efficiency in using said library.

Useful resources:

Once again the Mozilla Developer Centre provides an excellent DOM reference section, although it is slightly skewed in favour of the Gecko browser engine. You can also find plenty of information on the DOM at the W3C site.

Interaction with the browser

When writing client-side JavaScript, it is important to take into account how HTML, CSS, and JavaScript interact with the browser directly. At every available opportunity, you should attempt to make sure you are not contradicting the browser, and that you are not rewriting interactions that are already well handled by the browser and operating system.

A good example of this is to make sure you do not break the “back” button. Often this can be achieved by simply using the window.location.replace() method to control which pages appear in the browser history.

Libraries

Using libraries in JavaScript has become standard practice. A JavaScript library exists to save time and effort, and to provide a single source of truth (as discussed earlier) for many common tasks within that language.

A JavaScript library provides you a toolbox full of handy devices to make your life easier. It needs to be treated as such, and not as a language in its own right. It is very important to understand the core structures of JavaScript, and how the library of your choice interacts within those structures and your own code.

Every library is different and has its own pros and cons. I’m not going to go into my opinion of those here; perhaps that’s something for another post. They do, however, all interact somewhat differently with JavaScript, and it’s important to understand how the library you choose does that specifically.

Summary

Hopefully that should get you in the right frame of mind to develop high quality JavaScript code. In the second half of this article I will be looking at specific use of JavaScript in regard to object-orientation.

Continue reading part 2 of my object-oriented JavaScript follow-up…

jQuery Accordion

2010-08-22T11:06:57+01:00

Update November 2010: Documentation for this jQuery Accordion can now be found here: jQuery Accordion - Tim Huegdon’s Projects

Following on from my post about my jQuery Carousel, here’s another widget: an accordion.

Until recently, I’ve usually used my friend and ex-colleague Marco’s jQuery accordion plug-in. However, whilst prototyping some stuff for work, I noticed a rather uncomfortable animation jump which, after some investigation, seems to be a well documented issue with jQuery’s slideDown() method. Since I was prototyping at the time, I quickly knocked up my own very basic accordion and, over time, it grew into something more.

The current version of my accordion follows a recent refactor–there are still several features I’d like to add–but for now I’m happy to let it out into the wild.

Demo

As was the case with my carousel, I’d definitely recommend taking a look at my page of example accordion implementations. This should give you a reasonable idea of the capabilities of the script; and as I update the widget, I’ll update this page.

Source Code

All the source code for the accordion–and the example page above–is available in the following GitHub repository:

jQuery Accordion – GitHub repository

Implementation

HTML

Basic HTML for the accordion is as follows:

<ul class="accordion">
    <li>
        <h3>Handle 1</h3>
        <div class="panel">
            &hellip;
        </div>
    </li>
    <li>
        <h3>Handle 2</h3>
        <ul class="panel">
            &hellip;
        </ul>
    </li>
    <li>
        <h3>Handle 3</h3>
        <p class="panel">
            &hellip;
        </p>
    </li>
</ul>

For each panel in the accordion, you will require both a panel (the content of our accordion pane) AND a handle (an item that is clicked to open a panel). These can be anything you want as you can specify the jQuery selector used for each as part of your configuration. By default, the selector for handles is “h3”, and the selector for panels is “.panel”. Obviously that was fairly specific to my original implementation, so I’ll look to fix that in the near future.

Notice that, because we’re specifying our own selectors, we’re able to use flexible HTML for either the panel or the handle. However, it’s also worth remembering that the accordion script injects a link around the contents of whatever you specify as the handle, so that the handle becomes focusable and keyboard navigable. If you are likely to end up with invalid markup (i.e. the inline link wrapped around a block level element), you may get unpredictable behaviour cross-browser.

CSS

The included CSS is really only a reset style-sheet for the list elements within the accordion. You will probably already be achieving this with your own reset stylesheet, but you can add the following link to the head of your document:

<link rel="stylesheet" href="css/accordion.core.css" type="text/css" charset="utf-8">

JavaScript

To activate the accordion in your pages, you will require both jQuery and the accordion script itself to be included at the bottom of your page, just before the closing tag, but before any script that applies the accordion to a jQuery object.

A default implementation would look like this:

    &hellip;
    <script type="text/javascript" src="js/jquery-1.4.2.min.js" charset="utf-8"></script>
    <script type="text/javascript" src="js/jquery.accordion.2.0.js" charset="utf-8"></script>
    <script type="text/javascript">
        $(".accordion").accordion();
    </script>
</body>

As with my carousel, you can configure the accordion more specifically by passing a configuration object to the .accordion() method:

<script type="text/javascript">
    $(".accordion").accordion({
        "handle":           "h3",
        "panel":            ".panel",
        "speed":            200,
        "easing":           "swing",
        "accordion":        true,
        "toggle":           false,
        "activeClassPanel": "open",
        "activeClassLi":    "active",
        "lockedClass":      "locked"
    });
</script>

Note: The above options are all the defaults.

Parameters for the configuration object are as follows:

Option	Value
handle	A selector that specifies the element that will contain the injected link to trigger the opening of each panel.
panel	A selector that specifies the panel element to open and close.
speed	The animation speed for opening and closing. Defaults to 200.
easing	The animation easing for opening and closing. Defaults to “swing”.
accordion	A boolean to specify whether other items should close when one is opened. If this is false, all items work independently.
toggle	A boolean to specify whether a handle can toggle opening and closing of an element.
activeClassPanel	HTML class for the active panel.
activeClassLi	HTML class for the active parent
lockedClass	HTML class used to lock any panels open.

I’ve tried to cover all the above concepts in the examples page. However, feel free to comment requests for more information, or bug reports. Alternatively, you can fork and append the GitHub repo, or register bugs there.

jQuery Carousel

2010-08-18T23:16:40+01:00

Update November 2010: Documentation for this jQuery Carousel can now be found here: Modern Carousel - Tim Huegdon’s Projects

Recently I’ve had the need to build several jQuery widgets, because, no matter how many already exist in the wild, they never seem to have all the features I require. Reinventing the wheel? Well yes, but there’s no harm if the current wheel doesn’t fit the vehicle.

Further to that, building widgets with various JavaScript libraries is actually pretty good fun, and is definitely a good way of familiarising yourself with that particular library. Having recently moved from using YUI to the more widely used jQuery, these widgets have aided my education.

So without further ado, here’s the first: the carousel.

Demo

Take a look at the page of example carousels first, to see the various implementation options.

Source Code

You can find all the source code hosted on GitHub here, with tarballs and zips:

Modern Carousel — GitHub repository

Implementation

HTML

The core HTML of the carousel should look something like this:

<div class="carousel">
    <div class="clip">
        <ul class="horizontal">
            <li>…</li>
            <li>…</li>
            <li>…</li>
            <li>…</li>
        </ul>
    </div>
</div>

The carousel, clip, and horizontal classes are all used by both the core CSS and the JavaScript to build the carousel. As a result, they are required features.

The carousel div must be the direct parent of the clip div, which in turn must be the direct parent of the ul element. The current version of the carousel script does not support the use of an ol, but it’s one of the features on my todo list.

For a vertical carousel, you should substitute the horizontal class with vertical.

CSS

The only required CSS for the carousel, is the carousel.core.css file:

<link rel="stylesheet" href="css/carousel.core.css" type="text/css" charset="utf-8">

My examples also include a CSS skin file that applies some styling to the carousel controls. Feel free to reuse this, or poke at it to come up with your own styles:

<link rel="stylesheet" href="css/carousel.skin.css" type="text/css" charset="utf-8">

JavaScript

To add the JavaScript, you simply need to include the script—along with jQuery—somewhere on your page. It’s best to include scripts at the bottom of your pages, before the closing body tag, as some browsers can delay loading of the rest of the page whilst the script loads.

<script type="text/javascript" src="js/jquery-1.4.2.min.js" charset="utf-8"></script>
<script type="text/javascript" src="js/jquery.carousel.2.0.js" charset="utf-8"></script>

Once you’ve done that, you can simply apply the carousel() method to any jQuery object, like so, to get the default options:

<script type="text/javascript">
            $(".carousel").carousel();
</script>

Alternatively, you can configure the carousel by passing it a configuration object like so:

<script type="text/javascript">
    $(".carousel").carousel({
        "loop": true,
        "autoplay": true,
        "hovercontrols": true,
        "hoverpause": true
    });
</script>

Parameters for the configuration object are as follows:

Option	Value
visiblePanes	The number of panes that are visible in the clipping area.
panesToMove	The number of panes to move the carousel when next/previous are clicked.
pagination	Boolean value to turn on or off pagination.
speed	Speed of the transition between panes.
loop	Boolean value to specify if the carousel loops after the final pane has been reached.
autoplay	Boolean value to specify whether the carousel autoplays.
hovercontrols	Boolean value to specify if the controls only display when hovering over the carousel.
hoverpause	Boolean value that activates pause when hovering the mouse over the carousel.
delay	The delay, in milliseconds, between transitions when the carousel is playing.
transition	A reference to a transition function defined at $.fn.carousel. E.g. “yourTransitionHere” would reference a function defined at $.fn.carousel.yourTransitionHere.

For more on the transition hooks, it’s probably best to take a look at the code. When I have more time, I’ll look at updating the GitHub wiki with details myself.

Controlling DHCP for VMWare Fusion VMs

2010-03-08T22:42:45+00:00

This week I built a brand new Mac for work. This meant I also needed to install VMWare Fusion and knock up a couple of virtual machines for development. Everything was working as expected until I tried using the same VMs at home, and discovered a complete lack of connectivity to—and within—the VMs, despite having active network and internet access on the host OS. After a little investigation, I discovered the issue:

When I’ve used VMWare Fusion in the past, my VMs have all been assigned IPv4 addresses in the range 172.16.0.0 through 172.31.255.255, via DHCP (the 20-bit block of RFC1918). This has always suited me fine, and has meant I’ve been able to assign host file mapping entries for those IP addresses with relative confidence since the addresses have never changed, even after reboots.

However, on the new Mac, I was surprised to find my VMWare VMs being assigned IPv4 addresses in the range 192.168.0.0 through 192.168.255.255 (the 16-bit block of RFC1918). As you can imagine, this could cause a certain degree of havoc if you’re LAN is also assigning DHCP IP addresses in that range. This is exactly what was happening in my case—and on both my work LAN and my home LAN—so VMWare was having to change the IPs each time I changed network.

To fix this issue, all I needed to do was tell VMWare to use a specific range of IP addresses when assigning DHCP. Unfortunately, VMWare doesn’t make this particularly easy (despite including a handy network configuration script). So here’s what you have to do…

A quick word on VMWare Fusion networking

Before I jump in with the “how to”, let me just take a moment to explain how networking works in VMWare Fusion:

VMWare sets up two network interfaces during installation, plus a bridge from Fusion to your host’s active network interface. The two network interfaces that are configured are “vmnet1”–the host-only network–and “vmnet8”–the NAT network interface. It is the latter of the two that we need to configure for control over VM DHCP.

Resetting the network configuration

Firstly, make sure you have shut down all your VMs (even the headless ones), and that the VMWare GUI isn’t running. We’ll also need to make sure, before we start our configuration, that we’re in a predictable state for all the configuration files used by VMWare. Handily, VMWare Fusion supplies a network interface configuration script that will allow us to do exactly that.

VMWare records a bunch of information–in regard to DHCP and the vmnets–in a file called “locations”. To begin with, we’ll need to delete that file so that it is regenerated later:

$ cd /Library/Application\ Support/VMWare\ Fusion/
$ rm -f locations

Following that, we run the handy configuration Perl script, which will automatically answer it’s own questions with the default answers:

$ sudo ./vmware-config-net.pl

Configuring a NAT network for vmnet8.

‘ping’ -q -t 10 192.168.119.1 > /dev/null status = 2
The subnet 192.168.119.0/255.255.255.0 appears to be unused.

The file /Library/Application Support/VMware Fusion/vmnet8/dhcpd.conf that this program was about to install already exists. Overwrite? [yes]

The file /var/db/vmware/vmnet-dhcpd-vmnet8.leases that this program was about to install already exists. Overwrite? [yes]

The file /var/db/vmware/vmnet-dhcpd-vmnet8.leases~ that this program was about to install already exists. Overwrite? [yes]

…

I haven’t included all the output here, but it continues on for a while.

Once you have a prompt again, it’s time to restart the network services with these update values (even though they’re not going to be final):

$ sudo ./boot.sh --restart

This will give you a “Shutting down VMware Fusion:” message, followed by a “Starting VMware Fusion:” message.

Once you get your prompt back, make sure the network interfaces have been set up correctly by running the interface configurator:

$ ifconfig

…

vmnet8: flags=8863 mtu 1500
        inet 192.168.119.1 netmask 0xffffff00 broadcast 192.168.119.255
        ether 00:50:56:c0:00:08
vmnet1: flags=8863 mtu 1500
        inet 172.16.225.1 netmask 0xffffff00 broadcast 172.16.225.255
        ether 00:50:56:c0:00:01

…

You should see something like the above, even if the addresses are slightly different. As long as the vmnet8 and vmnet1 interfaces are showing, we’re up and running again.

Defining your IP range

Remember that “locations” file we junked earlier? That’s where we need to define our IP ranges so that the network configuration script can work its magic. Following our little clean-up, that file should now be nice and simple, and ready for our changes. To do that, you’ll need to edit with root permissions:

$ mate locations

I’m using TextMate, which will prompt me for my admin password when I save the file. If you’re using an editor like vi or vim, you’ll need to sudo:

$ sudo vim locations

The settings you need to define in the locations file are as follows:

VNET_8_HOSTONLY_HOSTADDR
VNET_8_HOSTONLY_NETMASK
VNET_1_HOSTONLY_HOSTADDR
VNET_1_HOSTONLY_NETMASK
VNET_1_HOSTONLY_SUBNET

I defined mine like so, with the third digit matching the vmnet to which it is assigned, just so I have a point of reference if I need it:

…
answer VNET_8_HOSTONLY_HOSTADDR 172.16.8.1
answer VNET_8_HOSTONLY_NETMASK 255.255.255.0
…
answer VNET_1_HOSTONLY_HOSTADDR 172.16.1.1
answer VNET_1_HOSTONLY_NETMASK 255.255.255.0
answer VNET_1_HOSTONLY_SUBNET 172.16.1.0
…

Once we’ve saved our changes, we simply need to reset the network configuration using the same Perl script as earlier:

$ sudo ./vmware-config-net.pl

Configuring a NAT network for vmnet8.

Configuring a host-only network for vmnet1.

The host-only network is currently configured to use the private subnet 172.16.1.0/255.255.255.0. Do you want to keep these settings? [yes]

Finally, restart VMWare Fusion daemons and reconfigure the network interfaces:

$ sudo ./boot.sh --restart

Once that is complete, VMWare is configured to use your defined range of IP addresses for both its vmnets. You can check this by running the interface configurator again:

$ ifconfig
…
vmnet8: flags=8863 mtu 1500
        inet 172.16.8.1 netmask 0xffffff00 broadcast 172.16.8.255
        ether 00:50:56:c0:00:08 
vmnet1: flags=8863 mtu 1500
        inet 172.16.1.1 netmask 0xffffff00 broadcast 172.16.1.255
        ether 00:50:56:c0:00:01

Summary

So now you can configure your vmnets in VMWare Fusion to use a predictable range of IP addresses. That, as they say, is that.

Website builds using Make

2010-02-26T11:17:54+00:00

In the interests of improving quality in production, of eliminating repetitive tasks, and of general development time saving, it’s often a good idea to automate some of the website build process. What do I mean by “website build process”? Put simply, the task of preparation and publication to production (your live, open-to-the-internet environment), from a development environment.

In many cases, this may only be a push from your local machine to your web server, via SCP (or worse, FTP). However, if you’re working with any level of intricacy—or if you’re working in a team environment—you will, no doubt, have experienced (amongst other things) the logistical nightmare of multiple CSS and JavaScript files: First building page specific style sheets, so you maintain a single HTTP request for each, and secondly compressing those files so that any unwanted elements are removed for production (e.g. comments, needless whitespace etc.). If you’re not sure why you’d want to do these things, I’d recommend having a look at Steve Souder’s “High Performance Web Sites: Essential Knowledge for Front-End Engineers”.

In this post, I’m going to look at how you can automate the CSS and JavaScript part of the build process using Make, a handy little program that is installed with the standard build tools on most *nix based systems.

What we want to achieve

In the future, we may wish the build to pull completed code from a source-control repository (i.e. Git or SVN) down to a staging environment, and the subsequent push from staging to production. We may also wish our build to handle the pushing of static assets—including images—to a CDN, like Amazon Cloudfront, or its platform, Amazon S3.

For the purposes of this article, we simply want to handle the generation of page-specific CSS and JavaScript static assets. This is by no means a complete build, but it’s as good a place to start as any. So let’s break down that process into steps, to better understand it:

We want to amalgamate our various style sheets, or JavaScripts, into a single, page-specific sheet, so that we can reduce HTTP requests for the page. Imagine we’ve developed our pages to include CSS like so:
```
/index.html
<link rel="stylesheet" href="reset.css" type="text/css">
<link rel="stylesheet" href="core.css" type="text/css">
<link rel="stylesheet" href="top-articles.css" type="text/css">
<link rel="stylesheet" href="tna-module.css" type="text/css">

/article.html
<link rel="stylesheet" href="reset.css" type="text/css">
<link rel="stylesheet" href="core.css" type="text/css">
<link rel="stylesheet" href="article-content.css" type="text/css">
<link rel="stylesheet" href="tna-module.css" type="text/css">
```
As you can see, both pages include several of the same style sheets. However, they also each have different sheets, which means we can’t really have One Big Style Sheet To Rule Them All™ (a “method” I’ve had to suffer so often in the past on large corporate projects) without including every style ever used across the entire site. That would mean downloading styles that could be entirely redundant in the current page. If we adopt a build process to combine style sheets for a page based on what is actually needed, we can generate page-specific sheets without having to maintain duplicated style rules (which would be the case if we maintained page-specific style sheets manually). This means we reduce our HTTP requests for CSS to a single link per page:
```
/index.html
<link rel="stylesheet" href="page-index.css" type="text/css">

/article.html
<link rel="stylesheet" href="page-article.css" type="text/css">
```
This also rings true for JavaScript, which should be treated in exactly the same way.
We’d like to compress our page-specific style sheets and external JavaScript using minification. In my case, I tend to use the YUI Compressor tool, since it handles both JavaScript and CSS. There’s really no reason why you couldn’t use separate tools for each; Google Closure Compiler, for example.
The new page-specific files will require versioning so that the filenames change each time they are updated. This means we can adopt a far (or, at least, moderately far) future Expires HTTP header to better make use of caching. You can read more on this in the YDN performance best practices under ”Add an Expires or a Cache-Control Header”. There are several methods I could adopt for this:
- Add a version number on the end of the filename:
```
styles.1.0.css
```
  This means a certain degree of management in terms of major and minor releases, and often results in amusing version numbers like 1.132.
- Add some form of timestamp to the filename:
```
styles.20100221170134.css
```
  Generation of timestamps may not be consistent across servers, particularly if you’re working in a load balanced environment (i.e. the files are duplicated across front-end servers). Also, the filenames, and any related strings, are much longer.
- Create a hash (MD5, SHA etc. of the updated file and use that, or a trimmed version of it, as the filename:
```
6c7f5ed2.css
```
  This is a method I’ve only recently been introduced to, but one that clearly has its advantages. Firstly, hashing the file means we will have exactly the same result on all servers, since the file itself will be the same. It also means we don’t need to remember previous version numbers. Finally, there’s really no reason why we can’t trim the filename down to 8 characters, since the chances of generating duplicates would still be significantly low—and if we do get a duplicate, we can regenerate easily anyway.
You’ve probably figured out that I intend to use the latter hashing method in my build, since it is probably the most robust solution and the one that I prefer.
The HTML that links to our external static files will need to be updated to reflect the new versions of our built files. This simply means finding and replacing the old values.

We want to achieve these four steps with the minimum possible fuss, so we should be aiming for a “single button build process” where we can literally click a button, or run a single command to complete the build. Just because we’re only writing a small part of the final process here, doesn’t mean we should approach our solution any differently; after all, the final process will probably only run specific sub-builds in turn.

Choosing a build method

Scripting a build process on the server can be achieved in several ways:

Shell scripting: available in all operating systems and a good way to automate common command line tasks.
Server-side scripting languages: Perl, Python, Ruby, or even PHP. Any script that can be run from the command-line and that has access to the file-system would be fine. Since we’re running a web server, most of the above are probably already installed.
Specific build languages: Make, Rake, Ant, NAnt, or MSBuild. There are plenty of these around, each with its own pros and cons. They are specifically designed for the build process and, as such, are a perfect fit for what we want to achieve.

As previously stated, I’m using Make, which is pre-installed on almost every *nix-based system (it wasn’t pre-installed on my JeOS VMs, but was trivial to install from packages—and is part of the “build-essential” package you may have already installed).

Using Make

Make basically processes a configuration known as a “Makefile”. The Makefile contains a list of all target files to be created, and the commands to create them.

A simple Makefile consists of “rules” with the following structure:

target … : prerequisites …
    command
    …
    …
    …

Where:

target is usually the name of a file that is generated by a program or a set of commands; in our case, this is going to be the amalgamated and minified page-specific static files. A target can also be the name of an action to carry out, such as “clean”.
prerequisite is a file (or another target) that is used as input to create the target. A target is often likely to depend on several of these.
command is an action that Make carries out. A rule may have more than one command, each on its own line.

If you’re interested in learning more specifics about Make, I’d recommend taking a look at the GNU Make manual. The manual is available in a bunch of different formats.

Building static assets

So now that we understand the (very) basics of Make, let’s try writing a Makefile for our proposed build. I’m going to gradually build up a Makefile in this tutorial, so that it’s easy to cut and paste at any given point.

To begin, create a Makefile in your static assets directory like so:

$ vim /path/to/staticfiles/Makefile

It’s best to use a capital M on the Makefile simply to make sure it appears at the top of any ll -l commands. You may also see all caps used. Personally, I prefer the single capital M.

Let’s start by adding a comment header, defining a target for our CSS, and also the prerequisites required:

#========================================
# Static Asset Build Makefile
#
# Author: Tim Huegdon
#========================================

/path/to/staticfiles/css/page.css: /path/to/staticfiles/css/src/reset.css /path/to/staticfiles/css/src/core.css /path/to/staticfiles/css/src/top-articles.css /path/to/staticfiles/css/src/tna-module.css

Straight away you should see a problem; there’s a lot of repetition of paths there. We can probably adjust that a little by assuming the Makefile is in the /path/to/staticfiles/ directory, and replacing that with ./. However, we could do a better job if we use a Makefile variable like so:

#========================================
# Static Asset Build Makefile
#
# Author: Tim Huegdon
#========================================

css-path = /path/to/staticfiles/css/

$(css-path)build/page.css: $(css-path)src/reset.css $(css-path)src/core.css $(css-path)src/top-articles.css $(css-path)src/tna-module.css

Here you can see we’ve defined the absolute path as a variable at the top of the Makefile, and are then reusing it in both our target and our prerequisites—using an absolute path seems safer to me, but you may disagree. If we plan ahead a bit more, we can add a few more variables like so:

#========================================
# Static Asset Build Makefile
#
# Author: Tim Huegdon
#========================================

css-path =      /path/to/staticfiles/css/
css-page-target =   $(css-path)build/page.css
css-page-prereq =   $(css-path)src/reset.css \
            $(css-path)src/core.css \
            $(css-path)src/top-articles.css \
            $(css-path)src/tna-module.css

$(css-page-target): $(css-page-prereq)

Here I’ve added the prerequisites in a space-separated list, since I can use them in that format for the prerequisites, and again when I make use of the cat command later…

Concatenating the files

We now have a target for Make to examine, and a list or prerequisites that our Make rule will depend upon before it runs.

The first set of commands should handle the first stage in our build; the concatenation of the files into a single file:

#========================================
# Static Asset Build Makefile
#
# Author: Tim Huegdon
#========================================

css-path =      /path/to/staticfiles/css/
css-page-target =   $(css-path)build/page.css
css-page-prereq =   $(css-path)src/reset.css \
            $(css-path)src/core.css \
            $(css-path)src/top-articles.css \
            $(css-path)src/tna-module.css

$(css-page-target): $(css-page-prereq)
    @rm -f $(css-page-target)
    @echo "Merging CSS files…\t\t\t\c"
    @cat $(css-page-prereq) > $(css-path)build/tmp.css
    @echo "[ Done ]"

Ok, so I have a few syntactical things to explain here:

When adding a command to a rule, make sure the command line starts with a tab character. This is required, and if you’re using an editor that uses the mythical “soft tab” (i.e. spaces) you’ll enter unto a world of woe.
If a line starts with ’@’, the echoing of that line is suppressed (but not any output from that command). This just means I’m hiding the actual commands in the process; if any errors occur, those will appear in stderr.
Ending an echoed string with a \c character prevents echo printing the default trailing newline character. I’m using this to allow me to display “[ Done ]” on the same line, assuming no problems have occurred.

Right, now that I’ve explained a little of the syntax, let’s take a look at what those commands actually do:

First of all, we make sure the target file doesn’t exist by forcing an rm. Then I’m echoing a message to show the CSS build has started. Next I cat the requisites together into a temporary file. Once this is all complete, I echo a “done” message (which will appear on the same line as the start message, since I ended the first echo with a /c character).

Minification

Now we have file containing all the prerequisite CSS files together, we want to minify it with the YUI compressor:

#========================================
# Static Asset Build Makefile
#
# Author: Tim Huegdon
#========================================

css-path =      /path/to/staticfiles/css/
css-page-target =   $(css-path)build/page.css
css-page-prereq =   $(css-path)src/reset.css \
            $(css-path)src/core.css \
            $(css-path)src/top-articles.css \
            $(css-path)src/tna-module.css

yui-jar =       ~/tools/yui/yuicompressor-2.4.2.jar

$(css-page-target): $(css-page-prereq)
    @rm -f $(css-page-target)
    @echo "Merging CSS files…\t\t\t\c"
    @cat $(css-page-prereq) > $(css-path)build/tmp.css
    @echo "[ Done ]"
    @echo "Compressing merged CSS…\t\t\t\c"
    @java -jar $(yui-jar) -o $(css-page-target) $(css-path)build/tmp.css
    @echo "[ Done ]"

Here I’ve specified the path to the YUI Compressor .jar file as another variable and have run our temporary CSS file through it, outputting to the target file. I’ve also wrapped that process in the same echo trick as before.

Next, we just need to clean up that temporary file, and this rule is complete:

#========================================
# Static Asset Build Makefile
#
# Author: Tim Huegdon
#========================================

css-path =      /path/to/staticfiles/css/
css-page-target =   $(css-path)build/page.css
css-page-prereq =   $(css-path)src/reset.css \
            $(css-path)src/core.css \
            $(css-path)src/top-articles.css \
            $(css-path)src/tna-module.css

yui-jar =       ~/tools/yui/yuicompressor-2.4.2.jar

$(css-page-target): $(css-page-prereq)
    @rm -f $(css-page-target)
    @echo "Merging CSS files…\t\t\t\c"
    @cat $(css-page-prereq) > $(css-path)build/tmp.css
    @echo "[ Done ]"
    @echo "Compressing merged CSS…\t\c"
    @java -jar $(yui-jar) -o $(css-page-target) $(css-path)build/tmp.css
    @echo "[ Done ]"
    @rm -f $(css-path)build/tmp.css

Now if we run Make, our merged and compressed CSS file will be built:

$ cd /path/to/staticfiles
$ make
Merging CSS files…        [ Done ]
Compressing merged CSS…       [ Done ]
$ ls -lah css/build
total 8
drwxr-xr-x  3 auser  agroup   102B 23 Feb 14:17 .
drwxr-xr-x  5 auser  agroup   170B 19 Feb 22:45 ..
-rw-r–r–  1 auser  agroup   2.9K 23 Feb 14:17 page.css

Great!

At this point, it’s worth sense-checking the contents of your new file, to see that all your prerequisite files have been included, and that it has been correctly minified. Assuming everything is correct, let’s duplicate that rule for JavaScript files:

#========================================
# Static Asset Build Makefile
#
# Author: Tim Huegdon
#========================================

css-path =      /path/to/staticfiles/css/
css-page-target =   $(css-path)build/page.css
css-page-prereq =   $(css-path)src/reset.css \
            $(css-path)src/core.css \
            $(css-path)src/top-articles.css \
            $(css-path)src/tna-module.css

js-path =       /path/to/staticfiles/js/
js-page-target =    $(js-path)build/page.js
js-page-prereq =    $(js-path)src/carousel.js \
            $(js-path)src/awesome.js \
            $(js-path)src/beacon.js

yui-jar =       ~/tools/yui/yuicompressor-2.4.2.jar

$(css-page-target): $(css-page-prereq)
    @rm -f $(css-page-target)
    @echo "Merging CSS files…\t\t\t\c"
    @cat $(css-page-prereq) > $(css-path)build/tmp.css
    @echo "[ Done ]"
    @echo "Compressing merged CSS…\t\c"
    @java -jar $(yui-jar) -o $(css-page-target) $(css-path)build/tmp.css
    @echo "[ Done ]"
    @rm -f $(css-path)build/tmp.css

$(js-page-target): $(js-page-prereq)
    @rm -f $(js-page-target)
    @echo "Merging JS files…\t\t\t\c"
    @cat $(js-page-prereq) > $(js-path)build/tmp.js
    @echo "[ Done ]"
    @echo "Compressing merged JS…\t\c"
    @java -jar $(yui-jar) -o $(js-page-target) $(js-path)build/tmp.js
    @echo "[ Done ]"
    @rm -f $(js-path)build/tmp.js

I’m using two separate rules here; one for CSS, and one for JavaScript. If I run Make at this point, only the first rule (for the CSS) will be run. This is because Make only ever runs the first rule automatically. If I want both rules to be run, I should set up another rule that defines these rules as prerequisites.

Cleaning up old builds

It’s a good idea to define a “clean” target that removes all the files that are built in the Makefile. This means the user can start-over easily if they so desire:

#========================================
# Static Asset Build Makefile
#
# Author: Tim Huegdon
#========================================

css-path =      /path/to/staticfiles/css/
css-page-target =   $(css-path)build/page.css
css-page-prereq =   $(css-path)src/reset.css \
            $(css-path)src/core.css \
            $(css-path)src/top-articles.css \
            $(css-path)src/tna-module.css

js-path =       /path/to/staticfiles/js/
js-page-target =    $(js-path)build/page.js
js-page-prereq =    $(js-path)src/carousel.js \
            $(js-path)src/awesome.js \
            $(js-path)src/beacon.js

yui-jar =       ~/tools/yui/yuicompressor-2.4.2.jar

all: $(css-page-target) $(js-page-target)

$(css-page-target): $(css-page-prereq)
    @rm -f $(css-page-target)
    @echo "Merging CSS files…\t\t\t\c"
    @cat $(css-page-prereq) > $(css-path)build/tmp.css
    @echo "[ Done ]"
    @echo "Compressing merged CSS…\t\c"
    @java -jar $(yui-jar) -o $(css-page-target) $(css-path)build/tmp.css
    @echo "[ Done ]"
    @rm -f $(css-path)build/tmp.css

$(js-page-target): $(js-page-prereq)
    @rm -f $(js-page-target)
    @echo "Merging JS files…\t\t\t\c"
    @cat $(js-page-prereq) > $(js-path)build/tmp.js
    @echo "[ Done ]"
    @echo "Compressing merged JS…\t\c"
    @java -jar $(yui-jar) -o $(js-page-target) $(js-path)build/tmp.js
    @echo "[ Done ]"
    @rm -f $(js-path)build/tmp.js

clean:
    @rm -f $(css-page-target)
    @rm -f $(js-page-target)

To run a specific target, rather than the first in the list, you simply pass it as an argument to Make. Let’s run our clean target like so:

$ make clean

Now if you run Make again, it should build both the CSS and the JavaScript page files:

$ cd /path/to/staticfiles
$ make
Merging CSS files…        [ Done ]
Compressing merged CSS…       [ Done ]
Merging JS files…     [ Done ]
Compressing merged JS…        [ Done ]
$

Hash pipe (cut)

The next step is to hash our new page-specific CSS and JS files, and copy them to a higher level (so we don’t need to have the “build” directory on production). To do that we’ll want to pass the contents of the file to something that will generate a SHA1 checksum. On some systems, you can use shasum, but on my Mac that didn’t exist, so I’m using openssl sha1 instead, like so:

$ cat file | /usr/bin/openssl sha1

This will output a full SHA1 checksum to stdout. We actually want to use it as our filename, and the full checksum is likely to be too long. We can fix that by using the cut command as follows:

$ cat file | /usr/bin/openssl sha1 | cut -c1-8

Now if we want to use that as a filename, we can generate a variable in our Makefile. Let’s do that for both the JavaScript and the CSS:

#========================================
# Static Asset Build Makefile
#
# Author: Tim Huegdon
#========================================

css-path =      /path/to/staticfiles/css/
css-page-target =   $(css-path)build/page.css
css-page-prereq =   $(css-path)src/reset.css \
            $(css-path)src/core.css \
            $(css-path)src/top-articles.css \
            $(css-path)src/tna-module.css

js-path =       /path/to/staticfiles/js/
js-page-target =    $(js-path)build/page.js
js-page-prereq =    $(js-path)src/carousel.js \
            $(js-path)src/awesome.js \
            $(js-path)src/beacon.js

yui-jar =       ~/tools/yui/yuicompressor-2.4.2.jar

all: $(css-page-target) $(js-page-target)

install: css-build = `cat $(css-page-target) | /usr/bin/openssl sha1 | cut -c1-8`.css
install: js-build = `cat $(js-page-target) | /usr/bin/openssl sha1 | cut -c1-8`.js
install: all

$(css-page-target): $(css-page-prereq)
    @rm -f $(css-page-target)
    @echo "Merging CSS files…\t\t\t\c"
    @cat $(css-page-prereq) > $(css-path)build/tmp.css
    @echo "[ Done ]"
    @echo "Compressing merged CSS…\t\c"
    @java -jar $(yui-jar) -o $(css-page-target) $(css-path)build/tmp.css
    @echo "[ Done ]"
    @rm -f $(css-path)build/tmp.css

$(js-page-target): $(js-page-prereq)
    @rm -f $(js-page-target)
    @echo "Merging JS files…\t\t\t\c"
    @cat $(js-page-prereq) > $(js-path)build/tmp.js
    @echo "[ Done ]"
    @echo "Compressing merged JS…\t\c"
    @java -jar $(yui-jar) -o $(js-page-target) $(js-path)build/tmp.js
    @echo "[ Done ]"
    @rm -f $(js-path)build/tmp.js

clean:
    @rm -f $(css-page-target)
    @rm -f $(js-page-target)

Here I’ve added a new “install” target, which uses the “all” target as its only prerequisite. I’ve also defined two variables above the target as target-specific variables. This scopes the variables to this rule alone.

I’ve used the target “install” to conform with accepted Make standards. This means that, each time we want to refresh the CSS, we only need to type the following (which is another accepted standard):

$ make clean install

As you can see, you are able to pass multiple targets into the make command as arguments.

Using the hash for good, not evil

This is all very well and good, but even though we have the new filename stored in a variable, we’re not doing anything with it. Let’s rectify that:

#========================================
# Static Asset Build Makefile
#
# Author: Tim Huegdon
#========================================

css-path =      /path/to/staticfiles/css/
css-page-target =   $(css-path)build/page.css
css-page-prereq =   $(css-path)src/reset.css \
            $(css-path)src/core.css \
            $(css-path)src/top-articles.css \
            $(css-path)src/tna-module.css

js-path =       /path/to/staticfiles/js/
js-page-target =    $(js-path)build/page.js
js-page-prereq =    $(js-path)src/carousel.js \
            $(js-path)src/awesome.js \
            $(js-path)src/beacon.js

yui-jar =       ~/tools/yui/yuicompressor-2.4.2.jar

all: $(css-page-target) $(js-page-target)

install: css-build = `cat $(css-page-target) | /usr/bin/openssl sha1 | cut -c1-8`.css
install: js-build = `cat $(js-page-target) | /usr/bin/openssl sha1 | cut -c1-8`.js
install: all
    @cp $(css-page-target) $(css-path)$(css-build)
    @cp $(js-page-target) $(js-path)$(js-build)

$(css-page-target): $(css-page-prereq)
    @rm -f $(css-page-target)
    @echo "Merging CSS files…\t\t\t\c"
    @cat $(css-page-prereq) > $(css-path)build/tmp.css
    @echo "[ Done ]"
    @echo "Compressing merged CSS…\t\c"
    @java -jar $(yui-jar) -o $(css-page-target) $(css-path)build/tmp.css
    @echo "[ Done ]"
    @rm -f $(css-path)build/tmp.css

$(js-page-target): $(js-page-prereq)
    @rm -f $(js-page-target)
    @echo "Merging JS files…\t\t\t\c"
    @cat $(js-page-prereq) > $(js-path)build/tmp.js
    @echo "[ Done ]"
    @echo "Compressing merged JS…\t\c"
    @java -jar $(yui-jar) -o $(js-page-target) $(js-path)build/tmp.js
    @echo "[ Done ]"
    @rm -f $(js-path)build/tmp.js

clean:
    @rm -f $(css-page-target)
    @rm -f $(js-page-target)
    @rm -f $(css-path)*.css
    @rm -f $(js-path)*.js

Here I’ve copied the files to the higher css-path/ and js-path/ directories. This means I won’t need to sync the build/ and src/ directories to production.

I’ve also added two rm commands to the “clean” target that remove all built CSS and JS files in the respective directories, regardless of name. Note: make sure you’re not doing this recursively, otherwise you’ll trash all your valuable source files.

Now if you look in the css-path/ directory, you should see something like:

$ ls -lah
total 4.0K
drwxr-xr-x 5 auser agroup  170 2010-02-25 22:36 .
drwxr-xr-x 7 auser agroup  238 2010-02-25 22:14 ..
-rw-r–r– 1 auser agroup 2.9K 2010-02-25 22:36 6c7f5ed2.css
drwxr-xr-x 3 auser agroup  102 2010-02-25 22:36 build
drwxr-xr-x 4 auser agroup  136 2010-02-25 21:54 src
$

Great, so now we have a (reasonably) unique filename for our concatenated and minified CSS; and if you look in your JavaScript directory, you should see something similar.

The next step is to actually find and replace the inclusion of your file in your HTML.

Automagically updating the HTML

We basically need to run a find and replace on the HTML that includes your external CSS and JavaScript files, whether that is an SSI or a full HTML page. To do that, we should probably specify those files as variables, so that they can be easily maintained:

#========================================
# Static Asset Build Makefile
#
# Author: Tim Huegdon
#========================================

css-inc-file =      /path/to/publicfiles/page.html
js-inc-file =       /path/to/publicfiles/page.html

css-path =      /path/to/staticfiles/css/
css-page-target =   $(css-path)build/page.css
css-page-prereq =   $(css-path)src/reset.css \
            $(css-path)src/core.css \
            $(css-path)src/top-articles.css \
            $(css-path)src/tna-module.css

js-path =       /path/to/staticfiles/js/
js-page-target =    $(js-path)build/page.js
js-page-prereq =    $(js-path)src/carousel.js \
            $(js-path)src/awesome.js \
            $(js-path)src/beacon.js

yui-jar =       ~/tools/yui/yuicompressor-2.4.2.jar

all: $(css-page-target) $(js-page-target)

install: css-build = `cat $(css-page-target) | /usr/bin/openssl sha1 | cut -c1-8`.css
install: js-build = `cat $(js-page-target) | /usr/bin/openssl sha1 | cut -c1-8`.js
install: all
    @cp $(css-page-target) $(css-path)$(css-build)
    @cp $(js-page-target) $(js-path)$(js-build)

$(css-page-target): $(css-page-prereq)
    @rm -f $(css-page-target)
    @echo "Merging CSS files…\t\t\t\c"
    @cat $(css-page-prereq) > $(css-path)build/tmp.css
    @echo "[ Done ]"
    @echo "Compressing merged CSS…\t\c"
    @java -jar $(yui-jar) -o $(css-page-target) $(css-path)build/tmp.css
    @echo "[ Done ]"
    @rm -f $(css-path)build/tmp.css

$(js-page-target): $(js-page-prereq)
    @rm -f $(js-page-target)
    @echo "Merging JS files…\t\t\t\c"
    @cat $(js-page-prereq) > $(js-path)build/tmp.js
    @echo "[ Done ]"
    @echo "Compressing merged JS…\t\c"
    @java -jar $(yui-jar) -o $(js-page-target) $(js-path)build/tmp.js
    @echo "[ Done ]"
    @rm -f $(js-path)build/tmp.js

clean:
    @rm -f $(css-page-target)
    @rm -f $(js-page-target)
    @rm -f $(css-path)*.css
    @rm -f $(js-path)*.js

I’ve added two variables here in case, at some point in the future, we decide to break the inclusions out into separate files.

Now we’ve defined those files in the Makefile, we should really use them for something. I’m going to use sed to find and replace the references to the external static files:

#========================================
# Static Asset Build Makefile
#
# Author: Tim Huegdon
#========================================

domain =            yoursite.com

css-inc-file =      /path/to/publicfiles/page.html
js-inc-file =       /path/to/publicfiles/page.html

css-path =      /path/to/staticfiles/css/
css-page-target =   $(css-path)build/page.css
css-page-prereq =   $(css-path)src/reset.css \
            $(css-path)src/core.css \
            $(css-path)src/top-articles.css \
            $(css-path)src/tna-module.css

js-path =       /path/to/staticfiles/js/
js-page-target =    $(js-path)build/page.js
js-page-prereq =    $(js-path)src/carousel.js \
            $(js-path)src/awesome.js \
            $(js-path)src/beacon.js

yui-jar =       ~/tools/yui/yuicompressor-2.4.2.jar

all: $(css-page-target) $(js-page-target)

install: css-build = `cat $(css-page-target) | /usr/bin/openssl sha1 | cut -c1-8`.css
install: js-build = `cat $(js-page-target) | /usr/bin/openssl sha1 | cut -c1-8`.js
install: all
    @cp $(css-page-target) $(css-path)$(css-build)
    @cp $(js-page-target) $(js-path)$(js-build)
    @echo "Linking to updated CSS and JavaScript…\t\c"
    @sed -i '' "s/http:\/\/static\.$(subst .,\.,$(domain))\/css.[^\"]*/http:\/\/static\.$(subst .,\.,$(domain))\/css\/$(subst .,\.,$(css-build))/" $(css-inc-file)
    @sed -i '' "s/http:\/\/static\.$(subst .,\.,$(domain))\/js.[^\"]*/http:\/\/static\.$(subst .,\.,$(domain))\/js\/$(subst .,\.,$(js-build))/" $(js-inc-file)
    @echo "[ Done ]"
    @echo "Installation is complete."

$(css-page-target): $(css-page-prereq)
    @rm -f $(css-page-target)
    @echo "Merging CSS files…\t\t\t\c"
    @cat $(css-page-prereq) > $(css-path)build/tmp.css
    @echo "[ Done ]"
    @echo "Compressing merged CSS…\t\c"
    @java -jar $(yui-jar) -o $(css-page-target) $(css-path)build/tmp.css
    @echo "[ Done ]"
    @rm -f $(css-path)build/tmp.css

$(js-page-target): $(js-page-prereq)
    @rm -f $(js-page-target)
    @echo "Merging JS files…\t\t\t\c"
    @cat $(js-page-prereq) > $(js-path)build/tmp.js
    @echo "[ Done ]"
    @echo "Compressing merged JS…\t\c"
    @java -jar $(yui-jar) -o $(js-page-target) $(js-path)build/tmp.js
    @echo "[ Done ]"
    @rm -f $(js-path)build/tmp.js

clean:
    @rm -f $(css-page-target)
    @rm -f $(js-page-target)
    @rm -f $(css-path)*.css
    @rm -f $(js-path)*.js

The first thing I’ve done here is to define another variable for the domain name of the site. This is because I’m going to need to use it to identify the links to the filenames in the regular expression.

Next I added the sed commands to our “install” target, using the -i option (edit files in-place). The regular expression is basically looking for a reference to http://static.yourdomain.com/css/something and replacing it with the full URI to our newly generated file.

Once this step is finished, the installation of our newly built static files is complete; and, as such, so is our Makefile. As previously stated, to run a new build—that also cleans up old builds—simply type the following:

$ make clean install

A better development environment

Instigating a build process in your environment allows you the flexibility to design your development environment for ease of maintenance, rather than having to think about optimisation and structure for production. Your development environment should be quite heavily different from staging or production; you want it to be optimised for development, maintenance, and bug-fixing, rather than optimised for speed, efficiency, and load.

Ultimately, this means that the build allows you the freedom to write better CSS and JavaScript, in well separated files. These files are then easily maintained by developers—since they’ll be easier to read and well commented—and also easier to maintain through source control—since the separation into a greater number of files means more control of more specific code.

Premature Publish

2010-02-26T10:50:22+00:00

Last night I accidentally published a new post, “Website builds using Make” before it was complete. Apologies if you read it and wondered why it never reached a proper conclusion; I’m still finishing it off.

I really really need to move off of WordPress.

Needless to say, the completed version of that post should be published very soon!

Mac VirtualBox Linked Clones

2010-02-12T14:50:45+00:00

I’ve been a VMWare Fusion user for quite some time, and I’ve been more than happy with the experience, but the most recent version has dumped an unsupported feature that is essential to my personal workflow: Headless Mode.

One of the VMWare developers responded to requests for information on the missing headless feature in the “Fusion 3 Headless?” VMWare communities thread. Needless to say, the feature may or may not reappear in a newer version of Fusion, but certainly doesn’t in the current version.

With that in mind, I decided it was about time I took a look at VirtualBox, the Open Source virtualisation product from Sun (now Oracle). I’d already heard mostly good things about VirtualBox—except some complaints of slow Windows VMs—and it definitely includes a Headless Mode of its own. What’s more, after a bit of investigation on t’internet, I discovered that linked cloning also appeared to be supported, and linked clones are essential to my workflow.

Create the master

Before we can create clones, we’re going to need a master. In my examples I’ll be creating and cloning an Ubuntu VM, but there’s really no reason why you couldn’t do the same with Windows—in fact, it would be of benefit for cross-browser testing in the various versions of IE, since you can have an instance of Windows for each.

Before we begin creating the master VM, it’s worth updating the VirtualBox preferences so that it puts your files in a predictable place. The default is ~/Library/VirtualBox/ but I have a specific ~/vms/ directory that I like to use. When doing this, be aware that new virtual disk files are created before new VM directories in the “Create New VM” process. This means it’s worthwhile having two directories; one for virtual disk files, and one for the virtual machine directories. This is a pain in the proverbials for portability, but don’t worry too much about it since VirtualBox creates the snapshot difference disks (which will be used by our clones) in the virtual machine directory.

Let’s begin:

Create new VM (cmd+N) and name it whatever you want. It’s probably best to make some reference to it being your master, but that’s entirely up to you. I used “atum” (first of the Egyptian Gods), and specified my OS as Linux / Ubuntu, unsurprisingly.
When asked to specify, I went with the recommended RAM.
The next step in the wizard is to specify a new or existing virtual disk. Since this is our master, select “Create New Disk”.
A second wizard will pop up, specifically for defining the new virtual disk. Use the following options: dynamically expanding, 10Gb (although, VirtualBox recommends 8Gb for Ubuntu, which would also be fine), located in your new default virtual disks folder.
Click finish to end the Create New Virtual Disk Wizard. VirtualBox may pause for a second whilst creating the new .vdi and updating the various XML files that reference it.
After a second or two, you should be presented with the original wizard again. Click “Finish” to complete the creation process.
Make sure new VM is selected in the list of VMs in your VirtualBox window.
We need to make sure we can connect to the our new VM locally. To do that, you’ll need to add a host-only network adaptor in addition to the standard NAT one. Click the “Network” heading on the right side of the VirtualBox window, and the settings dialog should pop up. Make sure “Adaptor 1” is enabled and using “NAT”, then select “Adaptor 2” and click “Enable”. Change “Attached to” to “Host-only Adaptor” and just use the suggested name.
Next we need to tell our VM to boot and install Ubuntu the first time we run our VM. Click “Storage” either in the right hand side of the VirtualBox window, or in the tab bar of the settings dialog that should still be open from the previous step. Choose the CD/DVD device and point it at your Ubuntu ISO, ready for your install.
Click “Ok” to close the settings dialog.
Start your new VM and it should boot into the Ubuntu Installation.
At the installation introduction screen, push F4 and select “Install Minimal Virtual Machine”, then start the install as normal. This will install Ubuntu as JeOS, which is specifically configured for virtual machine use.
Install Ubuntu however you like.
Once your install is complete, make sure to remove the Ubuntu ISO from the CD/DVD device. This won’t come back to bite you straight away since the first few reboots appeared to ignore it during my install. However, I did end up spending a good 10 minutes wondering why I could no longer SSH to a VM that had previously been working fine before reboot during my install.
Top tip: Depending on the keyboard set-up you opted for during the installation of Ubuntu, you may want to change the AltGr key to be the left alt (option) on your Mac keyboard. You can do this with the following bash command in Ubuntu:
```
$ sudo dpkg-reconfigure console-setup
```
If you don’t do this, you may have trouble trying to enter # characters later.
Next, let’s make sure apt is all up to date. Do that by running the following commands:
```
$ sudo apt-get update
$ sudo apt-get upgrade
```
If you’re using an older than current version of Ubuntu, you may want to do the following (note: the second command reboots the VM):
```
$ sudo apt-get dist-upgrade
$ sudo shutdown -r now
```
Next I installed vim as my editor of choice.
```
$ sudo apt-get install vim
```
Since the basic Ubuntu terminal is a little sparse, you’ll probably want to be able to SSH into the VM and use your terminal of choice. To do that, we need to activate the secondary interface for the adaptor we added earlier.

First, edit the following file:
```
$ sudo vim /etc/network/interfaces
```
Add the following lines at the bottom:
```
# The secondary network interface
auto eth1
iface eth1 inet static
    address 192.168.56.2
    netmask 255.255.255.0
```
It’s important to use that IP address, since the vboxnet0 is configured on 192.168.56.1 by default.
Activate the eth1 interface:
```
$ sudo ifup eth1
```
Now make sure ssh is installed:
```
$ sudo apt-get install ssh
```
Following all that, you should now be able to SSH to your new VM from your host:
```
$ ssh 192.168.56.2
```
You should now install all the software you want shared across your clones. In my case, that was the usual scripting languages, web servers, database servers etc. used for web development.

Once you’re happy with your build, move on.
Let’s clear the persistent net rules on our master disk, just so they don’t come back to bite us later. In my VMWare tutorial, I did this on the clones, but it makes far more sense to do it on the master before cloning:
```
$ sudo cp /dev/null /etc/udev/rules.d/70-persistent-net.rules
```
Now that we’re done with SSH, we should bring down the secondary interface so that it’s easier to configure on our clones later. Bring the interface down like so:
```
$ sudo ifdown eth1
```
Next, go back into the interface configuration file, and comment out the second interface you added:
```
$ sudo vim /etc/network/interfaces
# The secondary network interface
# auto eth1
# iface eth1 inet static
#     address 192.168.56.2
#     netmask 255.255.255.0
```
Now, when you create your clones, you’ll be able to uncomment and make sure the IP doesn’t clash with other cloned VMs.
Finally, shutdown your VM so we can break it down to the parts we need:
```
$ sudo shutdown -P now
```

Prepare the master for cloning

Unlike VMWare, preparing a VirtualBox VM for cloning is actually a fairly straightforward procedure, and can be completed in three fairly simple steps:

First, you’ll need to “release” the virtual disk from the virtual machine using the Virtual Media Manager in VirtualBox.

To do this, just open VirtualBox and choose the Virtual Media Manager from the File menu (alternatively, push cmd+D). Select your master virtual disk in the “Hard Disks” tab, and click the “Release” button in the toolbar.
Next it’s worthwhile telling VirtualBox that your master virtual disk is immutable. Run the following command in a terminal on your host machine:
```
$ vboxmanage modifyhd --type immutable atum.vdi
```
Where atum.vdi is the name of your master virtual disk. Don’t worry about using the full path, VirtualBox should already know where your virtual disks all are (thanks to the Virtual Media Manager).
Lastly, you may like to make the .vdi file read-only:
```
$ chmod a-w /path/to/atum.vdi
```

And we’re done. The master virtual disk is now ready to be cloned.

Create a clone

So let’s go ahead and make a clone of our master:

Create a new VM (cmd+N) and name it whatever you like (I used “ra”, to follow that Egyptian deity theme). Obviously, make sure to use the same OS as your master when you define your new VM.
Once again use the recommended RAM.
Choose “Use existing hard disk” and select your master, which should already appear in the drop-down since it’s already in the Virtual Media Manager.

Once you do this, VirtualBox will automatically create a snapshot difference disk in the “Snapshots” child directory of your virtual machine directory. So in my case, that was here:
```
/path/to/vms/ra/Snapshots/
```
Be aware that the difference disks are named with a UUID, and that they reset automatically when the cloned virtual machine is rebooted.

To stop the virtual disk from auto-resetting every time you reboot, you’ll need to run the following command once the snapshot has been created:
```
$ vboxmanage modifyhd --autoreset=off /path/to/vms/ra/Snapshots/{long-uuid}.vdi
```
Click “Finish” in the “Create New VM” dialog.
Once again, we need to set up the second network interface, so go ahead and do the same as we did for the master: Click “Network” and make sure “Adaptor 1” is enabled and using “NAT”. Select “Adaptor 2” and click “Enable”. Change “Attached to” to “Host-only Adaptor” and use the suggested name.
Start your new VM and log in with the same user you used on the master. Note your clone is still using the hostname of your master. We’ll update that in a second.
It’s time to reactivate the secondary interface within Ubuntu:
```
$ sudo vim /etc/network/interfaces
```
Uncomment the secondary interface and update the last number of the IP address so that it is unique on the vboxnet.

Now activate the interface again:
```
$ sudo ifup eth1
```
You can now ssh to the clone on the address you specified in the secondary interface configuration.
Finally, let’s update the hostname of our clone in two places, so that it’s unique. First, the hostname file itself, and secondly in the /etc/hosts file so the loopback doesn’t get confused:
```
$ sudo vim /etc/hostname
ra
$ sudo vim /etc/hosts
127.0.0.1    ra
```
Finally restart the hostname service:
```
$ sudo /etc/init.d/hostname.sh start
```

And that’s it; you now have a fully functioning linked clone. Feel free to make some more from your master.

Headless Mode

Now that we have an army of working linked clone virtual machines, it’s probably worthwhile making use of Headless Mode so that they are all running in the background without you needing the VirtualBox GUI invading your desktop and draining resources.

First, shutdown your VM by logging in and running this:

$ sudo shutdown -P now

Now you can start it back up by running the following command on your host:

$ (VBoxHeadless -v=off -s ra &)

Be aware that this runs the VBoxHeadless script in the background, as it doesn’t normally auto-exit. I’m running it in a sub-shell because someone far more knowledgeable than myself suggested that this was safer. Cheers Norm.

Summary

Firstly, I’d like to thank a few people for their help with this: Bradley Wright for being the usual tech. sounding board, Stuart Colville for clearing up the mystery of VirtualBox network adaptors, and Mark Norman Francis for being considerably better at server administration and bash than me.

You should be able to get a good virtualised development environment up in VirtualBox now. Feel free to post any questions in the comments, and if I can’t answer personally, I’ll pester someone that might be able to.

In the future I’m going to look at automating this process, but the VBoxManage command is considerably different to VMWare’s vmrun and it’s going to take me a little while to figure out stuff like creating new VMs, and releasing virtual disks without removing the entire IDE controller from the master (yeah; I already tried).

The value of trying things out

2010-01-27T19:25:06+00:00

I was recently reminded of an interesting situation that arose a while back at Yahoo! London; a situation that was resolved using simple testing and demonstration:

We had recently begun extensive performance optimisation of the Yahoo! Eurosport site in an attempt to improve our YSlow grades. At the time, the design of the Yahoo! Eurosport site used rounded corners on all of the modules. Due to the fact that we had developed the modules to be bulletproof, and because the backgrounds varied slightly for different modules, this ended up with a considerable overhead on CSS backgrounds.

Yahoo! Eurosport before:

Our first idea was to approach UED to find out if we could just remove the rounded corners and instantly improve our performance. In our eyes, this was an insignificant visual change with an extremely high pay-off in regard to the performance boost. However, in UED’s eyes, this change was unacceptable and would diminish the impact of the design enough for them to categorically say no.

So we accepted the decision and went back to our desks to continue our optimisation. But, by its very nature, YSlow just couldn’t let it rest; the resulting reports continually highlighted those backgrounds as a significant issue.

With that in mind, one of the front-end developers on our team decided to take matters into his own hands and create a second style sheet without the rounded corner images just for the purpose of testing. The resultant comparisons were, unsurprisingly, an extensive boost in performance; both in download times, and in rendering time.

Several days later, we were chatting with the UEDs on the team and happened to apologise for leaving the non-rounded corner style sheet active on the testing environment. It was then that we discovered they hadn’t noticed at all—and, what’s more, they were pleasantly surprised to find the impact on the design was minimal. What’s more, we were able to demonstrate the difference in rendering time to them in their own browsers.

Yahoo! Eurosport after:

Following that little test, and subsequent demonstration to the UED team, it was unanimously agreed that the gain in performance was significantly greater than the loss of visual impact, and we were able to move the updated CSS live.

The only downside to this process was the fact that we had to leave a lot of legacy HTML wrappers in place that would normally have hosted the extra background images. In a perfect world, we would have been able to strip all that HTML as well. Unfortunately, the templates were developed in such a hurried way that this task was basically impossible, and those redundant wrappers are in place to this day.

I think that little anecdote illustrates the value in testing changes even if you thoroughly disagree with them at the outset. You’ll either be proven entirely right, or you’ll be pleasantly surprised.