jane and robot

URL Referrer Tracking

Nathan Buggia — Sun, 30 Nov 2008 13:21:00 -0700

There may be instance when you want to track the source of a request, and a common way of doing so is by using tracking parameters in URLs. Unfortunately, implementing referrer tracking in this way can result in significant issues with search engines. In particular, it can cause duplicate content issues (since the search engine bot finds multiple valid URLs that point to the same page) and ranking issues (since all the links to the page aren't to the same URL).

Let's say that Jane and Robot uploaded two different online training seminars to YouTube as part of a viral marketing effort to drive more traffic to our site. To gauge our return on investment from each of these seminars, we've added a tracking parameter to the link within each YouTube description that a customer can click on to learn more, here are the two URLS: http://janeandrobot.com/?from=promo-seminar-1 and http://janeandrobot.com/?from=promo-seminar-2. Each would bring the customer to our home page (the same page served by http://janeandrobot.com) and we would track the conversions based on the from parameter in the URL.

While this solution may seem to work well initially, it can result in low quality tracking data and impact our search acquisition. Here's a summary of the major problems:

Duplicate content - search engines sometimes have difficulty determining if two URLs contain the exact same page (see canonicalization for more information). In this case, we're creating this problem because we've created multiple URLs for the same page. Search engines are likely to find all three URLs for the home page and store/ rank them as separate content within their index. This could cause the search engine robots to crawl the page three times instead of just once (which may not be a big deal if we are only tracking two promotions, but could become a big problem if we used similar tracking parameters for many other campaigns and URLs). Not only are the robots using more bandwidth than is necessary, but since they don't crawl a site infinitely, they could spend all the allotted time crawling duplicate pages and never get to some of the good unique pages on the site.
Ranking - search engines use the number of quality links pointing to a URL as a major signal in determining the authority and usefulness of that content. Because we now have three different URLs pointing to the same page, people have three choices when linking to it. The result is a lower rank for all of the variations of the URL. Search engines generally filter out duplicates, so for instance, if the original (canonical) home page has 100 incoming links and each URL with a tracking parameter has 25 links, then search engines might filter out the two URLs with fewer links and show only the canonical URL, ranking it at position eight for a particular query based on those 100 incoming links. If all incoming links were to the same URL, then search engines would count 150 links to the home page and might rank it at position three for that same query.

Another danger is that if one of the YouTube promo videos becomes exceptionally popular, its promo URL might gain more links than the original home page URL. Using this same example, if one of the promo URLs gained 200 links, search engines might choose to display it in the search results over the original home page. This could cause a confusing experience for potential customers who are looking for your home page (http://janeandrobot.com/?from=promo-seminar-1 doesn't look like a home page and searchers might be less likely to click on it, thinking it's not the page they're looking for). It's also not ideal from a branding perspective.
Reporting quality - as social networking sites become more popular, we become more of a sharing culture online. Many people use bookmarks, and online bookmarking sites such as Delicious, email, and other sharing sites such as Facebook, Twitter, and FriendFeed to save and share URLs. They'll click on on a URL, and if they like it, copy and paste it from the browser's address bar. If the link they're saving/sharing happens to be one of our promotional links, then they have preserved this link for all time, and everyone who clicks through the link will look identical to someone coming through the promo. This skews the reporting numbers of who went to the site after viewing the video -- which was why we set up the tracking parameters in the first place!

Implementation Options

Unfortunately there is no perfect solution for this scenario, and what works best for you depends on your infrastructure and situation. Here we've listed several common solutions that you can choose from to improve your own implementation. We generally recommend the first solution (Redirects), but there are pros and cons to each option that you should review carefully before making your decision.

Redirects (and Cookies)

The first option strives to solve the problem by trapping all of the promotional requests, recording the tracking information, then removing the tracking parameter from the URL. This can be time consuming to implement, but it is the best all-round scenario to address the three major issues listed above.

If you wanted to get fancy, and track a user's entire session based on your referral parameter, then you can use this method as well and simply set a cookie on the client machine at the same time you trap the request. This is recommended to understand the value of traffic from different sources. In either case, here are the steps you'll need to undertake:

1. Trap the incoming request - find where you web site application's logic processes the HTTP request for your page. Trap each request at that point and check if it has a tracking parameter. If it does, record this in your internal referral tracking system. You can record this either in your server logs, or in a custom referral tracking database you maintain on your own.

If you also would like to track the entire user's session, then you should also use this opportunity to set a cookie on the client.

2. Implement the redirect - next step is to implement a 301 redirect from the current URL to the same page without the tracking parameter (or the canonical URL). Don't for get to use the cache-control attribute in the HTTP header to ensure that all the requests come to your server and don't get handled automatically in some network-based cache. Here's what a sample redirect header might look like:

301 Moved Permanently

Cache-Control: max-age=0

Note that ASP.Net and IIS both use 302 redirects by default, so you many need to manually create the 301 response code.

The way this works is that when a search engine encounters a promotional URL (http://janeandrobot.com/?from=promo-seminar-1) it issues an HTTP GET request to the URL. The HTTP response tells the search engine that this page has been permanently moved (301 Redirect) and provides the new address (the same as the old address but without the tracking parameter). The search engine then discards the first URL (with the tracking code) and only stores the second URL (without the tracking code). And everything is right in the world.

This implementation is one of the best options, but it does have some limitations:

One downside of this method is that it requires you to manage your own referral tracking system. Because it traps the referral parameters and removes them from the URL before the page actually loads, 3rd party referral tracking applications like Google Analytics, Omniture, WebTrends or Microsoft adCenter Analytics will not be able to track these referrals.

Canonical ULR <LInk /> Tag

Possibly the simplest option to solve this issue is to take advantage of a new standard recently adopted by Google, Yahoo and Microsoft Live Search. Their solution to this problem is to use a new attribute of the <link /> tag to explicitly tell them what the canonical URL for the page is. Assuming the <link /> tag has been created correctly, the search engines will treat this like the a 301 redirect to the canonical URL.

Here's an example of using this tag:

<html>

   <head>

      <link rel="canonical" href="http://janeandrobot.com" />

   </head>

</html>

Here's a few notes about implementing this tag:

Search engines view this as a hint, not a command. Implementing this tag isn't a guarantee, although Google said they will try their best to make it work. The reason they can't give any guarantees is because they may detect that you are implementing it incorrectly, or it is being used for some type of spammy scenario.
Relative or absolute URL are supported within the href attribute. However, I recommend that you use absolute URLs whenever possible. This helps the search engines further normalize the URLs because they see what protocol (http or https) you use, and whether or not you are prefixing your domain with "www.".
Sub-domains are supported, separate domains are not. With this tag you can specify a separate a different sub-domain, for example within this URL (http://janeandrobot.com?from=promo-seminar-2) you could specify this canonical URL (http://videos.janeandrobot.com). However, the <link /> tag would not be valid if you specific a completely different domain like this http://janeandrobot-videos.com.
Common Pitfalls... You'll want to ensure that you don't do anything silly like (i) create an infinite loop with two canonical tags pointing to each other (ii) have the canonical tag point to a page that returns a 404 status code. You should also make sure that your canonical URL is generally a short and simple URL.

While this implementation seems a little too good to be true, there are a few potential downsides. The first is that if you implement it incorrectly, the search engines will simply ignore it, and that could be complicated to debug. The other issue is that it fixes issues #1 (duplicate content) and #2 (ranking) but does nothing to fix the 3rd issue of reporting. Still, given all of that I would likely implement this option first and do the others when I had some spare dev cycles.

URL Fragment

A simple and elegant option is to simply place the tracking parameter behind a hash mark in the URL, creating a URL fragment. Traditionally, these are used to denote links within a page, and are ignored completely by search engines. In fact, they simply truncate the URL fragment from the URL.

Old URL

http://janeandrobot.com/?from=promo-seminar-1
http://janeandrobot.com/?from=promo-seminar-2

New URL with URL Fragment

http://janeandrobot.com/#from=promo-seminar-1
http://janeandrobot.com/#from=promo-seminar-1

By default Google Analytics will ignore the fragment as well, however there is a simple work around that was provided to us by Avinash Kaushik, Google's web metrics evangelist. Using the following JavaScript:

var pageTracker = _gat._getTracker("UA-12345-1");

// Solution for domain level only

pageTracker._trackPageview(document.location.pathname + "/" + document.location.hash);

// If you have a path included in the URL as well

pageTracker._trackPageview(document.location.pathname + document.location.search +

                           "/" + document.location.hash);

You can create a few additional variations of this if you also have additional queries in the URL you would like to track. Check with your web analytics provider to find out if you need to customize your implementation to account for using URL fragments for tracking.

Does this sound too simple and easy to be true? There are a couple downsides to this approach:

This option fixes issues 1 (duplicate content) & 2 (ranking) listed above, but it will not address the 3rd issue of reporting. You could still encounter some reporting issues using this method if people are bookmarking or emailing around the URL.
Typically you'll have to write some custom code to parse the URL fragment. Since it's a non-standard implementation, standard methods may not support this.

Robots Exclusion Protocol

Another relatively simple solution is to use robots.txt to ensure that search engines are not indexing URLs that contain tracking parameters. This method enables you to ensure that the original (canonical) version of the URL is always the one indexed and avoids the duplicate content issues involving indexing and bandwidth.

Assuming that all of our tracking parameters will follow a similar pattern to this:

http://janeandrobot.com/?from=<PromoID>

we can easily create a pattern that will match for this. Below is a robots.txt file that implements the pattern:

# Sample Robots.txt file, single query parameter

User-agent: *

Disallow: /?from=

The first line means that this rule should apply to all search engines (or robots crawling your site), and the second line tells them that they can't index any URLs that start with 'janeandrobot.com/?from=' and some type of promotional code of any length. See complete information on using the Robots Exclusion Protocol. Use this pattern if you will have multiple query parameters:

# Sample Robots.txt file, multiple query parameters

User-agent: *

Disallow: /*from=

Once you've implemented the pattern appropriate for your site, you can easily check to see if it is working correctly by using the Google Webmaster Tools robots.txt analysis tool. It enables you to test specific URLs against a test robots.txt file. Note that although this tool tests GoogleBot specifically, all the major search engines support the same pattern matching rules. In Google Webmaster Tools:

Add the site, then click Tools > Analyze robots.txt. (Unlike most features in Google Webmaster Tools, you don't need to verify ownership of the site to use the robots.txt analysis tool). The tool displays the current robots.txt file.
Modify this file with the Disallow line for the tracking parameter. (If the site doesn't yet have a robots.txt file, you'll need to copy in both the User-agent and Disallow lines.)
In the Test URLs box, add a couple of the URLs you want to block. Also add a few URLs you do want indexed (such as the original version of the URL that you're adding tracking parameters to).
Click Check. The tool displays how Googlebot would interpret the robots.txt file and if each URL you are testing would be blocked or allowed.

At this point you may be thinking, wow, I can do all this and not have to write any new code? Unfortunately, there are even more downsides to this approach than the others:

This option will fix issue 1 (duplicate content), but not issues 2 (ranking) and 3 (reporting). This can be a good interim solution while you're implementing the more complete redirects solution, but it often isn't useful enough on its own.
Likely this will take a little bit of extra testing to ensure you get the patterns correct in your robots.txt file and don't inadvertently block content you want indexed.

Yahoo Site Explorer

Yahoo provides an online tool designed to solve this scenario. However, the solution only helps with Yahoo search traffic. To use the Yahoo fix, simply go to http://siteexplorer.search.yahoo.com and create an account for your web site in the Yahoo Site Explorer tool. Once you've verified ownership of your web site, you can use their Dynamic URL Rewriting tool to indicate which parameters in your URLs Yahoo should ignore.

Simply specify the name of the parameter you use for referral tracking (in our example it is 'from'), and set the action 'Remove from URLs'. Yahoo will then remove that parameter from all of your URLs while processing them and give you a handy little report about how many URLs where impacted.

Again, this is another solution that seems too easy to be true, but again, there are some significant limitations with this approach:

At the end of the day this is still a Yahoo-only solution. With approximately 20% market share, it is likely this will not meet all of your needs. However, if you do get some percentage of your traffic from Yahoo, there is no harm in doing this in the short term while you implement another method in the longer term.
The other problem with this solution is that it doesn't solve issue #3 (reporting), so you are still susceptible to reporting errors due to folks bookmarking and emailing your URLs with tracking codes.

Common Pitfalls

Cloaking & Conditional Redirects

Some web sites and SEO consultants attempt to solve this by a technique called cloaking or conditional redirects. Essentially what these methods do is check if the HTTP GET request is coming from a search engine and then show them something different than normal users see. This something different could be a simple 301 redirect back to the page without the tracking parameter similar to our first solution above. The difference is that our solution implemented this redirect for all requesters, and cloaking/ conditional redirects implement it only for search engines.

The big problem with this implementation method is that cloaking and conditional redirects are explicitly prohibited in the webmaster guidelines for Google, Yahoo and Live Search. If you use this method, you risk your pages being penalized or banned by the search engines. The primary reason they prohibit this behaviors is because they want to know exactly what content they are presenting searchers using their service. When a web site shows something different to a search engine robot than to a general user, a search engine can never be sure what the user will see when they go to the web site. So, even if you're thinking of implementing cloaking for what seems to be a valid, and not deceptive, reason, it's still a technique search engines strongly discourage.

This leads to the second major problem with this implementation method - it adds significant complication and can be difficult to monitor whether or not it's working - e.g. you have to test it pretending to be each of the 3 search engines robots. When things go wrong, it is likely that you're not going to see it right away, and by the time you do, your search engine traffic may already be impacted. Check out this example when Nike ran into an issue with cloaking.

Crazy Tracking Codes

Many studies on the web that show customers prefer short, understandable URLs over long complicated ones, and are more likely to click on them in the search results. In addition, users prefer descriptive keywords in URLs. Therefore, it might be worth your time to spend a few extra minutes thinking about the tracking codes you use to see if you can make them friendlier.

Good examples

?from=promo
?from=developer-video
?partner=a768sdf129

Bad examples

?i=A768SDF129,re23ADFA,style-23423,date-2008-02-01&page=2
?IAmSpyingOnYou=a768sdf129&YouAreASucker=re23adfd

Testing Your Implementation

So you've implemented your new favorite method, it compiles on your dev box, and now it's time to roll it into production, right? Maybe not! The initial goal of referrer URL-based tracking was to understand where your traffic was coming from so you can use that information to optimize your business. To ensure the data your collecting is actually useful, we highly recommend that you do some testing to ensure that all the common scenarios are working the way you expect, and you know where the holes are in your measurement capabilities. As with all metrics on the web, there will be holes in your data so you need to know what they are and account for them.

The first step in testing the implementation is to try it with a test parameter, walking the full scenario through start to finish.

Create several phoney promotional links that reflect the actual types of links you expect. This could be on your home page, product pages or with many additional query parameters that you might encounter.
Place these fake promotional links in a location that won't confuse your customers but are likely to get indexed by search engines. Using a social networking site or a blog might serve this well.
Click through those links as a customer and verify that you get to the correct page with a good user experience. Be sure to take these into account as well:
- Redirects operating properly (if you're using them) - use the Live HTTP Headers tool in FireFox to ensure the application is providing the correct headers (301 redirect and caching).
- Major browsers all work- if you're using cookies, you should test all the major browsers to ensure that they support cookies and that your scenario works the way you might expect. Don't forget to try common mobile browsers if your customers access your site this way.
Check out the search engine experience to ensure that you're not running into the duplicate content or ranking issues.
- Major Engines submit URL - if you place the test URLs in the right social network or place on your blog, they should get indexed within a week or so. If they don't you can also try the "submit a URL" from Google, Yahoo and Microsoft, though they are not guaranteed to work. Essentially you want to make sure the search engines have had the opportunity to see these URLs.
- Use 'site:' command to ensure tracking URLs are not indexed - here's an example query in Google, Yahoo, and Microsoft showing that our Jane and Robot example promotional URLs are not indexed.
Take a look at your metrics and ensure the numbers you're recording correlate to the testing you are doing. Some additional things to consider:
- Internal referrals - you might also want to add some logic to your application to filter out (or exclude) all referrals from the development team and your own employees. This is often done by checking requests against a list of known employee or company IP addresses and scrubbing those from your tracking data.
- Caching Issues - you might also want to try out several scenarios with multiple subsequent requests. You'll want to ensure that every request is going to your server and not getting cached somewhere along the way.

Related Resources

Related Internet Standards:
- W3C Standard For Cache-Control Header
- URI Specification (how URLs work)
Tools Used in Article:
- Google Webmaster Tools (Robots.txt Tester)
- Yahoo Site Explorer (Dynamic URL Rewriting)
- Live HTTP Headers (View HTTP Headers)
- Suggest URL Tool - Google, Yahoo, Microsoft
Canonical Link Tag Standard

Related articles
Occam’s Razor by Avinash Kaushik

Managing Robot's Access To Your Website

Vanessa Fox — Wed, 04 Jun 2008 12:18:00 -0700

Controlling what content is blocked from being found in search engines is crucial for many websites. Fortunately, the major search engines and other well-behaved robots observe the Robots Exclusion Protocol (REP), which has evolved organically since the early 1990's to provide a set of controls over what parts of a web site search engines robots can crawl and index.

Article Sections:

Capabilities of REP
Deciding What Should be Public vs. Private
Implementing the REP
Common Mistakes
Testing Your Implementation
Removing Content From Search Engine Indices
Additional Resources

Capabilities of the REP

The Robots Exclusion Protocol provides controls that can be applied at the site level (robots.txt), at the page level (META tag, or X-Robots-Tag), or at the HTML element level to control both the crawl of your site and the way it's listed in the search engine results pages (SERPs). Below is a table listing the common scenarios, directives, and which search engines support them.

Use Case	Robots.txt	META/ X-Robots-Tag	Other	Supported By
Allow access to your content	Allow	FOLLOW INDEX		Google Yahoo Microsoft
Disallow access to your content	Disallow	NOINDEX NOFOLLOW		Google Yahoo Microsoft
Disallow access to index images on the page		NOIMAGEINDEX		Google
Disallow the display of a cached version of your content in the SERP		NOARCHIVE		Google Yahoo Microsoft
Disallow the creation of a description for this content in the SERP		NOSNIPPET		Google Yahoo Microsoft
Disallow the translation of your content into other languages		NOTRANSLATE		Google
Do not follow or give weight to links within this content		NOFOLLOW	a href attribute: rel=NOFOLLOW	Google Yahoo Microsoft
Do not use the Open Directory Project (ODP) to create descriptions for your content in the SERP		NOODP		Google Yahoo Microsoft
Do not use the Yahoo Directory to create descriptions for your content in the SERP		NOYDIR		Yahoo
Do not index this specific element within an HTML page			class=robots-nocontent	Yahoo
Stop indexing this content after a specific date		UNAVAILABLE_AFTER		Google
Specify a sitemap file or a sitemap index file	Sitemap			Google Yahoo Microsoft
Specify how frequently a crawler may access your website	Crawl-Delay		Google WMT	Yahoo Microsoft
Authenticate the identity of the crawler			Reverse DNS Lookup	Google Yahoo Microsoft
Request removal of your content from the engine's index			Google WMT Yahoo SE Microsoft WMT	Google Yahoo Microsoft

Deciding What Should be Public vs. Private

One of the first steps in managing the robots is knowing what type of content should be public vs. private. Start with the assumption that by default, everything is public, then explicitly identify the items that are private.

If you want search engines to access all the content on your site, you don't need a robots.txt file at all. When a search engine tries to access the robots.txt file on your site and the server can't return one (ideally by returning a 404 HTTP status code), the search engine treats this the same as a robots.txt file that allows access to everything.

Every website and every business has a different set of needs, so there's no blanket rule for what to make private, but some common elements may apply.

Private data - You may have content on your site that you don't want to be searchable in search engines. For instance, you may have private user information (such as addresses) that you don't want surfaced. For this type of content, you may want to use a more secure approach that keeps all visitors from the pages (such as password protection). However, some types of content are fine for visitor access, but not search engine access. For instance, you may run a discussion forum that is open for public viewing, but you may not want individual posts to appear in search results for forum member names.
Non-content content - Some content, like images used for navigation, provides little value to searchers. It's not harmful to include these items in search engine indices, but since search engines allocate limited bandwidth to crawl each site and limited space to store content from each site, it may make sense to block these items to help direct the bots to the content on your site that you do want indexed.
Printer-friendly pages - if you have specific pages (URLs) that are formatted for printing you may want to block them out to avoid duplicate content issues. The drawback to allowing the printer-friendly page to be indexed is that it could potentially be listed in the search results instead of the default version of the page, which wouldn't provide an ideal user experience for a visitor coming to the site through search.
Affiliate links and advertising - If you include advertising on your site, you can keep search engine robots from following the links by redirecting them to a blocked page, then on to the destination page. (There are other methods for implementing advertising-based links as well.)
Landing pages - Your site may include multiple variations of entry pages used for advertising purposes. For instance, you may run AdWords campaigns that link to a particular version of a page based on the ad, or you may print different URLs for different print ad campaigns (either for tracking purposes or to provide a custom experience related to the ad). Since these pages are meant to be an extension of the ad, and are generally near duplicates of the default version of the page, you may want to block these landing pages from being indexed.
Experimental pages - As you try new ideas on your site (for instance, using A/B testing), you likely want to block all but the original page from being indexed during the experiment.

Implementing the REP

REP is flexible and can be implemented a number of ways. This flexibility lets you easily specify some policies for your entire site (or subdomain) and then enhance them more granularly at the page or link level as needed.

Site Level Implementation (Robots.txt)

Site wide directives are stored in a robots.txt file, which must be located in the root directory of each domain or sub-domain (e.g. http://janeandrobot.com/robots.txt.) Note that robots.txt files only apply to the hostname where they are placed, and do not apply to subdomains. So a robots.txt file located on http://microsoft.com/robots.txt will not apply to the MSDN subdomain http://msdn.microsoft.com. However, the robots.txt file does apply to all subfolders and pages within the specified hostname.

A robots.txt file is a UTF-8 encoded file that contains entries that consist of a user-agent line (that tells the search engine robot if the entry is directed at it) and one or more directives that specify content that the search engine robot is blocked from crawling or indexing. A simple robots.txt file is shown below.

User-agent: *

Disallow: /private

user-agent: - Specifies which robots the entry applies to.

Set this to * to specify that this entry applies to all search engine robots.
Set this to a specific robot name to provide instructions for just that robot. You can find a complete list of robot names at robotstxt.org.
If you direct an entry at a particular robot, then it obeys that entry instead of any entries defined for user-agent: * (rather than in addition to those entries).

The major search engines have multiple robots that crawl the web for different types of content (such as images or mobile). They generally begin all robots with the same name so that if you block the major robot, all robots for that search engine are blocked as well. However, if you want to block only the more specific robot, you can block it directly and still allow web crawl access.

Google - The primary search engine robot is Googlebot.
Yahoo! - The primary search engine robot is Slurp.
Live Search - The primary search engine robots is MSNbot.

Disallow: - Specifies what content is blocked

Must begin with a slash (/).
Blocks access to any URLs that begin with the characters after the /. For instance, Disallow: /images blocks access to /images/, /images/image1.jpg, and /images10.

You can specify other rules for search engine robots in addition to the standard instructions that block access to content as noted in other robot instructions.

Some things to note about robots.txt implementation:

The major search engines support pattern matching using the asterisk character (*) for wildcard match and the dollar sign ($) for end of sequence matching as described below in using pattern matching.
The robots.txt file is case sensitive, so Disallow: /images would block http://www.example.com/images but not http://www.example.com/Images.
If conflicts exist in the file, the robot obeys the longest (and therefore generally more specific) line.

Basic Samples

Block all robots - Useful when your site is in pre-launch development and isn't ready for search traffic.

# This keeps out all well-behaved robots.

# Disallow: * is not valid.

User-agent: *

Disallow: /

Keep out all bots by default - Blocks all pages except those specified. Not recommended as is difficult to maintain and diagnose.

# Stay out unless otherwise stated

User-agent: *

Disallow: /

Allow: /Public/

Allow: /articles/

Allow: /images/

Block specific content - The most common usage of robots.txt.

# Block access to the images folder

User-agent: *

Disallow: /images/

Allow specific content - Block a folder, but allow access to selected pages in that folder.

# Block everything in the images folder

# Except allow images/image1.jpg

User-agent: *

Disallow: /images/

Allow: /images/image1.jpg

Allow specific robot - Block a class of robots (for instance, Googlebot), but allow a specific bot in that class (for instance, Googlebot-Mobile).

# Block Googlebot access

# Allow Googlebot-Mobile access

User-agent: Googlebot

Disallow: /

User-agent: Googlebot-Mobile

Allow: /

Pattern Matching Examples

The major engines support two types of pattern matching.

* matches any sequence of characters
$ matches the end of URL.

Block access to URLs that contain a set of characters - Use the asterisk (*) to specify a wildcard.

# Block access to all URLs that include an ampersand

User-agent: *

Disallow: /*&

This directive would block search engines from crawling http://www.example.com/page1.asp?id=5&sessionid=xyz.

Block access to URLs that end with a set of characters - Use the dollar sign ($) to specify end of line.

# Block access to all URLs that end in .cgi

User-agent: *

Disallow: /*.cgi$

This directive would block search engines from crawling http://www.example.com/script1.cgi but not from crawling http://www.example.com/script1.cgi?value=1.

Selectively allow access to a URL that matches a blocked pattern - Use the Allow directive in conjunction with pattern matching for more complex implementations.

# Block access to URLs that contain ?

# Allow access to URLs that end in ?

User-agent: *

Disallow: /*?

Allow: /*?$

That directive blocks all URLs that contain ? except those that end in ?. In this example, the default version of the page will be indexable:

http://www.example.com/productlisting.aspx?

Variations of the page will be blocked:

http://www.example.com/productlisting.aspx?nav=price
http://www.example.com/productlisting.aspx?sort=alpha

Other robot instructions

Specify a Sitemap or Sitemap index file - If you'd like to provide search engines with a comprehensive list of your best URLs, you can provide one or more Sitemap autodiscovery directives. Note, user-agent does not apply to this directive so you cannot use this to specify a Sitemap to some but not all search engines.

# Please take my sitemap and index everything!

Sitemap: http://janeandrobot.com/sitemap.axd

Reduce the crawling load - This only works with Microsoft and Yahoo. For Google you'll need to specify a slower crawling speed through their Webmaster Tools. Be careful when implementing this because if you slow down the crawl too much, robots won't be able to get to all of your site and you may lose pages from the index.

# MSNBot, please wait 5 seconds in between visits

User-agent: msnbot

Crawl-delay: 5

# Yahoo's Slurp, please wait 12 seconds in between visits

User-agent: slurp

Crawl-delay: 12

Page Level Implementation (META Tags)

The REP page-level directives allow you to refine the site wide policies on a page-by-page basis

Placing a meta tag on the page - Place the meta tag in the head tag. Each directive should be comma delimited inside the tag. E.g. <meta name="ROBOTS" content="Directive1, Directive 2>.

<html>

<head>

<title>Your title here</title>

<meta name="ROBOTS" content="NOINDEX">

</head>

<body>Your page here</body>

</html>

Targeting a specific search engine - Within the meta tag you can specify which search engine you would like to target, or you can target them all.

<!-- Applies to All Robots -->

<meta name="ROBOTS" content="NOINDEX">

<!-- ONLY GoogleBot -->

<meta name="Googlebot" content="NOINDEX">

<!-- ONLY Slurp (Yahoo) -->

<meta name="Slurp" content="NOINDEX">

<!-- ONLY MSNBot (Microsoft) -->

<meta name="MSNBot" content="NOINDEX">

Control how your listings - there are a set of options you can use to determine how your site will show up on the SERP. You can exert some control over how the description is created, and remove the "Cached page" link.

<!-- Do not show a description for this page -->

<meta name="ROBOTS" content="NOSNIPPET">

<!-- Do not use http://dmoz.org to create a description -->

<meta name="ROBOTS" content="NOODP">

<!-- Do not present a cached version of the document in a search result -->

<meta name="ROBOTS" content="NOARCHIVE">

Using other directives - Other meta robots directives are shown below.

<!-- Do not trust links on this page, could be user generated content (UCG) -->

<meta name="ROBOTS" content="NOFOLLOW">

<!-- Do not index this page -->

<meta name="ROBOTS" content="NOINDEX">

<!-- Do not index any images on this page (will still index the if they are linked

     elsewhere) Better to use Robots.txt if you really want them safe.

     This is a Google Only tag. -->

<meta name="GOOGLEBOT" content="NOIMAGEINDEX">

<!-- Do not translate this page into other languages-->

<meta name="ROBOTS" content="NOTRANSLATE">

<!-- NOT RECOMMENDED, there really isn't much point in using these -->

<meta name="ROBOTS" content="FOLLOW">

<meta name="ROBOTS" content="UNAVAILABLE_AFTER">

HTTP Header Implementation (X-ROBOTS-Tag)

Allows developers to specify page-level REP directives for non text/html content types like PDF, DOC, PPT, or dynamically generated images.

Using the X-Robots-Tag - to use the X-Robots-Tag, simply add it to your header as shown below. To specify multiple directives you can either comma delimit them, or add them as separate header items.

HTTP/1.x 200 OK

Cache-Control: private

Content-Length: 2199552

Content-Type: application/octet-stream

Server: Microsoft-IIS/7.0

content-disposition: inline; filename=01 - The truth about SEO.ppt

X-Robots-Tag: noindex, nosnippet

X-Powered-By: ASP.NET

Date: Sun, 01 Jun 2008 19:25:47 GMT

The X-Robots-Tag directive supports most of the same directives as the meta tag. The only limitation with this method over the meta tag implementation is that there is no way to target a specific robot - though that probably isn't a big deal for most use cases.

X-Robots-Tag: noindex
X-Robots-Tag: nosnippet
X-Robots-Tag: notranslate
X-Robots-Tag: noarchive
X-Robots-Tag: unavailable_after: 7 Jul 2007 16:30:00 GMT

Content Level Implementation

You can further refine your site level and page level directives within several content tags.

Each anchor tag (link) can be modified to tell search engines that you do not trust where this URL is pointing to. This is typically used for links within user generated content (UCG) like wikis, blog comments, reviews and other community sites.

<a href="#" rel="NOFOLLOW">My Hyperlink</a>

Also, in Yahoo Search you can specify which <div> elements on a page you would not like indexed using the class=robots-nocontent attribute. However, we don't highly recommend using this tag because it is not supported in any other engine, making it not super-useful.

<div class="robots-nocontent">

No content for you! (or at least Yahoo!)

</div>

Common Mistakes

While implementing the REP is generally straight-forward, there are a few common mistakes.

GoogleBot follows the most specific directive, ignoring all others. In the robots.txt file, if you specify a section for all user-agents (user-agent: *) and also declare a section for Googlebot (user-agent: Googlebot), Google will disregard all sections in the robots.txt file except the Googlebot section. This could potentially leave you exposing much more content to Google that you might have thought.

# This keeps out all well-behaved robots

User-agent: *

Disallow: /

# This looks like it is giving Google access to only this directory, but since it is a

# GoogleBot specific section, Google will disregard the previous section

# and access the whole site.

User-agent: Googlebot

Allow: /Content_For_Google/

NOFOLLOW will most likely not prevent indexing - if you use NOFOLLOW at either the page or the link level, it is still possible for the links from the page to be indexed because the search engine may have found a reference to them from another source. Another note, using rel="NOFOLLOW" within your anchor text is still perceived as a recommendation by the search engines, not a command.

To ensure that content is not indexed, either use the Disallow directive at the site level, or use NOINDEX at the page level.
Directives that are not recommended - the directives in the REP are all about exceptions, by default the robots assume they can crawl your whole site. Therefore, you do not need to explicitly use the FOLLOW and INDEX directives as they will not be taken into account by the search engines. It sounds silly but I've seen a few sites that have implemented these on every page and every link.

Another directive that is not recommended is the NOCACHE directive. This was created by Microsoft, and is synonymous with NOARCHIVE. While they will most likely always continue to support the directive, it is better to use NOARCHIVE so it will work on all the search engines.
Be cognizant of case - when referencing files and URLs in the Robots.txt file, use a defensive approace to URL case, as the major engines do not handle it the same way. (e.g. /Files does not always equal /files).

Testing Your Implementation

As you're implementing your REP design, you should test it both before you deploy it and after. The easiest way to test this is to use the robots validator in either Google or Microsoft's Webmaster Tools. These tools are generally good enough test beds for most folks, however advanced developers (or paranoid ones with critical business requirements) will want to definitively know what the robots are doing, not simply rely on what the robots say they are doing. These folks will want to look at their tools as well look at their server logs.

In addition to using validation tools, reporting tools from the search engines on what they couldn't acces, and looking at logs data to see what the search engine robots are crawling, you should check the search engine results to see if any pages you are intending to block are being indexed. If they are, use the methods described in this section to ensure you are blocking them correctly and use the search engine tools to request that the pages be removed.

When Blocked Content Appears to be Indexed - If search engines are blocked from crawling pages, they may still index the URL if the robot finds a link to that URL on a page that isn't blocked. The listing may display the URL only, such as shown below.

Or, it may include a title and in some instances, a description. This makes it appear as though the search engine robot is disregarding the directive that blocks access to the page, but the search engine is in fact obeying the directive not to crawl the page and is using anchor text from the link to that page and descriptive details from either the page that contains the link or a source such as the Open Directory Project.

For more details, see:

The Easy Way

Search Engine Tools For Validation - Both Google and Microsoft provide some tools as part of their Webmaster Centers to help you verify if you've configured your REP the way you expect. Let's start with Google's tools:

The first thing you should check are the list of URLs that Google has seen from your website and not indexed due to the REP. Note you can also download the list and filter, sort, and have-your-way-with-it in Excel.

The next step is to use their interactive robots.txt tool to analyze your rules and test specific URLs for blockage. When you pull up the tool they already should have it pre-populated with the robots.txt file they have on file from the last time they crawled. You can input a list of URLs you'd like to check below, select the user-agent you'd like to check against and the tool will tell you if they are blocked or not. You can also use the tool to test changes to your robots.txt file to see how Google would interpret things.

Microsoft has a similar tool in their Webmaster Center that will validate a robots.txt file against the standard that MSNBot supports. To use the tool, simply log in copy & paste your robots.txt file into the top field and select Validate. A list of all detectable issues are displayed in the bottom box.

The Hard Way

More Accurate Views of Robot Access Through Your Logs - If you have a specific business need to ensure that the robots are following your rules, (or you're just paranoid) then you should not simply rely on the tools they provide to test compliance. You're going to need to go straight to the horse's mouth and analyze your web server logs to see exactly what they are doing. There is no one easy tool for doing this, you'll likely have to use an existing tool like one of these (Microsoft HTTP Log Parser) or write your own. It isn't difficult, it will simply take some time to implement. A useful reference for this is a list of all the robot user agents, and more complete list of bots from Google, and Microsoft.

Verifying Robot Identity - Another thing you'll likely want to consider in this endeavor is to validate that the robots are who they actually say they are. Google, Yahoo and Microsoft all support Reverse DNS authentication of their robots. The process is pretty simple and described here by Google, Yahoo and Microsoft, essentially you simply find out what range their robot's DNS is hosted in, and use that in your tool. This way, if the address changes (which it will), you don't need to update your code.

Should you find any issues, where one of the robots are not minding the REP, or are misbehaving in some other way, you can always communicate directly with each engine through one of their forums:

Removing Content From Search Engine Indices

If you find that you haven't implemented the techniques described here correctly and private content from your site is indexed, each of the major search engines has methods available for requesting that it be removed. For more information, see:

Additional Resources:

Google
Yahoo!
- Wild card support
- X-Robots tag directive support
Microsoft Live Search
- Search robots in disguise
Other resources

Effectively Using Images

Vanessa Fox — Wed, 14 May 2008 07:30:00 -0700

A picture is worth a thousand words. Unfortunately, when it comes to major search engines (which are still primarily text-based), a picture is worth a lot of blank space. Does this mean you shouldn't use images on your site if you want to rank in search? Not at all. Just keep some simple things in mind when adding those images to your pages. As a bonus, these tips help not only with seach engine robots, but with Jane as well! You want your site to be accessible in screen readers, to those who have images turned off in their browsers, and to those who have slow connections or are on mobile browsers and may have trouble loading images.

By providing search engine robots with textual information about the images on your site, your site can benefit not only from better placement in web search results, but in image search results also. Image Seach can provide substantial search traffic, so don't overlook this as an acquisition channel.

Below are recommendations for using images effectively for both Jane and search engine robots.

Don't put text in images

Put text in straight HTML whenever possible. Sometimes web designers like to put text in images because they can use a wider variety of fonts and can manipulate the design more freely. Much of this styling can be done with CSS and in cases where it can't, the extra design a graphical version of the text provides may not really add visitor value. In fact, it may detract from usability because it may be difficult to read. It also may hurt viral efforts since it can't be copied and pasted. If I want to send an email to all of my friends suggesting we all go to a hot new restaurant, I may want to copy and paste a few menu items from the restaurant's web site to send to them. If the menu is in an image, I can't do that.

Use the ALT attribute

The most well-known method for making images accessible is effective use the ALT attribute in the IMG element. And yet it's very common to find empty ALT tags all over the web.

<img src="/images/lavender-plant.jpg" alt="Picture of a lavender plant">

Make the text in the ALT tag descriptive. It should describe the image concisely. Think of someone browsing your site with a screen reader. How will they want the image presented?
Don't stuff the ALT tag with keywords. A long ALT attribute, full of keywords your want to rank for looks spammy to both your visitors and to search engines and may make both devalue your content. How can you tell if your ALT text is spammy or simply descriptive? It's a judgment call, but if you can't tell, get some objective opinions. ALT="buy cheap viagra now cheap viagra online get viagra here" is probably going to be pretty obviously spammy to anyone you ask.
Make the ALT text relevant to the image. Use the ALT text to describe the image, not as a place to add descriptive text abou the page that isn't directly relevant to the image. For instance, if the image is of a car, your ALT text should be something like "blue mini cooper" not "cheapcars.com has cheap cars available in every make and model including mini coopers, volkswagons, and Ferraris like Magnum PI used to drive".

What about the TITLE attribute? It likely doesn't provide direct search engine value, although it may be useful for your visitors.

Make image filenames descriptive

If possible, describe the image in name of the image file. For instance, lavender-plant.jpg is better than image123.jpg. If you are importing a lot of images, for instance, for a product database, it may be problematic to manually name each file. In this case, find programmatic ways to rename the images using text from how the images are tagged or categorized. If your filename includes multiple words, use hyphens to separate them (search engines tend to see a hyphen as a separator and an underscore as a joiner (so lavendar_plant would be seen as one word and lavender-plant would be seen as two).

Use image captions

Provide a caption below or above the image that describes what it's about and gives context for how it relates to the rest of the page.

Provide textual clues around the image

Try to include text around the image that relates to what the image is about. Text on the page helps seach engines know what the page itself is about, which helps the page rank for relevant queries, but text near images can help those images rank in image search results as well.

Be cautious about using images for navigational links

If you use images in menus and other navigation, make sure that you use ALT text that replicates how the image represents that menu option. But also test the implementation by turning off images in your browser and making sure the links still work. Some implementations incorrectly require images to be enabled, causing search engine robots to be unable to follow those links.

Another potential usability issue with images and navigation is that if you use a textual link combined with a background image, the text may disappear if the image doesn't load. (This issue can happen with this type of design in places other than menus, but that scenario is where it can be commonly seen.)

Navigational Link With Images Enabled	Navigational Link with Images Disabled

Be cautious about using images for headings and logos

Many web sites use an image for the header of the page or for the company logo. This implementation works well, but be sure that you replicate the company name, heading text, or other words from that image in the ALT text.

If you have an image as the page's H1 tag, keep in mind that the H1 is one of the most important clues for a search engine to determine what the page is about, so consider using text instead of an image or at least using descriptive ALT text. In the example below, the code is using CSS to display an image of the company logo as the H1 tag. A better implementation would be to display the image in the header of the page, and use the H1 tag to provide visitors and search engines description information about the page.

								<h1 class="home-logo">Company Name</h1>

The CSS for this implementation positions the text at -999em. This is not recommended both because it means that when a visitor loads the page with images turned off, the text can't be seen (and so the heading space is simply blank) and because search engines may find the practice deceptive (the text is hidden).

								.home-logo {	background:transparent url(/images/logo1.gif)	no-repeat scroll center top;	height:63px;	margin-top:35px;	text-indent:-99999em;	}

If your header includes an image of your company logo, avoid commonly used ALT text such as "home" or "logo". Instead, succinctly describe your company or home page (using either the company name or a brief description of the site). (Also, avoid naming your company logo something like logo.jpg.)
If your site includes a header that consists entirely of a large image, test the layout of the page with images turned off. In some cases, the result can be a large area of white space that pushes all content below the fold. In the example below, all company information and details about the site are lost without the header image.

Block Non-Content Images

If you use a lot of non-content images (for instance, arrows, bullets, and boxes), you likely don't want those indexed. Since search engine robots spend limited time crawling each site, it may make sense to block them from crawling these types of images so they can spend all the available resources on the pages and images you do want indexed. As a bonus, if you want to provide an image search on your site (for instance, using the Live Search API), if only content images are indexed, then the image results will be more useful for your visitors.

A good way to block non-content images is to place them in a separate folder from your content images and then block that folder using robots.txt. For instance, if you place these images in a folder called no_index_images, your robots.txt file would contain:

User-agent: *

Disallow: /no_index_images/

Opt-in to Google's Image Labeler

In Google's Webmaster Tools, you can opt your images into their Image Labeler. This enables others on the web to tag your images, which in turn provides Google with additional details about what the images are about and can help them rank for a wider variety of queries in Google Image Search.

Images can be search engine and user friendly

With a little planning and good structure, you can effectively use images on your site in ways that benefit both Jane and robots. And by optimizing images in the ways described in this article, you may also be able to tap into an additional acquisition channel - image search.

Domain Canonicalization

Nathan Buggia — Fri, 02 May 2008 11:52:00 -0700

Pop quiz: what's the difference between the following URLs:

http://website.com
http://www.website.com
http://website.com/default.php
http://www.website.com/default.php

Give up? If you're a user, then chances you expect all of those URLs will lead you to the same page. Robots, however, are not as good at determining if pages are the same, so they often store each separately. A big part of how search engines rank pages is based on how many external links those pages have. If other sites on the web link to the different versions of your home page, then search engines may calculate the value of each URL separately, based on the number of links to each version. This can effectively diminish the potential rank your page would have if it were found (and linked to) by only one URL.

The practice of consolidating all versions of a page under one URL is referred to as "canonicalization" (because you collapse all versions under the "canonical" or true version). The four examples listed above are the most common, but there are potentially many, many URLs that lead you to the same page. By adhering to several best practices, you should be able to address 90% of common site-wide canonicalization issues on your site and consequently increase how your site ranks.

Recommendation

The solution is to be explicit about the canonical form of your URLs. Following are four best practices to achieve this, with specific code and configuration examples.

Select WWW or Non-WWW, then redirect the other option to your preferred version.
The hard part is choosing if you want your site to be "www.website.com" or simply "website.com". There is no right answer for every company so you'll have to figure this out on your own (but, removing the "www." saves your customers 4 keystrokes, which really add up on a mobile device, and it makes your brand the first thing your customers see).

Once you've selected, you then need to find a way to trap all requests to your application, check which form is being used, and if it is not the correct form, initiate a 301 Redirect to the correct form. For example, if the user types in wikipedia.org, they will automatically get redirected to www.wikipedia.org.
Remove the default filename from the end of your URLs.
All web servers allow you to select one or more default filenames to serve when the browser requests a directory. For example, this website is run on IIS, so when the user requests "http://janeandrobot.com" we really serve "http://janeandrobot.com/default.aspx".

In the same code you use to enforce www vs. non-www, you should also check and see if the default filename is at the end of the URL and then trim it off. So, "http://janeandrobot.com/default.aspx" would be converted to "http://janeandrobot.com".
Link internally to the canonical form of your URL.
Make sure you always link to the proper canonical form of your URLs from within your site. This practice helps encourage external sites to link to the site using the correct version as well (since those linking to you often cut and paste from your pages or RSS feed.) Note there is a degree of diminishing returns here, so you don't need to spend the whole weekend hunting down every last URL. Just make sure to review your site's primary navigation, top landing pages and blog.
Use Google Webmaster Tools to tell Google the correct form.
Implementing these best practices on your site are ideal, since they address the problem for all search engines and give your customers a consistent, properly branded navigation experience. But what can you do if you reviewed steps 1-3 and found that it would take six months to implement on your production site? There is something that you can do today: using Google's Webmaster Tools, you can navigate to the "Tools" section and select "Set preferred domain." Here you can specify if you'd like Google to use "www.website.com" or "website.com" in their index and search results, as well as consolidate links to both versions. Note that while this will provide you short-term benefit from Google, it does not help you in Yahoo! or Live Search.

Checking Your Website

To check your website to see if you're handling domain canonicalization correctly, you can use the Live HTTP Headers add-on for Firefox.

Open the Live HTTP Headers tool, then try all the variations of the URL at several different levels to ensure they all redirect back to the appropriate canonical form. As you're checking each variation, look at the HTTP headers using the Firefox plug-in to ensure they are all 301 redirects (and not, for instance, 302 redirects).

Here's an example test case:

Canonical URL Form	Test Case	Test Result
http://janeandrobot.com	janeandrobot.com	Success
	janeandrobot.com/default.aspx	Success
	www.janeandrobot.com	Success
	www.janeandrobot.com/default.aspx	Success
http://janeandrobot.com/about.aspx	janeandrobot.com/about.aspx	Success
	www.janeandrobot.com/about.aspx	Success
http://janeandrobot.com/folder	janeandrobot.com/folder	Success
	janeandrobot.com/folder/default.aspx	Success
	www.janeandrobot.com/folder	Success
	www.janeandrobot.com/folder/default.aspx	Success
http://janeandrobot.com/folder/test.aspx	janeandrobot.com/folder/test.aspx	Success
	www.janeandrobot.com/folder/test.aspx	Success

Examples

Canonicalization issues are very common and being an Microsoft employee, I don't have to go far to find an example. Check out the website for Microsoft's annual Mix conference for web developers.

I was able to generate the table below by plugging the common URL variations into Yahoo's Site Explorer to find a list of links to each variation.

URL Variation	Number of Links from within website	Number of Links from outside websites
http://visitmix.com	17,663	59,498
http://www.visitmix.com	9,074	22,179
http://visitmix.com/default.aspx	0	22
http://www.visitmix.com/default.aspx	0	12

Looking through these numbers yields some interesting insights:

Not doing "www" vs "non-www" is definitely hurting their ranking - you can tell because they have a similar number of inlinks for each version. Ranking is done on a logarithmic scale, so every additional link is more valuable than the one before. If they redirected all versions to one canonical form, search engines would see their home page has having 81,711 external links, would would be a substantial boost.
They are not good about using the same version of the URL within their site. If you're not cognizant of this on your site, others won't be either. It looks like they use visitmix.com about 75% of the time internally, and www.visitmix.com the other 25%.

Additional Resources

Matt Cutt's Article on Canonicalization
Additional Canonicalization Scenarios from Ian Ring - a few other great scnearios like capitalization and other default values.
Yahoo Site Explorer - see how many inlinks you have for each URL variation
Live HTTP Headers - check your redirects to make sure you're implementing 301 redirects, not 302s.

Search-Friendly Design Patterns For Web Developers

Vanessa Fox — Thu, 24 Apr 2008 03:17:00 -0700

Search engine optimization (SEO) isn't a marketing gimmick, spammy scheme, or destroyer of web usability. It's a fundamental building block of effective infrastructure design that ensures web applications can thrive in an online environment in which potential customers turn to search first. It also can dramatically improve site usability and visitor happiness.

Watch this space for design patterns, code snippets, case studies, and implementation techniques for elegant code that both ensures potential customers can find your site online as well as provides them a compelling experience once they arrive.

Learn more about us and our philosophy.

View our Web 2.0 Expo slides and resources.