But until now, I’ve never felt like mobile has really changed the basic way I have experienced the world. I go to trade shows and listen to all the ideas for the latest mobile services or how mobile concepts should change my daily life, but I have never felt that I had crossed a Rubicon with the real-time nature of mobile in the same way I did when I got my first laptop or sent my first Tweet.

That changed yesterday. I had the pleasure to take my family to a performance of The Smuin Ballet at the Sunset Center in Carmel. I am an admitted ballet snob, and Smuin is a wonderful company with talented, disciplined dancers and creative choreography. So whenever they are in Carmel we go to see them. The second act was a performance of Smuin’s Medea, which was first performed in 1997. It is a dramatic ballet that retells the story of Medea, the wife of Jason, the intrepid explorer of Jason and the Argonauts.

The myth of Jason and Medea is a dark and haunting tale of revenge and self-destruction. But in the dark as the curtain rose on Act 2, despite being a student of mythology, I couldn’t for the life of me remember even the outlines of the tale to tell my wife and eight year old daughter, neither of whom knew the first thing about this particular myth.

So what did I do? I pulled out my handy iPhone and while shielding it so as not to disturb others, I did a web search on Medea and pulled up the Wikipedia entry.

Despite the dark and small print, I was able to glean from Wikipedia the details of the story. Jason met the sorceress Medea, daughter of King Aeetes of Colchis. Medea falls in love with Jason, and he convinces her to help him to acquire the Golden Fleece in return for a pledge of marriage.  After acquiring the Fleece, Jason and Medea flee to Corinth and have children. In Corinth, King Creon offers his daughter Glauce to Jason in marriage, and Jason feels he cannot pass up the opportunity to marry a royal princess. Despite explanations and promises of support from Jason, Medea feels betrayed. She gives Glauce a wedding gown covered in poison that kills the bride. Then to ensure that nothing of Jason’s will outlive him, she kills their two sons.

I quickly related the story to my wife and daughter (who was especially engaged by the dark and intense drama onstage).  I then sat back and enjoyed the performance, knowing that they now could now understand what they were seeing and, as a result, enjoy it with more insight.

Not a big deal, you might say. I say otherwise. Prior to this, I had used my mobile capabilities to get directions or find a resource nearby. I was using the mobile device as a tool for location-based information. It was a parallel use to a GPS, which while practical, was not a fundamental change in my experience over a map. It was easier than a map and had better information, but the experience was just a replacement of one form of information (paper) for another (digital).

In this case, for the first time I used my iPhone to plug into the wisdom of the commons, into the global village, to enhance and extend the quality or content of an experience.   In other words, the phone and the information it provided in real-time became an integral part of the experience, although it was unintentional (from the perspective of the choreographer or the dancers) and because it was unintentional, it was distracting. But this little thing, this one act, was a fundamental change in how I interacted with the world. This was not a substitution of paper-based data with a digital version. Instead it delivered the true promise of the mobile web. It allowed me to access a completely new set of information that was then added to a real-time, real world event to enhance the experience. The equation is real-time event + Internet information = a real-time multimedia experience.

Admittedly, the experience was not perfect because it wasn’t intentionally developed by the show’s producers. But what if Smuin developed a mobile video app that during the Intermission allowed audience members with iPhones to see the full story of Medea in an entertaining way that also tied the story to how the ballet attempted to recreate that story in dance. That would be a more intentional, and less invasive, way to provide the same integrated (but completely new) experience.

So yesterday for the first time, I experienced the true power of the mobile web.  I can now say “Eich Bin Ein Mobile Netizen.”

Well, let’s see if I can change that with a quick post today.  To help those who couldn’t be at SMX West these last three days, I will list some tidbits here that I found useful.   These will go all over the place, so bear with me. This is Part 1 – I will cover more of the highlights in subsequent posts.

Also, before I get started, let me say a big “thanks” to the folks at SearchEngineLand for putting on another great and informative event. It was fun and you never fail to educate me on things I hadn’t discovered for myself. I’ll see you, of course, at SMX Advanced in June.

Insights for Mobile Search

1. Isolate mobile into its own unique campaign.
   • KPIs are lower than in traditional paid search – so isolate it to protect your other programs.
   • Isolating helps you optimize to raise KPIs of mobile specifically.
   • Adwords lets you segment your campaigns by carrier and site you are on. Cool!
2. In mobile search that are only 5 results above the fold and you must be in position 1 or 2 or not at all – there is a HUGE falloff after those 2 positions.  Literally near zero clicks.
3. Mobile query lengths are shorter – no big surprise there.  Search Term length example:  in a study of 414 broad match queries on mobile, the avg word length was 2.8 and longest phrase was 8 words (used 3 times).  So it is very important to run broad match on mobile campaigns.
4. Length of query versus Click Through Rate is pretty similar between mobile and desktop search.
5. Most popular categories for mobile right now are sports, celebrity, news, wall papers, videos, and ringtones.
6. Other Best Practices for Mobile
   • Reinforce mobile friendliness in ads – “mobile optimized, “4 UR phone”, “Mobile ready” – indicates to viewer that site will work on their phone with best performance.
   • Have a URL that indicates mobile friendliness – it’s another signal to the consumer that you’ve made the experience usable on mobile devices.
   • Build landing pages specifically for mobile. Test landing pages for mobile separately.
   • Simplify the experience for mobile – you have to really get the content down to an essence – we saw an example of Home Depot versus Best Buy. Home Depot’s cluttered page really made the experience unusable.

New Tools to Explore

Of course I’m going to have a section on this, toolhound that I am.  Just a list for you to explore:

• usertesting.com. Quick testing of your user experience for landing pages.  Cool idea: run this on your competitors sites as well – $29/user.  I’ve used this, btw – it has drawbacks, but for testing landing pages it is very appropriate.
• crossbrowsertesting.com. See how your page looks in various browsers easily.
• attentionwizard.com. A free tool that shows where people are looking on your page.
• clicktale.com.  Records sessions and gives feedback/analytics on experience.  Free for one domain and 2-page playback.
• crazyegg.com. Simple and affordable heat mapping tools that allow you to visually understand user behavior.
• Lynx or seobrowser.com. A dedicated browser that allows you to see your pages the way the search engine crawlers see it
• charles. An HTTP proxy / HTTP monitor / Reverse Proxy that enables a developer to view all of the HTTP and SSL / HTTPS traffic between their machine and the Internet.
• Wave toolbar. Provides a mechanism for running WAVE reports directly within Firefox for debugging site issues.
• gsite crawler. Another free sitemap generator for Google, Yahoo and Bing.

That’s it for today. Back to the day job.

One of my more unusual pet peeves in SEO is the number of general purpose SEO toolbars available for use. Actually not so much the number, but the fact that each one claims to be the best of the best, each one has its champions in the community, and each reports slightly different data for certain metrics. Which one is best for me and in what situations? Which provides the most accurate data?

I have been asked so many times by customers “What is the best general purpose toolbar for Firefox?” that I finally decided to do a detailed comparison, if nothing else to satisfy my curiosity about:

• how similar the features/functions of these toolbars are.
• how accurate they were (e.g. did they all get the same numbers for similar analyses).
• which one was best in which situation.

I could spend days writing reviews of these toolbars, but very few would read these and I doubt the added information would prove all that useful versus downloading and trying the tools. SEOs are, by definition, very much experimenters. They prefer to test rather than read or guess. On the other hand, a high level visual summary, which can provide a sense of the tools coverage as well as its focus/strengths, would probably help those evaluating tools so they can know which features they should explore in which tool.

Ergo the table below, which shows a feature-by-feature comparison of six popular SEO toolbars for Firefox – FoxySEO Tool, SEMOMoz, SEOBook, SEOpen, SEOQuake, and SEO for Firefox. Hopefully the categories and line items are self-explanatory. If not and I get enough comments, then I will add an addendum explaining the line items – but that would be painful and probably not add a lot of value for most of the audience.

No doubt, readers are going to be upset because I didn’t get to their favorite toolbar, and for that my humble apologies. In the process of this research, I found several more general purpose SEO toolbars, like ToolbarBrowser, which need review. I will have to cover those on in a separate post and I will update the downloadable pdf comparing all of the toolbars as I extend the research.

What I found interesting:

1. There was no single metric/line item that all six toolbars analyzed except the number of backlinks to site in Yahoo! SiteExplorer.
2. Five of the six tools included metrics for PageRank of the current page, pages in Google Cache, number of backlinks to page in SiteExplorer, DMOZ Entries, Keyword Density, and Meta Tag Analysis. These were the most widely shared metrics.
3. Each tool has different strengths and weaknesses. For example:
   • SEOBook’s toolbar clearly has a much deeper set of tools for keyword analysis compared to the others.
   • FoxySEO tool has a broader coverage of metrics in the social media, site performance, and indexing domains.
   • SEOQuake and several others provides the set of analyses for every line item in the SERPS, which is very handy for competitive analysis.
   • SEOMoz, on the other hand, has fewer tools but they cover areas not included in other toolbars, as well as including metrics unique to SEOMoz – e.g. MOZRank, MozTrust. The toolbar also links to the SEOMoz Pro tools on the SEOMoz website, where you can run other, deeper analyses. Clearly, its purpose is not to perform all the analysis “on page” but rather to give a high-level view from which further data can be gleaned by using the more detailed SEOMoz tools. However, those tools (and the toolbar) are only available to paid SEOMoz members.
4. As far as accuracy goes, when checked most of the tools reported consistent data on everything but link-related metrics, where the data tended to vary widely. Yahoo SiteExplorer links probably had the least variation, but when you looked at bookmarks on social media sites or directory entries in DMOZ or Yahoo, you can see results that differ by a factor of 10 in some cases. What is even more interesting, when I manually went to some of these sites and typed in the same query, I actually got a set of results that differed from the data I got in SEOQuake and FoxySEO Tool.

So which toolbar is right for you depends on:

• What metrics you feel are important to you.
• What services you subscribe to.
• How cluttered you like your Firefox screen.
• How much performance degradation you are willing to tolerate. In the case of tools which provide metrics for every line in the SERPs, the wait time can be substantial.

As you can see from the picture, as a self-confessed tool addict I have all of them running – which explains why I never use Firefox for anything but SEO work. Advantage to Chrome for regular browsing and social media work, at least until all the SEO plugins port over to Chrome.

Today’s post is about whether the long-tail is useful for companies with small keyword universes. This became of interest to me when a colleague of mine, Steven Ebin, suggested that this strategy was useful even for small search markets, which had not been my experience.

I define a small keyword universe as one that is less than 1,000 core keywords. The short-tail is defined as keywords that represent 60% of traffic (in my experience usually about 10-15 keywords for small universes, although that can vary substantially). The mid-tail is defined as keywords that represent the next 25% and the long-tail is defined as keywords that represent the balance. I find that many small customers have core keyword universes of this size and distribution, although size of the keyword universe really ties to the specific market, not the size of customer. Be that as it may, my experience is that there is a correlation between size of company and size of keyword market.

Let’s assume that the overall number of monthly clicks for a keyword market, including the long-tail search terms, is 2,200,000 visits.

Next, we know that not everyone clicks – so we need to ignore that traffic. Some aged results reported for AOL in the Webmaster World forums suggest that 46% of searchers don’t click through.

We then have to make some assumptions about where we will rank in each of the positional categories. For purposes of this analysis, I’ll assume that the above average SEO can get an average position of 5 for the short-tail terms, a position of 3 on average of the mid-tail terms, and an average position of 2 for the long-tail terms.

We also have to estimate the clickthrough rates for results in each of these positions in the SERPs. This data varies widely. One study from Cornell University suggests that 56% of searchers who click click on the first result. Some calculations by Jay Geiger suggest that 42.3% of searchers who click click on the first result, and some reported statistics for AOL from the prior-mentioned source show the same number as 23%.

Clickthrough Rates Based on Position in Search Results

We also have to make some assumptions about conversion rates. Estimating conversion rates is hard because it can vary so much by industry, but let’s assume we have a 0.5% conversion rate for our short-tail terms, 1% for our mid-tail terms, and 3% for our long-tail terms. This gives us a weighted-average conversion rate (based on the percentages in each part of the tail) of 1.0%, which is not unrealistic for organic traffic and may even be low. If you play with these numbers, you will see that a reasonable variation in conversion rate assumption on the long-tail doesn’t change the results of the analysis.

Why increase the conversion rate so much for the long tail? Long-tail searches are more “perfected.” The fact that people type in more specific keyword strings indicates that they are further down the decision-making cycle, and thus the conversion rates are significantly higher. The “spread” that I have assumed for the model is actually conservative. We have often seen conversion rate differentials between the short- and long-tail that are even greater.

When we run out the numbers, using all three sets of clickthrough rates we get the following results:

Results of Analysis on Long-Tail Keywords

This analysis shows that the long-tail can potentially double the business for a small company.

We can also look at this same analysis using the length of the keyword string. How does keyword string length relate to location in the tail? Are long keyword strings (3+ words) equivalent to the “long tail”? The answer is “no” – in fact the ratios of searches for length of tail versus length of keyword string are almost the inverse (60/25/15 vs 20/24/56). However, since we could also segment our keyword universe based on length of keyword string and develop a traffic strategy based on that, let’s look at the analysis that way.

Hitwise completed research on the percentage of searches based on number of terms in the keyword string in January 2009 with these results:

Searches By Number of Terms from Hitwise January 2009

It you add the searches with 3+ terms in the keyword, you get 56.06% – so a pretty substantial amount of traffic, but not as high as the 70% I have heard from others.

We also need to adapt the conversion rates to maintain an average 1% across all three categories, so that the analysis between the two approaches is “apples-to-apples.” When you run the analysis based on length of keyword string with a conversion rate to get the same number of conversions as in the prior case, you get the following results:

Results of Analysis for Small Keyword Markets, By String Length

In this case, the results are even more dramatic, with the number of conversions for keywords with three+ terms dwarfing the one- and two-keyword strings. Even if you take the conversion rate for the three+ keywords down to 0.8% (the same as for two keyword search strings), the conversions are still almost double what is in the one-and two-keyword string categories combined.

So the answer to the question is an absolute “Yes” – the long-tail can be a very valuable source of business even for small keyword universes.

So I am going to take advantage of this negative and turn it into a positive for you. You will see how to perform various elements of site analysis by watching me debug this blog in near real time. Yesterday, I spent three hours working through the issues, and I am not done yet. So this first piece will take us about halfway there. But even now you can learn a lot from my struggles.

The first step was to find out just how bad the problem was. The way to do this is to use Pingdom’s Full Page Analysis tool. This tool not only tests page loading speeds but also visualizes which parts of the page are causing problems. An explanation of how to use the tool can be found here, and you should read it before trying to interpret the results for your site. Here is what I got back when I ran the test:

A load time of 11.9 seconds? Ouch!  Since Pingdom runs this on their servers, the speed is not influenced by my sometime unpredictably slow Comcast connection.

Pingdom shows I had over 93 items loading with the home page of which the vast majority were images (a partial listing is shown below).  There were several (lines 1, 36, 39, 40, 41, 54) where a significant part of the load time was occurring during rendering (that is, after the element had been downloaded into the browser).  This is indicated by the blue part of the bar.  But in the majority of cases, the load time was mainly caused by the time it took from either the first request to the server until the content began downloading (the yellow portion of the bar), or from the time of downloading to the time rendering began (the green portion).  This suggested that

1. I had too big a page, because the download time for all the content to the browser was very long.
2. I might have a server bandwidth problem.

But rather than worrying about item 2, which would require a more extensive fix – either an upgrade in service or a new host – I decided to see how far I could get with some simple site fixes.

The first obvious thing to fix was the size of the home page, which was 485 KB – very heavy.  I tend to write long (no kidding?) and add several images to posts, so it seemed only natural to reduce the number of entries on my home page below the 10 I had it set for.  I set the allowable number in Wordpress to 5 entries, saved the changes, and ran the test again. 

Miracle of miracles: My page now weighed 133 KB (respectable), had 72 total objects, and downloaded in six seconds.  That was a reduction in load time by almost 50% for one simple switch.  

Good, but not great.  My page loading time was still 6 seconds – it needed to be below 2.  So more work was needed.

If you look at the picture above, you can just make out that some of the slowest loading files – between 4 and 6 of them  – were .css or Javascript files.  Since these are files that are part of Wordpress, I chose to let them go for the moment and move onto the next obvious class of files – images.  Since images usually represent 80% of page loading times, this was the next obvious place to look.   There were between 6 and 10 files –  mainly .png files – that were adding substantially to download times.  Most of these were a core portion of the template I was using (e.g. header.png).  So they effected the whole site and, more importantly, they had been part of the blog before I ever made one entry.  The others were the icons in my Add-to-Any toolbar, which also showed on every post on the site.

Since I developed the template myself using Artisteer when I was relatively new to Wordpress, I hypothesized that an image compression tool might make a substantial improvement for little effort. 

Fortunately, the ySlow Firefox plugin, which is a site performance analyzer we will examine in my next entry, contains smushit, an image compression tool created by Yahoo! that is easy to use, identifies and shows just how much bandwidth it saves, produces all compressed files at the push of a single button, and produces excellent output quality.

So I ran the tool (I sadly did not keep a screenshot of the first run, but a sample output is below), and Smushit reduced image sizes overall by about 8%, and significantly compressed the size of the template elements.  So I downloaded the smushed images and uploaded them to the site

As you can see below – my home page was now 89.8 KB, but my load time had increased to 8.8 seconds! – and note on the right of the image that several prior runs confirmed the earlier 6 second load time. So either compression did not help or some other factor was at play. 

The fact is the actual rendering times had basically reduced from measurable amounts (e.g. 0.5 seconds) to milliseconds – so the actual file sizes had improved rendering performance.  Download times had increased – once again pointing to my host.  But before  going there, i wanted to see if there were any other elements on the site I could manipulate to improve peformance.

More in next post.  BTW, as I go to press this am, my site speed was 5.1 seconds – a new, positive record.  Nothing has changed – so more and more I’m suspecting my ISP and feeling I need a virtual private server.

NOTE: Even more important: as I go to press Google has just announced that it is adding a site performance tool to Google Webmaster Tools in anticipation of site performance becoming a ranking factor.

This was an instinct call, and while I believed I had data to support the theory, I had no research to prove that perhaps an underlying algorithm had been considered/created to measure this phenomenon in universal search. 

I thus considered myself twice lucky while doing my weekly reading on the latest patents to find one that indicates someone is thinking about the issue of "author rank."  On October 29th, Jaya Kawale and Aditya Pal of Yahoo!  applied for a patent with the name "Method and Apparatus for Rating User Generated Content in Search Results."  The abstract reads as follows:

Generally, a method and apparatus provides for rating user generated content (UGC) with respect to search engine results. The method and apparatus includes recognizing a UGC data field collected from a web document located at a web location. The method and apparatus calculates: a document goodness factor for the web document; an author rank for an author of the UGC data field; and a location rank for web location. The method and apparatus thereby generates a rating factor for the UGC field based on the document goodness factor, the author rank and the location rank. The method and apparatus also outputs a search result that includes the UGC data field positioned in the search results based on the rating factor.

Let's see if we can't put this into English comprehensible to the common search geek.  Kawale and Pal want to collect data on three specific ranking factors and to combine these into a single, weighted ranking factor, that is then used to influence rank ordering based on  what they term "User Generated Content" or UGC.  The authors note that typical ranking factors in search engines today are not suitable foir ranking UGC.  UGC are fairly short, they generally do not have links to or from them (rendering the back-link based analysis unhelpful) and spelling mistakes are quite common.  Thus a new set of factors is needed to adequately index and rank content from UGC.

The first issue the patent/algorithm has to deal with is defining what the term UGC includes.  The patent specifically mentions "blogs, groups, public mailing lists, Q & A services, product reviews, message boards, forums and podcasts, among other types of content." The patent does not specifically mention social media sites, but those are clearly implied. 

The second issue is to determine what sites should be scoured for UGC.  UGC sites are not always easy to identify.  An example would be a directory in which people rank references based on 5-star rating, where that is the only user input.  Is this site easy to identify as a site with UGC?  Not really, but somehow the search engine must make a decision whether this site is within its valid universe.  Clearly, some mechanism for categorizing sites with UGC needs to exist and while Kawale and Pal use the example of blog search as covering a limited universe of sites, their patent does not give any indication of how sites are to be chosen for inclusion in the crawl process.

Now we come to the ranking factors.  The three specific ranking factors proposed by Kawale and Pal are:

• Document Goodness.  The Document Goodness Factor is based on at least one (and possibly more) of the following attributes of the document itself: a user rating; a frequency of posts before and after the document is posted; a document's contextual affinity with a parent document; a page click/view number for the document; assets in the document; document length; length of a thread in which the document lies; and goodness of a child document. 
• Author Rank.  The Author Rank is a measure of the author's authority in the social media realm on a subject, and is based on on or more of the following attributes:  a number of relevant posted messages; a number of irrelevant posted messages; a total number of root documents posted by the author within a prescribed time period; a total number of replies or comments made by the author; and a number of groups to which the author is a member.
• Location Rank.  Location Rank is a measure of the authority of the site in the social media realm.  It can be based on one or more of the following attributes: an activity rate in the web location; a number of unique users in the web location; an average document goodness factor of documents in the web location; an average author rank of users in the web location; and an external rank of the web location.

These ranking factors are not used directly as calculated.  They are "normalized" for elements like document length and then combined in some mechanism to create a single UGC ranking factor. 

The main thing to note – and the item that caught my attention, obviously – is Author Rank.  Note that is has ranking factors that correspond with what I have been hypothesizing exist in the universal search engines.  That is to say, search results are not ranked only by the content on the page, but by the authority of the author who has written them, as determined by how many posts that author has made, how many sites he has made them on, how many groups he or she belongs to, and so on.

Can I say for certain that any algorithm like this has been implemented?  Absolutely not.  But my next task has to be to design an experiment to see if we can detect a whiff of it in the ether.  I'll keep you informed.

As you would expect, the conclusion is that if your site is hugely slow you will not get indexed and will not rank in the SERPs.  What is "hugely slow"?  Google has indicated that slow is a relative notion and is determined based on the loading times typical of sites in your geographical region.  Having said that, relative or not, from an SEO perspective I wouldn't want to have a site where pages are taking more than 10 seconds on average to load.  We have found from the sites we have tested and built that average load times higher than approximately 10 seconds to completely load a page will have a significant impact on being indexed.  From a UE perspective, there is some interesting data that the limit on visitors patience is about 6-8 seconds.  Google has studied this data, so it would probably prefer to set its threshhold in that region.  But I doubt it can.   Many small sites are not that sophisticated, do not know these kinds of rules, and do not know how to check or evaluate their site loading times.  Besides this, there are often problems with hosts that cause servers to run slowly at times.  Google has to take that into account, as well.  So I believe that the timeout has to be substantially higher than 6-8 seconds, but 10 seconds as a crawl limit is a guess, 

I have yet to see a definitive statement by anyone as to what the absolute limit is for site speed before indexing ceases altogether (if you have a reference, please post it in the comments).  I'm sure that if a bot comes to a first page and it exceeds the bot's timeout threshold in the algorithm, your site won't get spidered at all.  But once the bot gets by the first page, it has to do an on-going computation of average page loading times for the site to determine if the average exceeds the built-in threshold, so at least a few pages would have to be crawled in that case. 

Now here's where it gets interesting.  What happens between fast (let's say < 1-2 second loading times, although this is actually pretty slow but a number Matt Cutts in the video below indicates is ok) and the timeout limit?  And how important is site speed as a ranking signal?  Let's answer one question at a time.

When a site is slow but not slow enough to hit any built-in timeout limits (not tied to the number of pages), a couple of things can happen.   We do know that Google allocates bot time by the number of pages on the site and the number of pages it has to index/re-index.  So for a small site that performs poorly, it is likely that most of the pages will get indexed.  Likely, but not a guarantee.  It all depends on the cumulative time lag versus the average that a site creates. If a site is large, then you can almost guarantee that some pages will not be indexed, as the cumulative time lag will ultimately hit the threshold set by the bots for a site of that number of pages. By definition, some of your content will not get ranked and you will not get the benefit of that content in your rankings.

As an aside, by the way, there has been a lot of confusion around the <meta name="revisit-after"> tag.  The revisit-after meta tag takes this form <meta name=”revisit-after” content=”5 days”>. 
This tag supposedly tells the bots how often to come back to the site to reindex this specific page (in this case 5 days).  The idea is that you can improve the crawlability of your site by telling the bots not to index certain pages all the time, but only some of the time.  I became aware of this tag at SMX East, when one of the "authorities" on SEO mentioned it as usable for this purpose.  The trouble is that, from everything I have read, the tag is completely unsupported by any of the major engines, and was only supported by one tiny search engine (SearchBC)  many years ago. 

But let's say you are one of the lucky sites where the site runs slowly but all the pages do get indexed.  Do Google or any of the other major search engines use the site's performance as a ranking signal?  In other words, all my pages are in the index.  So you would expect that they would be ranked based on the quality of their content and their authority derived from inbound links, site visits, time-on-site, and other typical ranking signals.  Performance is not a likely candidate for a ranking signal and isn't important. 

If you thought that, then you were wrong. Historically, Google has said, and Matt Cutts reiterates this in the video below, that site load times do not influence search rankings.  But while that may be true now, it may not be in the near future.  And this is where Maile's comments took me by surprise.  In a small group session at SMX East 2009, Maile was asked about site performance and rankings.  She indicated that for the "middle ground" sites that are indexing but loading slowly, site performance may already be used to influence rankings.  Who is right, I can't say.  These are both highly respected professionals who choose their words carefully. 

Whatever is true, Google is sending us signals that this change is coming.  Senior experts like Matt and Maile don't say these things lightly.  They are well considered and probably approved positions that they are asked to take.  This is Google's way of preventing us from getting mad when the change occurs.  Google has the fallback of saying "we warned you this could happen."  Which from today's viewpoiint means it will happen.

Conclusion: Start working on your site performance now, as it will be important for SEO rankings later. 

Oh and, by the way, your user experience will just happen to be better, which is clearly the real reason to fix site performance. 

And it isn't only Google that may make this change.  Engineers from Yahoo! recently filed a patent with the title "Web Document User Experience Characterization Methods and Systems" which bears on this topic.  Let me quote paragraph 21:

With so many websites and web pages being available and with varying hardware and software configurations, it may be beneficial to identify which web documents may lead to a desired user experience and which may not lead to a desired user experience. By way of example but not limitation, in certain situations it may be beneficial to determine (e.g., classify, rank, characterize) which web documents may not meet performance or other user experience expectations if selected by the user. Such performance may, for example, be affected by server, network, client, file, and/or like processes and/or the software, firmware, and/or hardware resources associated therewith. Once web documents are identified in this manner the resulting user experience information may, for example, be considered when generating the search results.

In does not appear Yahoo! has implemented any aspect of this patent yet, and who knows what the Bing agreement will mean for site performance and search.  But clearly this is a "problem" that the search engine muftis have set their eyes on and I would expect that if Google does implement it, others will follow.

Today's topic is part of my Technical SEO series (I just named it that – now I have to go back and change all my titles and meta tags…sigh) – site load times and whether or not they effect how you rank in the SERPs.  It is another one of those topics that came out of SMX East.  In this case it was Maile Ohye, Senior Support Engineer at Google, who spoke to this issue.  Maile is a wonderfully knowledgable evangelist for Google.  I have seen her speak at many shows. Her presentations are always clear and contain good, actionable techniques for improving your rankings in Google's SERPs. I am not alone in thinking her knowledgable.   Stephan Spencer, one of the guys I most look up to in SEO,  thought enough of Maile to interview her in August of 2007, and she was also recently interviewed by SEOMoz, another leading light in the industry (and if you haven't used their pro tools, then you are one arrow short of a full quiver for your SEO work).   

So when Maile says "stuff," I listen.  In her talk at SMX East, she made note that poor site load times (we are talking something between good and absolutely horrible) could harm your rankings in Google search results. Let me define the problem, then try to explain what Maile was referring to, and finally my take on all this.

Basic Concepts of Site Loading Times for Getting Indexed

One the one hand, that site loading times effect search rankings isn't news.  Let's take some time to lay a bit of foundation, because the how of site speeds effecting search rankings didn't really hit me until Maile's talk.  It's one of those things that is obvious once you think about it, but it doesn't really come top of mind when you are focused on specific tasks in an SEO project.  It's a "given" in the background of your work.  Unless the site is so horribly slow that it is obviously impacting the user experience, you really don't think about load times when you are focusing on keywords and meta tags.  The site works, move on. 

But that's not really true from the perspective of the search bots.   Google and the other engines have to crawl billions of pages on the web on a regular basis, bring that information back, and then index it.  Some pages can be crawled infrequently, but as more of the web moves to more real-time information due to social media, the bots have to crawl more sites in real time in order to provide good results.  But there are only so many bots and so much time to crawl these billions of pages.  So if you are Google, you write your bots with algorithms that allocate this scarce resource most efficiently and, hopefully, fairly. 

How would you or I do this?  Well, if I were writing a bot, the first thing I would give it is a time limit based on the size of the site.  That's only fair.  If you have the ability to create more content, bravo.  I want to encourage that, because it is beneficial to the community of searchers.  So all other factors being equal (e.g. site loading time), I want to allocate time to ensure all your pages get into the index.  There is also the issue of search precision and relevance: I want all that content indexed so I can present the best results to searchers.   

Of course, I can't just set a time limit based on the number of pages.  What if one site has long pages and another one short, pithy pages (clearly not mine!)?  What if one site has lots of images or other embedded content while another does not?  My algorithm has to be pretty sophisticated to determine these factors on the fly and adapt its baseline timeout settings to new information about a site as it crawls it.

The next algorithm I would include would have to do with the frequency at which you update your data.  The more often you update, the more often I need to have my bot come back and crawl the changed pages on your site. 

Another set of algorithms would have to do with spam.  From the perspective of my limited resource and search precision, I don't want to include pages in my index that are clearly designed only for the search engines, that are link spammers, or that may only contain PPC ads and have no relevant information for the searcher. 

You get the picture.  I only have a limited window of time to capture continually changing data from the web in order for the data in my index to be reasonably fresh.  Therefore I've got to move mountains (of data) in a very short period of time but only so many processing cycles to apply.  And the number of variables I have to control for in my algorithms are numerous and, in many cases, not black and white.

This is where site load times come in.  If a site is large but slow, should it be allocated as much time as it needs to be indexed?  Do I have enough processing cycles to put up with the fact it takes three times as long as a similar site to be crawled?  Is it fair given a scarce resource to allocate time to slow site if it means I can't index five other better performing sites in my current window of opportunity?  Does it optimize search precision and the relevance of results I can show to searchers?  And last but not least, as one of the guardians of the Web, is poor site performance something I want to encourage from the perspective of user experience and making the Web useful for as many people as possible?  Let's face it, if the web is really slow, people won't use it, and the eyeballs that will be available to view an ad from which I stand to make money will be less. 

Hello?  Are you there?  Can you say "zero tolerance?"  And from the perspective of the universal search engines, there is also my favorite radio station – "WIFM."  What's In it For Me?  Answer: nothing good.  That is why Google has made page load times a factor in Adwords Quality Score, as an example.

So, in the extreme case (let's say a page takes 30 seconds to load), the bots won't crawl most, if any, of the site.  The engines can't afford the time and don't want to encourage a poor user experience.  So you are ignored – which means you never get into the indexes.

When Is a Page's or Site's Loading Time Considered Slow?

What is an "extreme case?"  I have looked that up and the answer is not a fixed number.  Instead, for Google, the concept of "slow loading" is relative. 

The threshold for a 'slow-loading' landing page is the regional average plus three seconds.

The regional average is based on the location of the server hosting your website. If your website is hosted on a server in India, for example, your landing page's load time will be compared to the average load time in that region of India. This is true even if your website is intended for an audience in the United States.

Two things to note about how we determined the threshold: 

• We currently calculate load time as the time it takes to download the HTML content of your landing page. HTML load time is typically 10% to 30% of a page's total load time. A three-second difference from the regional average, therefore, likely indicates a much larger disparity.
• We measure load time from a very fast internet connection, so most users will experience a slower load time than we do.

Moreover, Google has a sliding scale with which is grades a site.  The following quote applies to Adwords and landing pages, but my guess is similar algorithms and grading are used in determining how often and long a site is crawled:

A keyword's load time grade is based on the average load time of the landing pages in the ad group and of any landing pages in the rest of the account with the same domain. If multiple ad groups have landing pages with the same domain, therefore, the keywords in all these ad groups will have identical load time grades.

Two things to note:

• When determining load time grade, the AdWords system follows destination URLs at both the ad and keyword level and evaluates the final landing page.
• If your ad group contains landing pages with different domains, the keywords' load time grades will be based on the domain with the slowest load time. All the keywords in an ad group will always have the same load time grade.

We'll stop here for today.  Next time, we'll talk about happens in the nether regions between fast and clearly slow.

1. The tone was so angry and demeaning.  As I get older (and I hope wiser), I want to speak in a way that bridges differences and heals breaches, not stokes the fire of discord.
    
2. I believe the tone was angry in order to evoke strong responses in order to build links in order to rank high in the search engines.  Linkbuilding is a tried-and-true, legitimate SEO practice and so invalidates the entire argument Derek makes that understanding and implementing a well thought-out SEO program is so much flim-flam. Even more important to me, do we need to communicate in angry rants in order to get attention in this information and message-overwhelmed universe?  Is that what we’ve come to?  I sure hope not.
    
3. The article’s advice about user experience coming first was right (and has my 100% agreement).  But it’s assumptions about SEO and therefore its conclusions were incorrect.
    
4. The article’s erroneous conclusions will hurt a number of people who could benefit from good SEO advice.  THAT is probably the thing that saddens me most – it will send people off in a direction that will hurt them and their businesses substantially.  Good SEO is not a game.  It has business implications and by giving bad advice, Derek is potentially costing a lot of good people money that they need to feed their families in these tough times.
    
5. The number of responses in agreement with his blog was overwhelming relative to the number that did not agree.  That also bothered me – that the perception of our industry is such that so many people feel our work does not serve a legitimate purpose.
    
6. The comments on Danny Sullivan’s response to Derek were few, but they were also pro-SEO (of course).  Which means that the two communities represented in these articles aren’t talking to each other in any meaningful way.  You agree with Derek, comment to him.  You agree with Danny, comment there.  Like attracts like, but it doesn’t ultimately yield to two communities bridging their difference.

I, too, started to make comments on both sites.  But my comments rambled (another one of those prerogatives I maintain in this 140 character world) , and so it became apparent that I would need to create a blog entry to respond to the article – which I truly do not want to do because, frankly, I really don’t want to "raise the volume" of this disagreement between SEO believers and SEO heretics.  But I have some things to say that no one else is saying, and it goes to the heart of the debate on why SEO IS important and is absolutely not the same thing as a good user experience of web development.

So to Danny, to Derek, and to all the folks who have entered this debate, I  hope you find my comments below useful and, if not, my humble apologies for wasting your valuable time.

Good site design is about the user experience. I started my career in online and software UE design when that term was an oxymoron.  My first consulting company, started in 1992, was inspired by David Kelley, my advisor at Stanford, CEO of IDEO (one of the top design firms in the world),  and now founder and head of the Stanford School of Design.  I was complaining to David about the horrible state of user interfaces in software and that we needed an industry initiative to wake people.  His response was "If it’s that bad, go start a company to fix it."  Which I did.  That company built several products that won awards for their innovative user experience. 

That history, I hope, gives credibility to next next statement: I have always believed, and will always believe, that good site experience trumps anything else you do.  Design the site for your customer first.  Create a "natural" conversation with them as they flow through the site and you will keep loyal customers.

Having said that, universal search engines do not "think" like human beings.  They are neither as fast or as capable of understanding loosely organized data.  They work according to algorithms that attempt to mimic how we think, but they are a long way from actually achieving it.  These algorithms, as well as the underlying structures used to make them effective, also must run in an environment of limited processing power (even with all of Google’s server farms) relative to the volume of information, so they have also made trade-offs between accuracy and speed.  Examples of these structures are biword indices and positional indices.  I could go into the whole theory of Information architecture, but leave it to say that a universal search engine needs help in interpreting content in order to determine relevance. 

Meta data is one area that has evolved to help the engines do this.  So, first and foremost, by expecting this information, the search engines expect and need us to include data especially for them that has nothing to do with the end user experience and everything with being found relevant and precise.  This is the simplest form of SEO.  There are two points here:

1. Who is going to decide what content goes into these tags? Those responsible for the user experience?  I think not.  The web developers? Absolutely positively not.  It is marketing and those who position the business who make these decisions.
    
2. But how does marketing know how a search engine thinks?  Most do not.  And there are real questions of expertise here, albeit for this simple example, small ones that marketers can (and are) learning.  What words should I use for the search engines to consider a page relevant that then go into the meta data?  For each meta data field, what is the best structure for the information?  How many marketers, for example, know that a title tag should only be 65 characters long, or that a description tag needs to be limited to 150 characters, that the words in anchor text are a critical signaling factor to the search engines, or that alt-text on an image can help a search engine understand the relevance of a page to a specific keyword/search?  How many know the data from the SEOMoz Survey of SEO Ranking Factors showing that the best place to put that keyword in a title tag for search engine relevance is in first position, and that the relevance drops off in an exponential manner the further back in the title the keyword sits?  On this last point, there isn’t one client who hasn’t asked me for advice.  They don’t and can’t track the industry and changes in the algorithms closely enough to follow this.  They need SEO experts to help them – a member of the trained and experienced professionals in the SEO industry, and this is just the simplest of SEO issues.

How about navigation?  If you do not build good navigational elements into deeper areas of the site (especially large sites) that are specifically for search engines and/or you build it in a way that a search engine can’t follow (e.g. by the use of Javascript in the headers or flash in a single navigation mechanism throughout the site), then the content won’t get indexed and the searcher won’t find it.  Why are good search-specific navigational elements so important?  It comes back to limited processing power and time.  Each search engine has only so much time and power to crawl the billions of pages on the web, numbers that grow every day and where existing pages can change not just every day but every minute.  These engines set rules about how much time they will spend crawling a site and if your site is too hard to crawl or too slow, many pages will not make it into the indices and the searcher, once again, will never find what could be hugely relevant content.

Do UE designers or web developers understand these rules at a high level?  Many now know not to use Javascript in the headers, to be careful how they use flash and, if they do use it in the navigation, to have alternate navigational elements that help the bots crawl the site quickly.  Is this about user experience?  Only indirectly.  It is absolutely positively about search engine optimization, however, and it is absolutely valid in terms of assuring that relevant content gets put in front of a searcher.

Do UE designers or web developers understand the gotchas with these rules?  Unlikely.  Most work in one organization with one site (or a limited number of sites).  They haven’t seen the actual results of good and bad navigation across 20 or 50 or 100 sites and learned from hard experience what is a best practice.  They need an SEO expert, someone from the SEO  industry, to help guide them.  

Now let’s talk about algorithms.  Algorithms, as previously mentioned, are an attempt (and a crude one based on our current understanding of search) at mimicking how searchers (or with personalization a single searcher) think so that searches return relevant results to that searcher.  If you write just for people, and structure your pages just for readers, you are doing your customers a disservice because what a human can understand as relevant and what a search engine can grasp of meaning and relevance are not the same.  You might write great content for people on the site, but if a search engine can’t understand its relevance, a searcher who cares about that content will never find it. 

Does that mean you sacrifice the user experience to poor writing?  Absolutely, positively, without qualification not.  But within the structure of good writing and a good user experience, you can design a page that helps/signals the search engines, with their limited time and ability to understand content, what keywords are relevant to that page. 

Artificial constraint, you say? How is that different than the constraints I have when trying to get my message across with a good user experience in a data sheet?  How is that different when I have 15 minutes to get a story across in a presentation to my executive staff in a way that is user friendly and clear in its messaging?  Every format, every channel for marketing has constraints.  The marketer’s (not the UE designer’s and not the web developer’s) job is to communicate effectively within those constraints. 

Does a UE designer or the web developer understand how content is weighted to create a ranking score for a specific keyword within a specific search engine?  Do they know how position on the page relates to how the engines consider relevance? Do they understand how page length effects the weighting?  Take this example.  If I have two pages, one of which contains two exact copies of the content on the first page, which is more relevant?  From a search engine’s perspective they are equally relevant, but if a search engine just counted all the words on the second page, it would rank higher.  A fix is needed.

One way that many search engines compensate for page length differences is through something called pivoted document length normalization (write me if you want a further explanation).  How do I know this?  Because I am a search engine professional who spends time every day learning his trade, reading on information architecture and studying the patents filed by the major search engines to understand how the technology of search can or may be evolving.  Because – since I can’t know exactly what algorithms are currently being used –  I run tests on real sites to see the impact of various content elements on ranking.  Because I do competitive analysis on other industry sites to see what legitimate, white hat techniques they have used and content they have created (e.g. videos on a youtube channel that then point to their main site) to signal the relevance of their content to the search engines. 

And to Derek’s point, what happens when the algorithms change?  Who is there watching the landscape for any change, like an Indian scout in a hunting party looking for the herd of buffalo?  Who can help interpret the change and provide guidance on how to adapt content to maintain the best signals of relevance for a keyword to the search engines?  Derek makes this sound like an impossible task and a lot of hocus-pocus.  It isn’t and it’s not.  Professional SEO consultants do this for their clients all the time, by providing good maintenance services.  They help their clients content remain relevant, and hopefully ranking high in the SERPs, in the face of constant change.

So to ask again, do UE designers or product managers understand these issues around content?  At some high level they may (a lot don’t).  Do web developers? Maybe, but most don’t because they don’t deal in content – it is just filler that the code has to deal with (it could be lorem ipsum for their purposes).  Do any of these folks in their day-to-day struggles to do their jobs under tight time constraints have the time to spend, as I do, learning and understanding these subtleties or running tests? Absolutely, positively not.  They need an SEO professional to counsel them so that they make the right design, content and development choices.

I’ll stop here.  I pray I’ve made my point calmly and with a reasoned argument.  Please let me know.  I’m not Danny Sullivan, Vanessa Fox, Rand Fishkin, or Stephan Spencer, to name a few of our industry’s leading lights.  I’m just a humble SEO professional who adores his job and wants to help his clients rank well with their relevant business information.  My clients seem to like me and respect what I do, and that gives me an incredible amount of satisfaction and joy. 

I’m sorry Derek, but I respect your viewpoint and I know that you truly believe what you are saying.  But as an honest, hard-working SEO professional, I couldn’t disagree with you more.

So what happened?  Yesterday I published an article about the fact we should be using the term tailored versus targeted advertising and how our research showed that consumers would accept any number of emails as long a they were tailored to their specific concerns.  Today,  the Berkeley Center for Law & Technology at UC Berkeley School of Law ( Berkeley Law ), and the Annenberg School for Communication at the University of Pennsylvania published a study that reports that contrary to my research, most adult Americans (66%) do not want online advertisements tailored (yes, they used the right term) by marketers to their specific interests.  Moreover, when Americans are informed of three common ways that marketers gather data about people in order to tailor ads, between 73% and 86% say they would not want such advertising.

The study is called "Americans Reject Tailored Advertising and Three Activities that Enable It.".  What I find most surprising is that the study reports that even 18-24 year old feel this way (55%), which is lower than the average but still higher than I would have thought.  86% of young adults, the study goes on to say, don’t want tailored advertising if it is the result of following the online behavior, and 90% reject it if it results from following their activities offline.

I  went through the study in some detail and, so you know, I am very comfortable with survey methodologies and dealing with the vailidity of statistical results reported.  As part of this, I am always looking for holes in the methodology that could indicate bias.  The Berkeley/U Penn study had a better methodology that two prior surveys on the subject- one from Consumer’s Union conducted in 2008 and one that was conducted in 2008 by TRUSTe and repeated in 2009 by the Privacy Consulting Group.  So I think that, as a general matter, the survey is a better indicator of attitudes than anything done previously.  Having said that, there are two issues I have with the study:

1. Nowhere in the study was it asked whether or not the consumers would put up with tailored advertising if it is what paid for the information they got for free on the web.  This is a critical issue, because at the end of the day, consumers know that they are making this tradeoff – otherwise they would have fled sites with advertising in droves.
    
2. Asking people about what they would do is always dangerous.  There is a famous study by Oral-B (I think) where they put consumers into focus groups and asked them what color of toothbrush they liked most.  Universally, the two top colors reported were red and blue.  The participants were thanked for their participation and told that they could take several Oral-B toothbrushes free from baskets as they walked out the door (and yes, the baskets had a random mix of all colors).  What was chosen most frequently?  Amber.  So you see, what people tell you and how they act can be very different.

I would like to see a follow-on study that actually tests behavior, not responses to questions, especially in light of objection #1.  Having said that, I think the study is definitely on to something and we as an industry need to pay attention before concern turns into anger and the public vocally demands tighter privacy laws from their legislators.

The survey also uncovered other attitudes we need to be concerned about, and I have not seen them reported – so I quote them here for your convenience: 

• Even when they are told that the act of following them on websites will take place anonymously, Americans’ aversion to it remains: 68% “definitely” would not allow it, and 19% would “probably” not allow it.
   
• A majority of Americans also does not want discounts or news fashioned specifically for them, though the percentages are smaller than the proportion rejecting ads.
   
• 69% of American adults feel there should be a law that gives people the right to know everything that a website knows about them.
   
• 92% agree there should be a law that requires “websites and advertising companies to delete all stored information about an individual, if requested to do so.”
   
• 63% believe advertisers should be required by law to immediately delete information about their internet activity.
   
• Americans mistakenly believe that current government laws restrict companies from selling wide-ranging data about them. When asked true-false questions about companies’ rights to share and sell information about their activities online and off, respondents on average answer only 1.5 of 5 online laws and 1.7 of the 4 offline laws correctly because they falsely assume government regulations prohibit the sale of data.
   
• Signaling frustration over privacy issues, Americans are inclined toward strict punishment of information offenders. 70% suggest that a company should be fined more than the maximum amount suggested ($2,500) “if a company purchases or uses someone’s information illegally.”
   
• When asked to choose what, if anything should be a company’s single punishment beyond fines if it “uses a person’s information illegally,” 38% of Americans answer that the company should “fund efforts to help people protect privacy.” But over half of Americans adults are far tougher: 18% choose that the company should “be put out of business” and 35% select that “executives who are responsible should face jail time.

 Well, at least I guess we can put to bed the use of tailored versus targeted (although the survey did interchange them regularly).  See how good I am as an evangelist?