But the thing that really interests me is how Google determines which links are paid and which aren’t. If you’re an SEO, when you see a paid link, in most cases it’s generally pretty obvious if it’s unnatural or paid for – but it’s not as simple for a machine to detect.

There’s been some speculation as to what kinds of signals Google is looking at – sites that have the link warnings apparently tend to have a lot of sitewide links, most likely in footers and sidebars – and they also often have a very high keyword to brand anchor text ratio.

I’m not convinced that the ratio of anchor text used is enough to flag links as suspicious, at least not on it’s own. If the site or page doesn’t have that many links, then it’s a small sample size that could be easily skewed, and could lead to a lot of false positives. Another issue is that exact match domains would effectively get a free pass (although, that might still be true).

Time as a signal

I have a theory – and please note, this hasn’t been proven – that Google is looking at another signal to work out which links are suspicious. One of the big differences between paid links and natural links is when they’re placed. The majority of paid links are added to pages retroactively – i.e. a website has a page that mentions car insurance and a company might then approach them and offer to pay them on a monthly basis to change that text to a link.

I believe that if Google has crawled a page, and then at a later date recrawls that page and discovers a new link – with hardly any extra content added – that link is now flagged as suspicious. They might devalue it, they might send out a webmaster tools message or they might do both – but that link could well be flagged. The exception to this is if the page they’re crawling is the homepage, and potentially category pages, where content might change frequently.

If a reasonable chunk of text is also added at the same time as the link, then it potentially wouldn’t be flagged (so genuine updates to news articles wouldn’t accidentally flag that link).

How Google could deal with sidewide links

Other times, a paid link might be added to a sidebar in the form of a banner ad, or in a blogroll link, or as a link in the footer. These are, in 99% of cases, now sitewide links. They’d potentially trip the same filter as above, because those links would appear on pages that Google has already crawled, but there’d also be a higher percentage of false positives here (i.e. good links being flagged as bad) as bloggers often link to sites they genuinely endorse in blogrolls too.

If I were Google, I’d treat those links differently to deal with the increase in false positives. Unless I was confident that the link was classified correctly as either paid or natural, I’d consider silently devaluing that link and not sending out a link warning. After a time limit (maybe 6 months, maybe a year), I’d allow that link to start flowing Page Rank. If you’re buying links, you don’t want to pay for them and not have them work for months – you might be more likely to notice that the links you’re building aren’t working, so you stop renewing them. If it’s a genuine editorial link in a blogroll, then it’s more likely that that site can wait a while before getting the link value – because that link is mainly serving to pass them useful traffic.

“I’d like to get a few paid link reports anyway because I’m excited about trying some ideas here at Google to augment our existing algorithms” – Matt Cutts, 2007

I think this is probably something that Google have been doing for a while, way before the webmaster tools warnings were sent out. Matt Cutts mentioned in the past that they’ve been working on algorithms to automatically detect paid links, and I imagine there are probably other signals they’re looking at too.

How Google could detect paid links is a post from: Shark SEO. Have you played The Search Game?

What’s less clear is just how much Google weights the anchor text in it’s algorithm compared to other search engines. Ordinarily it would be difficult to test this – you’d need to find a huge range of varied sites, all linked to with a common phrase. Luckily, Hacker News is a good example – if you use the phrase “Show HN” (as people often link to their new startup/project using that phrase).

Here’s what Duck Duck Go displays for the query “Show HN”:

While it (brilliantly) has a !bang syntax for searching HNSearch.com, it’s regular results show pretty much what you’d expect – sites that use “Show HN” in the title tag of the page and within the text of the page, along with something like hn-show.com which features the keywords within it’s domain name.

And here’s what Blekko shows:

Blekko is relatively similar, in that it promotes sites that use the word “Show” in the title tag and on the page a lot (maybe not as much with “HN” though).

Here’s Bing’s results:

Bing, weirdly, doesn’t have any results from Hacker News in it’s top 10 – the first result is from FriendFeed. After that, it very heavily focuses on the keyword being in the domain name or in the title tag & on-page text.

And finally, here’s Google’s results:

Other than the first result from HackerNews, not a single listing features the text “Show HN” in either the title, domain or on-page text. They’re ranking for the phrase, despite not mentioning it anywhere on the page, because some of the links pointing to them include the phrase “Show HN”.

Keep in mind, this may be an edge case – typical on-page weightings might be dialled up for search terms that are more heavily searched for.

I’m not saying that this is necessarily a flaw in Google’s results at all – I much prefer Google’s results in this edge case than to the other search engines. I just wanted to highlight how Google appears to weight anchor text very heavily – much more so than the others.

Google’s Love Affair with Anchor Text is a post from: Shark SEO. Have you played The Search Game?

Every time Googlebot arrives at your site, it will first check to see if you have a robots.txt file. If the robots.txt file blocks any pages, Google won’t crawl them.

For years, website owners and web developers have used the robots.txt file to block Google from accessing duplicate content. From blocking URLs that use tracking parameters, blocking the mobile or print version of sites or just to fix flaws in CMS’s, I’ve seen a lot of duplicate content blocked with robots.txt in my time.

Why blocking URLs doesn’t help

But the robots.txt file is a terrible way to deal with duplicate content. Even if you’re 301 redirecting the duplicate URL to the real one, or using the canonical tag to reference the proper URL, the robots.txt file works against you.

If you have a 301 that redirects to the proper page, but you block the old URL with robots.txt, Google isn’t allowed to crawl that page to see the 301. For example, have a look at Ebooker’s listing for ‘flights’:

The URL that’s ranking (on page 1 of Google for ‘flights’) is blocked in robots.txt. It’s got no proper snippet because Google can’t see what’s on the page, it’s had a guess at the title based on what other sites have linked to it with. And here’s the reason why Google can’t crawl that URL:

If Ebooker unblocked that URL, Google would be able to crawl it to discover the 301, and the page would most likely have a better chance of ranking higher (as it wouldn’t just appear to be a blank page to the search engines).

If you block Google from seeing a duplicate page, it’s not able to crawl it and see that it’s duplicate. If there’s a canonical tag on that page, it may as well not be there as Google won’t be able to see it. If it redirects elsewhere, Google won’t know.

If you have duplicate content, don’t block the search engines from seeing it. You’ll just prevent the links to those blocked pages from fully counting.

Flickr image from Solo.

Robots.txt & Duplicate Content is a post from: Shark SEO. Have you played The Search Game?

Patrick Altoft had an interesting tip about leaving the brand name out of the title tag – while this may not work for everyone, the idea is that for a generic keyword search (like “red widgets”), Google may display a title that’s optimised for that term. If the search term is branded, however (“Brand name”), then Google will most likely use the Dmoz title.

In a similar way, you can actually have multiple meta descriptions – potentially one for the keyword, and one for the brand name. This isn’t recommended for everyone, and I wouldn’t recommend it for many pages on your site, but it’s possible. The regular limit for meta descriptions to be displayed in full in Google is 156 characters (although I tend to stick to around 154 characters). I recently experimented with having a double length meta description – with the first snippet being designed to be well written for a generic keyword, and the second snippet written for a brand search. The full meta description for my homepage is this:

“Dave is a freelance SEO consultant, specialising in creative link building and in-depth technical site audits. To find out more, feel free to get in touch. Shark SEO is a search marketing blog with free advice on ranking your site better in Google, Bing & Yahoo. Check out the SEO blog today at SharkSEO.com.”

That’s twice the length of a regular meta description. Now check out the snippet for “freelance SEO consultant”:

And here’s the snippet for the search “Shark SEO blog”, which again returns the homepage:

When you put multiple snippets in the same meta description tag, it looks as if Google will use the snippet that’s most suitable for the query.

Flickr image by Amir K.

Multiple Meta Descriptions is a post from: Shark SEO. Have you played The Search Game?

The set-up process is very straight-forward. You first add your URL, and then sync it up with your Google Analytics account – this step is optional, you can skip the GA import and just copy and paste the keywords you want it to focus on (useful if you’re using a different analytics package to Google Analytics). The tool then runs through all of the selected keywords – the ones you’ve manually put in, or the ones that have driven traffic to the site if you’ve used the Google Analytics option – and checks your ranking for that term, along with the estimated search volume (on exact match) from Google’s keyword tool. The ranking data can take a while to be gathered, depending on how many keywords you’ve added, but the search volume appears remarkably quickly. You are then presented with your keywords, graphed with where they rank, what their rough potential is based on the expected search volume and by how much traffic they’ve actually sent you.

You’re then free to start creating categories and filters.

Categories

One of the main features is being able to group sections of your keywords into categories. For example, the default category is ‘Brand’, and automatically includes all of your brand terms. You can create your own categories if you want to further seperate things out – for example, if you’re a clothing retailer, you can create a category for things like ‘shoes’ and ‘boots’, one for ‘jeans’, and so on. As an example, the graphs used here are from a snowboard site that I own – I have categories for “brand name”, “boots” and “boards”.

The one thing that I’d really like to see is a negative option, so you could create a category that *doesn’t* include your brand terms. That change would mainly benefit sites that have a huge amount of longtail brand traffic that are using the GA import option, although it’s far from essential.

Setting up categories is ridiculously easy, and gives you the ability to quickly drill down into particular product areas (especially useful if you’re dealing with a huge data set). This is particularly useful for retail sites, as you can seperate out product types easily and quickly.

Filters

Filters allow you to manipulate your data by setting up rules – rules like “show me keywords that have a search volume greater than 1000 according to the AdWords keyword tool, that I rank at the bottom of page 1 for and that sent me at least 50 visits”. This is what that rule would look like:

Easy.

You’re then free to slice and dice the data to how you see fit – as an example, I now have a list, in order of priority, of which terms I should go after if I want to sell more snowboards:

The graph has been filtered to only show keywords that mention “boards” or “snowboards”, that have sent traffic to the site already and where I’m not ranking in the top 5.

The real beauty in this is the speed at which you can slice and dice the data. If I wanted to focus on boots, or snowboard bindings, I can be up and running within seconds. That’s especially useful if you have a large site with a lot of potential categories. The best way to see it in action, though, is to actually play with it – Richard has been kind enough to give out a coupon. At the sign up page, if you use the coupon code SHARKSEO, you can get 1 month free.

Flickr image by C.SooHoo.

Clever Keyword Research is a post from: Shark SEO. Have you played The Search Game?

I’m not going to explain it all in too much detail because I think those two articles cover it quite well, but as a quick summary:

1. Google suspected Bing of using some of Google’s data in Bing’s results
2. Google set up a test to prove this – by allowing pages to rank for “synthetic queries” (Googlewhacks), using IE8 with the Bing bar installed to search for and then visit those pages, and then found Bing returning around 9% of those results a few weeks later
3. Bing very strongly denied “copying” Google’s results once accused

Bing’s description of what’s happening appears to be around the use of “clickstream data” – it sounds like the Bing toolbar (and IE with suggested sites) looks at which pages you’re on and which pages you visit afterwards. This isn’t restricted to Google – this is, apparently, for all pages on the Internet.

There’s arguments from people saying that Google is right to find this unacceptable, and others saying that Bing is in the right.

I was actually quite surprised by the number of people siding with Bing over this, there’s something about Bing using it’s browser to collect user data from competitors that doesn’t sit quite right with me. Regardless, I was surprised by some of the things that Bing said to defend itself.

Google engaged in a “honeypot” attack to trick Bing. In simple terms, Google’s “experiment” was rigged to manipulate Bing search results through a type of attack also known as “click fraud.” That’s right, the same type of attack employed by spammers on the web to trick consumers

– Yusuf Mehdi, Bing.

What Bing is complaining about here, is that Google engineers chose to adjust Google’s results for specific terms, searched in Google for those keywords and then clicked on those listings. In Google. That’s not an “attack”, nor is it a “trick” and it’s definitely not “click-fraud”.

Bing also mentions that the clickstream data that they’re using is one of 1,000 signals used to determine where a site should rank, and that the honeypot keywords that Google used were noticeable because they were outliers – and as such they only really had the clickstream data to go on.

How much of the clickstream data, is actually data from Google?

But this is what I don’t fully understand – the clickstream data itself. Bing says that the clickstream data isn’t just for Google – it’s for all sites on the web. But of course, Google – their biggest competitor – is the second most visited site on the Internet from the US, so it’s fair to say that a very hefty chunk of that clickstream data actually contains data from people searching on Google.

What happens when the clickstream data is scaled?

The other thing I don’t understand is what happens when you scale that clickstream data. We’ve only seen what happens when it’s used on 100 invented terms from Google’s honeypot test, where around 9% of those queries then appeared to affect Bing’s results. Bing implies that this isn’t a lot, and that the effect is much smaller when it’s scaled – but I’m not so sure. I’d actually be quite surprised if, when this was scaled to something the size of the Bing toolbar’s userbase, there wasn’t a very noticeable impact on Bing’s results. This is one of those things that cannot really be proved – we have to take Bing’s word for it.

Is Bing morally right to take Google’s user data?

During the Farsight video, the Bing rep mentioned that they were only using publicly available clickstream data – but of course, that data isn’t publicly available. The data is coming from a toolbar, and the conditions are, let’s face it, buried away somewhere in a EULA which nobody in their right mind ever reads. These users have legally opted in to sharing that data, but I don’t think they’re aware of it.

Regardless of that, though – Bing is taking data from Google users, who are searching on Google and allowing it to influence Bing’s search results. It may be legal, but it doesn’t mean you have to agree with it.

Flickr image from reway2007.

Bing’s Google Argument Makes No Sense is a post from: Shark SEO. Have you played The Search Game?

While a lot of SEOs had begun to suspect that tweeted links were influencing rankings, it was really good to see it actually confirmed.

What Google & Bing didn’t mention, though, was how strongly they were using these social signals as a ranking factor. Google has claimed for years now that there are over 200 ranking factors, so it’s hard to say whether their use of Twitter is a majorly influential factor (like links) or whether it’s just one of many neglible factors.

Google also failed to mention how long the Twitter effect would last – I think quite a few people may expect it to be a very time-sensitive thing, particularly around breaking news. The assumption is that, when Google uses tweets to boost a page for a search term, the ‘Twitter effect’ will eventually stop being such a strong ranking factor after enough time (or when the tweets stop) and then the regular SEO factors (links, on-page keywords, etc) start to take over. This wasn’t confirmed or suggested, it’s just what I would have expected.

A final point that wasn’t mentioned is whether or not Google differentiates between tweets from specific countries – so whether tweets from UK users to a specific page helps boost that page in Google.co.uk, or whether it also helps in US results in Google.com.

These two points – tweet locations and how long the Twitter effect lasts for – is something that I wanted to look into because of a post I wrote a while ago on Raven Tools. I wrote it very shortly after Sugarrae published hers, and I noticed something interesting about the two posts – my post very quickly started to rank very well for the term “Raven Tools” in Google.co.uk, out-ranking Rae’s even though I linked to her post from mine, and despite the fact that Sugarrae’s post, by all the regular SEO metrics like number of links and domain authority, greatly deserved to outrank my post. My post ranked so well on Google.co.uk that the only domain that outranked it was Raventools.com itself. This wasn’t true in Google.com though, the US results showed the results that you’d normally expect, with Sugarrae outranking me and with my site towards the bottom of page 1. I should also point out, my site isn’t geo-targetted to any location in particular.

Is there a time limit to the Twitter effect?

At the time I assumed it was some kind of query-deserves-freshness effect, and that eventually my site would drop down the search results. That would fit with my original idea that Google’s use of Twitter is to spot breaking news and promote tweeted articles when the topic was hot, but then dropped those articles in favour of the most linked to over time, when the topic wasn’t being tweeted about as much.

It’s been over 5 months since my Raven post, and it’s still only outranked by Raventools.com in the UK.

This would imply that, in this case at least, the Twitter effect may not be time-based, and tweets from months ago may still help your page to rank well.

Does Google use tweet locations?

I wanted to look into why my post was ranking well in the UK results, but not anywhere else. It’s a .com, hosted in the US and it isn’t geo-targetted to any country, Google shouldn’t consider it a UK specific site.

Using Backtweets I grabbed a load of the data around who tweeted my post and compared it with who tweeted Sugarrae’s. An important point to remember is that Google is likely treating some tweets diffently to others, depending on how authoritative they think a Twitter user is.

While Sugarrae had more tweets to her article than I had mine (she had 23 to my 13), the majority of my tweets were from people who had their location set to somewhere in the UK (9 of the 13), while Sugarrae had the vast majority of her tweets from the US (17 of her 23), and she only had 2 UK tweets.

This would suggest that Google is using the location of tweets to determine which search engine the page gets a boost in. The theory is, if a page becomes incredibly popular amongst UK tweeters – it may only be relevant to people in the UK, and so it only gets a boost in Google.co.uk. This is an observation for just this one specific example – it’s not a cold, hard scientific fact – but if anyone was planning on testing how tweeted links can affect rankings, I’d suggest looking into how long the effect lasts for, and whether the location of the Twitter user plays a part.

And you can download the sheet here, if you’re so inclined.

Flickr image from view-askew.

Thanks to SEO Scientist Neyne for the title advice.

Geolocation of Tweets Affects the Rankings in Local Google is a post from: Shark SEO. Have you played The Search Game?

If anyone knows what this device is – please, please put me out of my misery and let me know in the comments.

Flickr photo from Leo Reynolds

Does Google Have A Secret Tablet? is a post from: Shark SEO. Have you played The Search Game?