This changed on December 3rd when Google started redirecting most traffic through a new subdomain, plus.url.google.com.

If you want to see referrals from both plus.google and plus.url.google in the same report, set up a Matching RegExp Advanced filter to ‘plus.*google’

This should filter to include all Google plus subdomains.

Something else we’ve noticed is that traffic referred from Google Plus appears to have increased in the last week for the first time ever after a continuous decline.

This is probably a side effect of the change in tracking …or could they finally have reversed the trend and started to see some genuine growth?

However this really only makes sense if you’re able to respond in real time to the data which doesn’t fit the pattern for the majority of analysis and optimisation. If the data is useful to analyse in real time then it suggests you’re required to continuously monitor the data for a significant event to occur which is completely unrealistic for all but the largest online enterprises. It also requires a lot of site traffic within a 30 minute window to get enough data to be meaningful. For most of us it’s just a distraction.

However a couple of types of online disciplines that can genuinely benefit from this kind of real time analysis are News and Social Media.

Online news teams are interested in knowing what content is working best at the current moment and the best keywords for each article so they can maintain their managed front pages and generate more articles around the strongest stories and themes. Social Media teams are interested in knowing what content is working well in the social media space so they can try and build the momentum.

By default Google Analytics Real-Time is missing some useful filtering options. For example it won’t show you the content being viewed from a specific referral source, only the referring URLs, but you can hack Real-Time to get what you need by modifying the filter.list part of the URL. Here’s a few examples to get you started.

Browse to the Google Analytics real time Content report and add this to the end of the URL to see pages viewed by visitors from Facebook….

%3Ffilter.list%3D14%3D%3DREFERRAL%3B7%3D%3Dfacebook.com/

Or use this one for Twitter…

%3Ffilter.list%3D14%3D%3DREFERRAL%3B7~%3Dt.co%3B/

Add this filter to the URL to see the content from Organic visits.

%3Ffilter.list%3D14~%3Dorganic%3B/

If you want to see your top content for visitors from the United States, just add this to the end of the content report URL.

%3Ffilter.list%3D1~%3DUnited%2520States%3B/

To filter the content report to traffic from a specific keyword, add the following to the URL, substituting your own keyword.

%3Ffilter.list%3D14%3D%3DORGANIC%3B7%3D%3Dgoogle%3B8~%3Dkeyword%3B/

To view the Google entry search terms for a specific page, browse to Traffic Sources > Organic > Google, remove the last / from the URL and add the following, substituting ‘URL.html’ for your own URL.

10%3D%3D%2FURL.html%3B/

If you can’t work out the correct filter string, try using the search filter in each of the reports to see how they are generated by Google.

There’s lots of play here so let us know which filters you find the most useful and any we’ve missed.

Some people who add Open Graph tags and the Like button to their site see an uplift in traffic from Facebook. They see lots of Likes appearing on their Like buttons and think the Like button is great.

However the number displayed on the Like button is actually the total number of shares in Facebook and the new Facebook Insights shows us the number of Like button clicks is often very small compared to the overall number of natural shares in Facebook.

You can start to see that Like button clicks often contribute very little compared to the viral nature of social sharing.

It’s actually the Open Graph tags that are driving most of the benefits, not the Like button.

This opening paragraph from the Open Graph explains it.

The Open Graph protocol enables any web page to become a rich object in a social graph. For instance, this is used on Facebook to enable any web page to have the same functionality as a Facebook Page.

Adding Open Graph tags to a page allows Facebook to start aggregating data about all the sharing within the Facebook network and understand its overall importance. Facebook can begin to understand who else might be interested in the page and it gains additional exposure such as in search results which drives additional traffic.

Without the Open Graph tags, any time the URL is shared it doesn’t add to the overall importance of the page and it won’t appear in certain places.

The Like button does contribute to the importance of the page but not as much as everybody thinks and is not as important.

Here’s the guide to implementing Open Graph

developers.facebook.com/docs/opengraph/

You can use this URL linter to check to see if your pages Open Graph tags are working properly.

developers.facebook.com/tools/lint/

In conclusion….

1. Implementing Open Graph tags across all your pages is relatively easy and should be a priority
2. Adding the Like button is often more complicated as it requires design and layout changes and should only be added to pages where it makes sense for users to Like the page

They were using template based formats for the descriptions so that every page would have a unique description but the system failed and every description ended up being identical.

The effect on performance for these pages was instant and significant.

There was an immediate drop in entry visits of about 33% the day following the release.

The problem was identified and the fixed was release 9 days later. The recovery in traffic to previous levels was also effectively instant.

This clearly shows the harm that a duplicate description tag can have on a page’s performance. You’re better off not having a description tag than a duplicate.

DeepCrawl, our site management tool, has a report that shows all the duplicate descriptions on a website site so we’re recommending all our clients take a close look at their reports and see if there is an opportunity for improvement.

User Journey based Segmentation Matrix

—

DESCRIPTION: Each user starts the journey from being a new visitor to a website. New visits are attracted by either brand or non-brand campaigns resulting in “new non-brand” or “new brand” visits.  The goal is to convert new visitors into ideally “registered brand” users.

Users of different type should be a subject of customised messaging (delivered via appropriate medium) and adequate intensity according to the state of the journey they are in (the journey is indicated by the green arrows).

—

The original Robots.txt syntax was pretty straightforward. You could only use the Disallow directive to exclude pages and each Disallow directive acted like a broad match at the end. This seemed pretty intuitive to most people and for a while the world was a a happy place.

A few people with large and complicated sites discovered some exceptions that couldn’t be covered and so the Robots.txt was extended to allow further control with a some new features.

• Allow:
• * (match any number of any characters)
• $ (ends with)

Unfortunately, these were interpreted differently by each search engine and the supporting documentation is pretty thin on the ground.

Google’s documentation on the Allow command extends to a single example combining all 3 features.

The Allow: /*?$ directive will allow any URL that ends in a ? (more specifically, it will allow any URL that begins with your domain name, followed by a string, followed by a ?, with no characters after the ?).

Wikipedia’s page on Robots.txt suggests that Google process all Allow Commands first and only then moves on to Disallow.

From this limited information we interpreted the following 3 rules…

1. An Allow directive would always take effect over a Disallow directive.
2. A specific match using a $ would always beat a wildcard match using *.
3. A directive with a * at the end would work the same as a match with nothing at the end as all matches are broad by default

If you thought the same as us then prepare to be very surprised. After experimenting with the Robots.txt testing tool in Webmaster Tools, we found something completely different.

Example 1

Disallow: /example.html
Allow:    /example*

In the example number 1 Disallow beats Allow directly contradicting the rule number 1

Example 2

Disallow: */example.html*
Allow:    /example.html$

Example number 2 tears rule number 2 into pieces as despite the strong $ Allow loses against Disallow.

Example 3

Disallow: /example.html*
Allow:    /example.html

Disallow: /example.html
Allow:    /example.html*

And the final examples comes with no surprise rebutting the rule number 3 – depending on the placement of the wildcard, a Disallow beats Allow or an Allow beats a Disallow.

It took us a while to figure out and might take a minute to get your head around, but the answer is rather simple. The number of characters you use in the directive path is critical in the evaluation of an Allow against a Disallow. The rule to rule them all is as follows:

A matching Allow directive beats a matching Disallow only if it contains more or equal number of characters in the path

Just to clarify we’re talking about the number of characters in the matching directive path after the Allow: or Disallow: statement. This includes all the * and $ characters. e.g.

Disallow: /example*      (9 characters)
Allow:    /example.htm$  (13 characters)
Allow:    /*htm$         (6 characters)

In the following example, the URL /example.htm will be disallowed because the Disallow directive contains more characters (7) than the Allow directive (6).

Allow:    /exam*
Disallow: /examp*

If you add a single character to the Allow directive, the number of characters is equal and the Allow wins. An Allow directive with equal or more characters always beats a Disallow.

Allow:    /examp*
Disallow: /examp*

This even applies to exact matches using $. In the example below, the URL /example.htm will be disallowed because the matching Disallow directive contains more characters.

Allow:    /example.htm$
Disallow: */*example*htm

Another interesting side effect is that a broad match using a wildcard at the end becomes more powerful than one without due to the additional character. In the following example, the URL /example.htm will be disallowed because the Disallow directive contains more characters than the Allow directive due to the additional * character.

Allow:    /example
Disallow: /example*

Top Tip #1 – Tab space your Allow and Disallow directives

Which of these would win? The directives are not lined up so it’s hard to see.

Allow: /example.htm
Disallow: /********htm

This is better. You can see they are the same length so the Allow would win.

Allow:    /example.htm
Disallow: /********htm

Top Tip #2 – Retest a list of sample URLs every time you update the robots.txt

Use Robotto for free to monitor your robots.txt text files for changes and remind you to re-test a list of sample URLs in Webmaster Tools.

Or DeepCrawl to crawl your site in full and show you exactly what’s indexable and what’s disallowed, noindexed or canonicalised.

Bonus Info

After playing around with the robots.txt testing tool for a while we found 2 other interesting anomalies. These don’t actually affect the way robots.txt works because they only occur within competing Allow or Disallow statements and when the lengths of the directives are identical. We’ve included them here because they might be useful to help explain something that hasn’t been discovered yet and for completeness. They also suggest that the solution developed by Google might not have been as carefully planned as one would have expected or could give a  clue as to the underlying technology.

Within Allow or Disallow, A * beats a $

In this example, the second Disallow wins because it uses a * whereas the other uses a $. Both have identical numbers of characters in the directive.

Disallow: /example.htm$
Disallow: /example.htm*

Within Allow or Disallow, the highest number of non-wildcard characters wins

In this example, the first Disallow wins because it has a greater number of non-wildcard characters excluding the wildcards. Both have identical numbers of characters in the directive.

Disallow: /*xample.htm
Disallow: /****ple.htm

Conclusion

Although the way Google handles robots.txt files allows very powerful combinations to cover any scenario, it’s not intuitive or even documented sufficiently which is likely to result in a number of sites being incorrectly indexed. What do you think?

Google obliged by updating ‘Links to your site’ in Webmaster Tools to provide a full list of all backlinks which could be exported in full. This posed a few problems for large sites which have a lot of links. One of our bigger clients had about 80,000,000 backlinks and the downloaded CSV was in the region of 800MB. Working with this amount of data is not easy.

Somebody must have realised that this was either unusable in some situations or perhaps giving away too much information because Google have recently updated the tool again.

The reports are now broken down in a new way.

Who links the most

Top 1,000 domains that have links to pages on your site.

This report shows the top 1000 domains with the most links to your site. Unfortunately this probably means you can’t get information about single links from high authority sites which will get pushed down below sites linking to more than 1 page.

You get both a number of Links and Linked Pages figures for each domain which is very interesting. You can combine the 2 figures to get a rough idea of the linking profile. e.g. A site with lots of links to lots of pages is probably more interesting than a site with lots of links to just a single page.

For any domain which links, you can see up to 1000 pages which it links to. For each of your pages, you can then drill own and get up to 1000 backlinks pointing to that URL.

Your most linked content

Your top 1,000 pages that are linked from other domains.

The report shows the top 1000 pages which the most backlinks so you might not be able to analyse a specific page that only has a few links.

For any of the top 1000 domains linking to one of your top 1000 pages, you can see the top 1000 linking pages.

How your data is linked

Top 200 anchor texts used in backlinks.

The information in this report is initially very interesting but you quickly start to see issues. It looks like it’s including anchor text on internal pages. The order doesn’t seem to make much sense but this could be because internal links are included which skews things. You can’t view any more information about any of the anchor texts that are listed. Apart from giving a few ideas about new keywords, it’s hard to see what value you can get from this report. Suggestions are welcome.

Major Issues

• Results are limited to 1000 – useless for any sizable site
• Breaking down the report into multiple pages makes it much easier to navigate through the interface…but exported reports only show a single dimension and are completely useless for any sizable analysis.
• You can only view 100 results per page and the interface is missing a search tool so you have to click through up to 10 pages of results just to find information on a specific page (or even to find if the page you want is included).
• Google includes nofollowed links so you can’t assume any links you see are actually adding value without making this additional check.

Conclusion

Google has broken Webmaster Tools as a good source of extractable backlink information. They’ve taken away the comprehensive information and added a new interface which makes some things easier but prevents you doing anything particularly valuable such as looking at the growth of links to new pages.

We’re looking back to MajesticSEO and LinkScape as our primary sources of backlink data.

In september 2009 Matt Cutts of confirmed the commonly held view that Google has never used meta keywords tag. It’s unusual for Google to be this direct so apparently this issue hasn’t been easy to ignore and they probably thought this would be the final nail in the meta keywords coffin.

Some people mistakenly believed that Bing was paying attention to Meta Keywords tag based on an SEO 101 guide they put together for webmasters which included a section recommending their inclusion. However in a comment on an article discussing Yahoo’s use of meta keywords, Stefan Weitz from Bing confirmed categorically that Bing does not include meta keywords.

One of the main reasons given by advocates to continue to include them was that Yahoo considered them as part of its relevancy algorithm. Just 12 months later Yahoo’s search function is being powered by Bing so this remaining argument is no longer valid.

In the most recent SEOmoz Ranking Factors study based on the opinion of 72 ‘experts’, the Meta Keywords tag was voted as the least important on-page ranking factor although it somehow still managed to get a score of 5%.

Despite being ignored by the big search engines, there are still a few arguments to keep using them.

5 Reasons to Use Meta Keywords

• Clients may ask why they’re missing and it might take longer to explain than to include them
• They are an easy way to include misspellings
• They work with the Google Search Appliance
• YouTube uses them, maybe they know something we don’t
• They might work somewhere, at some point in the future

5 Reasons NOT to Use Meta Keywords

• It’s a waste of time that could be spent on something useful
• Your competitors can see exactly what keywords you are targeting
• It makes you looks like an amateur to other SEOs and potential clients
• Using competitor names/trademarked names might get you sued
• It’s a waste of bandwitdth

Who is still using Meta Keywords?

Before sharing our conclusions, we wanted to measure the the current level of meta tag usage on the web and find out how common they still are.

Methodology

We analysed the home pages of the top 100,000 websites for the US according to Quantcast.

The domain list provided by Quantcast includes only the root domain which we preceded with a www. subdomain to generate the test domains. In some instances this did not work. e.g. Blogspot.com is listed as a top domain due to the high number of subdomains so www.blogspot.com is not representative. The final number of domains which returned a page that could be analysed was 89,506 which should be a sufficiently large sample.

Note that these are the most popular websites, not a representative sample of all websites.

We used our DeepCrawl site architecture tool to crawl the home pages and extract the meta keywords tag if available. This was extracted using customised regular expressions as the meta keywords tag is not supported by default in our tools.

Sites were crawled from a UK IP address which might have led to redirection in a small number of cases or sites failing to be returned.

Results

• 67% of the websites that we analysed are still using the meta keywords tag. (60,100 out of 89,506).
• Of those, 95% had at least one character, 5% contained no content value.

Other Highlights

Apple has taken a minimalist approach which is either a breathtaking display of brand arrogance or more likely a light-hearted approach to reinforcing their philosophy of simplicity.

<meta name="Keywords" content="Apple" />

Despite being part of Google, YouTube is continuing to use them. The keywords themselves are unusual and don’t appear to have been formulated by someone with a knowledge of search who would surely omit words like ‘discover’ and ‘engage’.

<meta name="keywords" content="video, free, simple, search, find, discover, watch, engage, share, sharing, upload, entertainment">

Our analysis looks at websites which may retain legacy meta keywords tags that have been in place for many years and there might be a much lower rate of inclusion of meta keywords on newly developed websites. This would require a set of data identifying newly developed websites which is not available.

We are planning to re-examine this on an annual basis and see if there is any change in their use over time. We’d hope to see a reduction over time but it’s likely that the meta keywords tag will remain on many sites for the foreseeable future.

Our Conclusion

Our advice to clients remains the same. The ‘potential’ value of the meta keywords tag doesn’t warrant the effort and time that could be put to better use on pretty much any other type of SEO activity.

For the same reasons we don’t recommend active removal which also takes time but we would suggest removing them at the next convenient opportunity.

We’ve also made the decision not to support the Meta Keywords tag in any way in our products.

Some other opinions

• Malcolm Coles has also been quite vocal in his opposition
• Joost wastes no time dismissing them
• Others are still recommending them as part of a complete SEO strategy.

What about You?

Note: There is a poll embedded within this post, please visit the site to participate in this post's poll.

Let us know if we’ve missed any good arguments for using them.

UPDATE2: Google Analytics have confirmed it on their blog.

The launch of Google Instant has caused a lot of excitement and SEOs around the world are wondering how this will affect their clients.

Google’s initial attempts to develop an instant search product were met with criticism from the SEO community as the referrer string was not being passed which would have pretty much ruined web analytics as we know it.

Fortunately they have solved the problem this time by making the Google URL update in the website brower address bar shortly after the search results are updated. There is a small delay before the URL appears to update but it always updates before the new page is loaded ensuring the correct referring URL is always passed.

As you type a search into Google, the results update to Google’s suggested search term which appears in grey. If you click on a result whilst a grey result is showing up then this suggested query is passed in the referring URL and this is what will show up in your keywords report in analytics.

In addition Google also include the original search query in another parameter called oq. If you check Google’s URL string you will see something like this.

aqi=&aql=&oq=seo&gs_rfai=&

Because this is passed in the referring URL, it’s possible to extract it and include it in a report using Google Analytics filters. Here is how you can track these original search queries alongside the suggested search queries with a few minutes work.

Setting up Google Analytics to track Google Instant

Edit: Dan Barker has pointed out that the oq parameter is still used by the old Google Suggest so this custom profile will show the results for both until the old version has been completely deprecated. It’s possible to update this to only include data from Google Instant by adding another filter to only include traffic which contains another unique parameter such as &sclient=psy.

1. On the Google Analytics Overview screen, click the Add new profile link.

2. Select ‘Add a Profile for an existing domain’ and then find the correct account from the drop down. Give your new profile a name like ‘Google Instant’ so it’s easy to find and click Finish. This will create a new profile that will start to capture all the raw data for your site.

3. On the Overview screen, select the Edit link for the new Google Instant profile we’ve just created.

4. Select the ‘Add Filter’ link on the right hand side of the ‘Filters Applied to Profile’ table.

5. Create a new filter by entering the following details.

Filter Type: Customer Filter > Advanced

Field A -> Extract A
Referral: [?|&]oq=([^&]*)

Field B -> Extract B
Campaign Term: (.*)

Output To -> Constructor
Campaign Term:  $B1|$A1

Field A Required: Yes
Field B Required: Yes
Override Output Field: Yes
Case Sensitive: No

To briefly explain is happening here, we use a regular expression in Field A to extract the original search query from the oq= parameter that has been passed in the Referral string. We then use Field B to get the final search query from the Campaign Term variable. In the final step we output both search queries back to the Campaign Term separated by a | character. We will now be able to see this showing in the Keywords report.

Click the Save Changes button to create the filter.

After a few days you should have collected some data in your new profile and you can view the results.

Go to your keywords report and search for \| using the keyword filter.

Then you should be able to see results that look like this example.

You can see that some people hadn’t even gone beyond ‘stric’ before they clicked on suggested search for ‘strictly come dancing 2010′ which is the 2nd suggested search from Google.

Conclusion

Using these reports we can begin to understand the way people use the new search function and look for ways to optimise towards it.

The example above it has shown that Google Instant might helped to generate a higher volume of searches for more detailed search phrases than previously seen.

Perhaps the biggest impact we’re likely to see is a significant reduction in the number of unique phrases as people take advantage of the suggested results, effectively killing off the long tail of search. This might be a good thing for users as it may improve the quality of search but it will make SEO much harder as we all have to start fighting over few phrases which will drive more traffic.

This is probably going to be a win for the big players who already do well outside of the long tail and bad news for new sites trying to get a foothold.

Analysing the data for a few large sites quickly showed that descriptions of 49 characters or less are included in the report.

So what is the impact of having a ‘short’ meta description?

On the main HTML Suggestions screen it says…

These issues won’t prevent your site from appearing in Google search results, but addressing them may help your site’s user experience and performance.

…which sounds like it could be important.

However the wording above the short meta description report says…

Meta description information can give users a clear idea of your site’s content and encourage users to click on your site in the search results pages.

…which is a bit more vague and doesn’t suggest this specific issue will affect performance.

On the title tag reports the wording is much stronger…

We recommend reviewing the list and updating the title tags wherever possible.

We checked the ranking performance for a few pages with short descriptions and there were no obvious issues. Google simply used the short description and appended snippets to make up a full description. This analysis was only really a spot check and not a detailed study.

It’s logical to assume that Google’s decision to set the alert level at 50 characters and to add snippets to short descriptions is based on their own extensive analysis of user behaviour. Although they are probably not including it as part of their ranking algorithm, they appear to be suggesting their is a benefit in a more detailed description.

Our Conclusion

Despite a lack of any evidence to suggest short descriptions negatively affect ranking performance it makes sense to avoid them because of the assumed benefits in terms of click-through-rate from a controlled description.

We’ll be adding our own Short Descriptions report to DeepCrawl, our site architecture management tool, in the next release.