What does this mean for my websites? A few things:

• I will not be building any new tools. All my tools that already exist here and elsewhere are going to remain as they are and will not be updated.
• Very important corollary: The tools here on eKstreme.com and elsewhere, especially the SEO tools, were built before I joined Google and thus are not in any way official Google tools and must not to be taken as endorsed by Google. So please no one try this one. OK?
• I’ve been running some SEO experiments here on eKstreme.com and elsewhere. Although I’ve stopped them, their effects might still be hiding in a cache somewhere or whatnot. If you see anything fishy/funny it is NOT an official Google recommendation of how to do things. Heck, as an SEO I’ve pushed the boundaries a bit and so if you try anything you think I’ve tried or trying, it’s at your own risk.
• My OpenCourseWare search engine (OCW Search) is now officially part of the OpenCourseWare Consortium. As I will not be able to work on it from now on, I’ve decided to donate it to the Consortium instead of shutting it down. Its dream lives on and it is now in much more capable hands than mine, hands that even have time to work on it!

Some Yes questions: So will I start blogging more? Hopefully. Will I be at conferences? Yes, though I don’t know when. I’ll post here and hopefully get to meet more of the people I’ve been talking to over the years.

Some No questions: Will I tell you SEO secrets if you ask me? Of course not! Can you hire me? No! I’m done with the freelance work. Can we exchange links? Nope.

A little disclaimer: This website is still my own and anything written on it is my own personal view/opinion and not my employer. This is not an official Google blog.

Finally, you should follow me on twitter.

Before I go through some of the data they share, note that they have a toolbar that allows you to "view SEO data in real time". And because they allow you to mark pages as spam to kill them completely from your search results, the toolbar has a button to do that too. Interesting way to crowdsource spam control.

Back to SEO data: To see it, you simply search for the domain name you’re interested in followed by a /domain. For example, for ekstreme.com: ekstreme.com /domain. That first page is a gold mine in its own right, as it tells you how many inbound links from how many domains. And for each domain it lists, you can dig in a bit deeper to understand that domain. This is actually the first in many tabs shown. Clicking through them you’ll see they display a few graphs (sadly, the ever useless pie charts), give crawl stats data, and duplicate content.

The crawl stats data hints at how fresh/stale their data is, and from testing a few domains I own and some of my SEO clients’ domains, it’s clear the freshness is a big issue: it’s very hit and miss, and for newer and/or smaller sites, they are lagging significantly. For example, when analyzing OCW Search, the numbers indicate they have crawled 8 pages (there are more ) and the average page length is 0kb, which is clearly wrong. They also tell you the reverse IP address and for OCW Search it’s Amazon, which is the previous host that I moved away from a couple of months ago. So take this data with a grain of salt!

The last report I want to highlight is the comparison. By default when you click on the compare tab, you will be comparing the www domain with the non-www domain. Again, for ekstreme.com: ekstreme.com /compare. This tells you who is linking to each domain which is a good rough estimate of the problem size if you are dealing with canonical URL issues. BUT, the real kicker here is that you can compare different domains, for example ekstreme.com and www.ocwsearch.com (screenshot below). This is a gold mine for market analysis.

So all in all an excellent free tool, but one that suffers from stale data – the problem for all SEO tools. My recommendation is like any other SEO tool: use it while understanding the data’s limitations.

Finally, I’d like to note that this is a genius marketing strategy on blekko’s part: getting SEOs to talk about them and use them is a great way to get early traction. Us search geeks are the very very leading edge of early adoptors, so well done on spotting this opportunity.

The last blog post was from way back in January. Things has moved on since then but I just haven’t had time to blog about them. So here goes:

• I’m now a freelance web developer and SEO. Want to hire me? Look at my funnily-named SEO company website.
• I built, launched, and got sucked into the resulting whirlwind of OCW Search, a free online courses search engine. OCW Search helps you find free downloadable courses from universities like MIT, Notre Dame, The Open University in the UK, and many more. Yes, I’m an SEO and I operate a search engine now. Let me tell you it’s an awesome experience.
• I’m expanding OCW Search into a full-featured online education startup. Sign up to the mailing list to get access before everyone else You know you want to!
• In June 2010, I gave a talk at MongoUK about the technology behind OCW Search.

And if you’re reading this, you’ll notice that eKstreme.com now looks different. Well I thought that the previous design was getting old and I was moving the site to a new server (and a new PHP framework I built), and thought this is a good chance to give it a fresh look.

Speaking of which, I retired a LOT of the tools and code previously available on eKstreme.com. Why? I just don’t have time to provide support and to keep updating them as APIs change and bugs are identified. I have a different focus now and it’s not fair for users or me to keep semi-functional code released without support.

My talk will be about analytics for social media marketing. Whenever you launch a marketing campaign, you need to measure it in detail to understand its performance. I will talk about some of the important actionable metrics you need to track, with code examples of how to track them. Other key topics covered are filtering, automation, and reporting, all of which feed into experimentation to find the most effective marketing messages.

For example: I keep an alert for [blogsci] because I have a website at blogsci.com. Up till recently, I used to get alerts only when the word "blogsci" was matched in a page. Now, I’m getting Alerts for pages that do not ever mention the word "blogsci" but the spell checked "blog sci". So I get matches for "…blog: sci-fi…". See what happened there?

Another example: I run a website with a domain name of XY.com where X is a word and Y is another word. My Alert is set to match it exactly as [XY]. This was going well until recently when I started getting alerts that match [X Y].

Another example: I have an alert for [cli.gs], my latest web app. I get a lot of spurious alerts for this because it matches [cli gs] which is a very popular combination apparently.

Anyone else seeing this weirdness? Any other interpretations? Thoughts in the comments please!

The problems?

• The red circles: The suggestion that English (UK) is different from English (GB). Psst. They’re the same thing. It’s an exceptional reservation in the ISO standard.
• The black circle: The whole message is apparently not in English because the link at the bottom right corner gives me the option to view it in English. When I click it, I get the same message, but instead of suggesting English (UK), it suggests just plain old English. And oh, it gives me the option to change my language to the real English of English (US).

Hey, I have news for you YouTube: English, English (UK), English (GB) and English (US) are all freakin’ English.

I was just searching with Yahoo! and I saw a survey request from "Yahoo! Surveys". It was a big purple box to the immediate right of the results list, and it was anchored to the bottom of the screen (so even if I scrolled down, it went down too). I clicked on it before I realized I should have taken a screenshot, but I did take a screenshot of the single question in the survey. The question opens in a new window:

Click for full size, and no, I’m not going to tell you what my answer was :p

But there is a problem: SEOmoz has not disclosed the user agent (UA) of its crawler. Here I will talk about why this is a bad thing, and also take a stab and go out on a limb and say: there is no SEOmoz crawler, at least not in the traditional sense. For the latter, I will offer a viable technical alternative, which may or not be correct, but the fact the alternative exists gives a sensible explanation as to why SEOmoz is not offering a straight answer to the UA question.

Why Disclosing the UA is Essential

Let’s not mince words: we as an SEO community like a little mud fight once in a while. We debate and discuss and yes fight. But one thing we all know how to recognize is malicious activity and differentiate it from aggressive activity.

Example: a bot scraping our content for an MFA site is a tolerated nusance. We take steps to negate the effects of scrapers but at the end of the day we don’t fight them hard. On the other hand, a bot probing for security holes is treated like a witch in 1209AD.

Which is why the Linkscape’s lack of disclosure hurts: We as a community work hard at identifiying bots. SEOmoz is supposed to be a good citizen of the SEO world, and yet the lack of transparency goes against the spirit and the image of SEOmoz. On the one hand we have a company with a strong community doing good deeds (SEO trademark fight anyone?) and yet it behaves in a way we expect out of the shady side of the net we deal with every day.

Not just that: the data collected from us, about us, will be used against us. It’s called competitive intelligence.

And not just that: SEOmoz is using the data to make money. The free version is pathetic and the Pro version needs a monthly subscription.

To me, this kind of behavior (stealth, harmful, and to make money) puts Linkscape squarely in the naughty corner. I certainly didn’t expect this out of SEOmoz. Tough luck Rand and co: you have a great brand and I for one expect better!

But I won’t ask for a UA because I think there isn’t one.

How To Build Linkscape

It’s actually quite easy on a conceptual level. However, just like cooking, having a recipe doesn’t make you a great chef – there are lots of details that SEOmoz must have tackled successfully to build Linkscape. I am not trying to belittle their achievment, and all I can show you is one recipe. This recipe is completely my guess and could very well be wrong. I have not talked to anyone at SEOmoz.

So come on Pierre, what is it? The answer is the Yahoo! Search API. It’s an API giving programmers complete access over the Yahoo! index without crawling to a single page. For example, the following URL:

fetches the first two hits from a Yahoo! [site:seomoz.org]. Interestingly, it tells you where the cache URLs are, and they reside on Yahoo! servers (unsurprisingly). So you fetch the cache from Yahoo!, do the analysis, save what you care about (links, titles, etc), and you’re done.

You’ll need to kick start this somehow with a seed set of sites. DMOZ and Wikipedia are usually good sources that are freely available. Wikipedia can even be downloaded so no one needs to know. Yahoo!’s very own Delicious, Digg, reddit, etc are also good starting points because they tell you what’s hot right now. The seed is basically a huge set of URLs from which you extract the domain names and do [site:domain] queries. Lather, rinse, repeat.

Notice that you won’t need to crawl a single page yourself. You let Yahoo! do the work for you. Neat, no?

So What Should SEOmoz Disclose?

Above I said two potentially conflicting things: SEOmoz should disclose the Linkscape user agent and then went on to show that it doesn’t need to have a user agent. So what exactly am I asking from SEOmoz?

Easy: complete disclosure. If SEOmoz is using a traditional crawler, we must have its UA and the IP addresses. It’s only a matter of time for us to find them. If not, SEOmoz needs to explain clearly why not.

What kind of analytics do you get? At launch right now:

• Cligs gives you tons of traffic data and analytics about the traffic your short URLs get. This includes:
  • Number of hits
  • Referral stats
  • Mentions on twitter, blogs, and the web
  • Mentions of the destination URL on twitter, blogs, the web, and delicious

  And lots more! And if you want a more data, just let me know!

• Cligs forwards with a 301 Permanent Redirect so your destination URL gets full SEO benefits of the link. If you are an affiliate marketer, this means you can hide your backlinks, get traffic, get statistics, and get the SEO benefits.
• With Cligs, you can create an unlimited number of short URLs for the same destination URL. This is great because you can promote the same destination at different sites like twitter or facebook by using different cligs and watch how each source sends you traffic.

That’s just the start. There are a ton of new features that are going to be added in the coming few days and weeks, including some SEO-useful analytics.

And, of course, there is a bookmarklet:

Shorten Link @ Cli.gs

So what are you waiting for? Stop using plain-vanilla short URL services and start using Cligs.

Comments and feedback most welcome.

Notice a few interesting details: No user-agent string, the fact it provides an HTTP_REFERER header that’s the same page being requested, it comes from *.yahoo.com not the usual yahoo.net for Slurp, and the fact it says "image" and "srch" in the host.

The tracking is very low-level, a few hits a day with lots of one-hit-a-day visits.

What’s really interesting is how laser-targeted it is: it’s only requested the same two pages many times since May. The pages are the specific blog post linked to above plus the archive page that contains that post, so it’s likely something about that post that’s of interest to the bot. And yes, the post contains an image, and the image is the only one in the main content of the archive.

I’ll dig deeper when I have a chance. Please let me know in the comments below if you’re seeing something similar.