Late last week I found myself watching a documentary on organized crime in New York City. Focused on the Gambino crime family, this documentary illustrated how the mafia (which included the Gambino and other families that made up the Cosa Nostra) had their hands in on nearly every facet of city life. From restaurants and delivery trucks to corrupting the law enforcement and legal systems – the organization was in control.

Less than an even a decade ago Google was simply a search engine. A player in the growing Internet that consumers and the public were excited to use and would eagerly embrace in their everyday lives. In the years following their cult like launch however, a blend of strategy and brute force has catapulted them to become what I now see as the Internet’s first organized crime family.

I must admit that drawing a comparison like this is a bit of a reach without conducting any research. So that’s what I started to do… And in the past few days I’ve learned that while not quite “criminal” in their everyday actions, Google’s dominance and reach has resulted in their becoming a monolithic organization. Precisely, I’m sure, as planned.

The Foundations of Activity
For the mafia to operate to its full potential there needs to be structure and reach that allows all activity to be seen, heard, felt and eventually capitalized on. Most of what the public knows about true organized crime is that these organization penetrated into labor unions, law enforcement agencies and massive commercial organizations. Unbeknownst to them, the public ends up living life feeding these organizations. Whether it be through paying higher prices or casting favorable public opinion behind devious characters – much of what organized crime controls is done so with a level of transparency that can never be rivaled.

Having worked in the search engine marketing industry, I’m more aware of what Google sees than most others would. Most of my blog’s readers know as much or more about Google’s reach than I do – but the public still remains unaware.

Google’s Bread and Butter
Let’s take a deeper look at the types of information that Google has access to, starting with their core competency – a simple search query on Google.com.

Google Account holders and Google Toolbar users are enrolled in something called Web History. Web History allows Google to track what you search for, and more importantly – what you click through to. Google’s willingness to track what you’re searching for and viewing online is masked at a usability feature. In fact, you can log into Web History and see any of the following types of Google Search results that you’ve clicked through to, including:

• Web
• Images
• News
• Products
• Sponsored Links
• Video
• Maps
• Blogs
• Books

Now, let’s be clear about a few things. Google Account holders have an ability to view their Web History and look at some of the information that Google is recording. Should we be naive and believe that Google’s being completely transparent here?

I vote no. And I hope with some more careful consideration – you may agree.

Google’s Penetrating Reach
Google is no longer simply a search engine. They may have started that way, but have since penetrated deep into the everyday lives of billions of people worldwide… They’ve done that not by forcing themselves on you – but acquiring or fueling the sites, services and networks the public relies on most frequently.

While I don’t like relying on Wikipedia for too much, I’d ask that you take a look of the List of acquisitions by Google article that can be found there. Having acquired more than 50 companies, Google now has the ability to control, monitor and view user activities across an overwhelming portfolio of products and services.

Here’s a comprehensive list I’ve built of known product and service sites. While I’m sure that I’m missing some from my list, the key here is the massive scale and reach that Google now has.

Google’s Standalone Applications
• AdWords Editor
• Chrome
• Desktop
• Earth
• Gmail/Google Notifier
• GoogleTalk
• Pack
• Photos Screensaver
• Picasa
• Picasa Web Albums Uploader
• Secure Access
• SketchUp
• Visigami
• Web Accelerator

Google Application & Desktop Extensions
• Blogger Web Comments
• Dashboard Widgets for Mac OS X
• Google Gears
• Send to Mobile
• Google Toolbar

Google’s Mobile Products & Services
• Blogger Mobile
• Calendar
• Gmail
• News
• Google Mobilizer
• iGoogle
• Maps
• Mobile Updater
• Mobile Search
• Product Search
• Reader
• Sync
• YouTube
• Picasa Web Albums
• Google Latitude

Google’s Web Based Products
• AdSense
• AdWords
• AdWords Website Optimizer
• Audio Ads
• Click-to-Call
• DoubleClick
• Grants
• TV Ads

Google’s Communication and Publishing Tools
• 3D Warehouse
• Apps
• Blogger
• Calendar
• Docs
• FeedBurner
• Friend Connect
• Gadgets
• Gmail
• iGoogle
• Notebook
• Knol
• Marratech e-Meeting
• Orkut
• Page Creator
• Picasa Web Albums
• Reader
• Sites
• Voice
• YouTube

Google’s Developer Tools
• Android
• App Engine
• Code
• Mashup Editor
• OpenSocial
• Subscribed Links
• Webmaster Tools
• Web Toolkit

Google’s Geographical and Mapping Tools
• Maps
• Map Maker
• Mars
• Moon
• Sky
• Ride Finder
• Transit

Google’s Search Based Products
• Accessible Search
• Alerts
• Base
• Blog search
• Book Search (Previously Google Print)
• Checkout
• Code Search
• Directory
• Directory (Google China)
• Experimental Search
• Finance
• Groups
• Image Labeler
• Image Search
• Language Tools
• Movies
• News
• News Archive Search
• Patent Search
• Product Search (Previously Froogle)
• Scholar
• Sets
• SMS
• Squared
• Suggest
• University Search
• U.S. Government Search
• Video
• Voice Local Search
• Web History
• Web Search

Google’s Statistical Tools
• Analytics
• Gapminder
• Trends
• Zeitgeist

Google Hardware
• Google Search Appliance
• Google Mini

Other / Miscellaneous Products
• GOOG-411
• Health

Google’s Key Players are Products and Services
Through decades of research conducted by prying journalists and investigative organizations like the FBI, we know that organized crime families have a strong hierarchy in place. This hierarchy puts the most influential and powerful people in a position to orchastrate specialists beneath them. This differs for Google in the sense that products and services take the place of people. What I mean by saying this is that Google’s brand and services represent leaders and shot callers that monitor and profit from all of our Internet based activity.

I am by no means an expert on organized crime, or Google for that matter. My years of being an Internet professional though suggest that the following products and services are the key players underneath the Google family:

• Search
• Analytics
• AdWords & AdSense
• GMail
• YouTube

These five players have market reach and dominance that is unrivaled by any other non-Google organization. The majority of Internet users rely on these five core services – and as a result, Google stands to profit.

The Profitability of Internet Dominance
What I find most disturbing about this trend though is that like criminal organizations in the past, Google is driving up the price for advertisers and consumers with the incredible control they have over all of the market variables. If you’re a web site owner, you’re likely providing Google access and permission to analyze what your visitors are doing on your site. If you have a GMail account, they’re able to parse the content of your emails and tailor advertising accordingly. If you’re an AdSense advertiser, you’re paying a premium through a veiled cost per click system that Google developed in a proprietary way to maximize their profit with. And if you’re relying on AdSense, or “Webmaster welfare” as Sugarrae has termed the program – you’re simply fueling Google’s greed further while minimizing your own profitability.

When Google first went public, their IPO priced each share of the organization’s stock at $85. At one point in late 2007, GOOG shares traded at $711.25 – an increase of more than 736% over the IPO’s value.

This cult like following, both in technology sectors as well as in public appeal forced Google to not only become a household name, but even a verb. To this day you can go practically anywhere and overhear someone having “Googled” something, or even worse, be willing to “GMail” you something. Which leads me to…

The Public Appeal and Fascination
Any staunch Google supporter already thinks I’m both crazy and stupid for suggesting that Google’s operation mirrors that of a criminal organization. Keeping with the comparison of Google’s brands mirroring various mafioso of the past, I’m now going to go out on a limb and state that to me, Google is the modern day Teflon Don – John Gotti.

John Gotti’s following came through two methods. First, he struck fear in the eyes of the opposition and second, he served as public figure for people and the media to fall in love with.

CNN once wrote that Gotti murdered his way to the top of the Gambino crime family. I think Google did the same exact thing – and here’s why.

Back in the days before the “big three” search engines, Google results were so accurate that they were shown to anyone conducting searches on major online properties like Yahoo!, AOL and dozens of others. While most simply appreciated the accuracy of clean and relevant search results, Google was deviating from their “do no evil” mantra that many foolishly believe exists today.

One by one, Google effectively sniped the competition either by becoming a provider of materials essential for business (as was the case with the search results I referenced above) or, by simply buying out entire organizations and overhauling or eliminating them. Since their creation, Google has acquired more than fifty leading Internet based organizations… These acquisitions were entirely strategic and have gone on to help feed the Google family of operations. Some of the properties simply stood in their way and were later shut down. Others, often commanding billion dollar buyouts like YouTube and DoubleClick primed Google to reach the majority of Internet users across all demographics and gain insight into every little nuance of their online behavior.

While Google was either eliminating the competition or forcing them to rely on Google’s materials to survive, they placed an emphasis on winning public opinion. John Gotti was famous and untouchable in his Queens neighborhood because he often threw parties, picnics and gave handouts to those that surrounded him. Google is no different, and as an example of that – I’d simply point to GMail being the leading free email service or AdSense and Google Analytics as both free and market leading tools for webmasters to run an online business.

Comparing Google’s Services to Criminal Activities
It was suggested to me that Google’s services actually mirror a number of ways in which a criminal family or organization would work. Google’s AdSense program for example was compared to prostitution, their free applications likened to the distribution of drugs, and Google AdWords being similar to Google’s offering of “protection” for web site owners. I can certainly see and understand these comparisons but am interested too in how readers of this post will react to those comparisons being drawn… So please do leave comments to further the discussion.

Thoughts from Other Search Marketers
I tend to be anti-Google when it comes to my opinion, so I asked a few contacts from the search marketing industry for their thoughts and opinions on my comparing Google to an organized crime family. Here’s some of the feedback I got that was worth sharing:

David Wallace (Twitter: @davidwallace) offered the following:

“It’s apparent that Google wants to get into everything, at least wherever they can stick AdWords. For the company that has made statements to the fact that “search is core to them” they have gone a long way from search… Google Docs, GMail, Calendar, Talk… Shall I go on? http://www.google.com/intl/en/options/ clearly shows that although the Google Search page remains simple, the company is involved in tons of stuff and upsetting many business models…In fact I’m just waiting for the day that we will hear the announcement, “Google acquires Wal-Mart.”

Ben Cook (Twitter: @Skitzzo) had this to say:

“Follow their rules or else, they pay you but can up their take without notice and you get away with more if you produce for them… That’s Google for you. Mobsters have to stay under the radar & deal with cops & other families. Google answers to no one.”

Greg Boser (@GregBoser), while not offering any direct comments commended the comparison and said he looked forward to reading the post when it was completed. Social media specialist Brent Csutoras (@BrentCsutoras) and optimization expert Andrew Gerhart (@andrewgerhart) both thought that Google is more like a government than a crime family.

Chris Hooley (@ChrisHooley) later added:

“Well, a crime family works under the law. Google works above the law. If you are going to compare them to a corrupt organization I would compare them to a corrupt government who puts the koolaid in the water supply.”

Michelle Robbins (@MichelleRobbins) provided a lot of assistance in creating this article. She offered up an editorial review and some recommendations, as well as this quote:

“Even though, in effect, Google has a stranglehold on internet advertising and marketing options – much like a crime syndicate can control and dictate how businesses operate and do or do not succeed in a given location or market – there is one key difference. The mob is self aware. They don’t pretend to be anything other than what they are.”

Please Help Further The Conversation…
Anytime someone takes interest in making such a harsh comparison, strong opinions will be formed. I am not saying that I am right or wrong for drawing or writing of the comparisons made. The above is simply my opinion – and I want to hear more from you, the reader. Please help the discussion continue through the commenting feature below.

Photo Credits: Header Image “Anybody There” by mooranguinho, licensed for use through Creative Commons. In Post image “Yes? Hello?” by rossinabossio also licensed for use through Creative Commons.

No related posts.

My plans for 2009 continue to become more exciting. I just got word back from Ross Dunn, and the folks at WebmasterRadio.FM have made it official… I’ll be Ross’ new co-host on the SEO 101 program that airs each Monday at 5:00PM ET.

What is SEO 101?
SEO 101 is weekly program that speaks to Search Engine Optimization, or “SEO” from the very beginning. The goal of the SEO 101 program will be to interact and teach listeners about the many aspects of SEO by sharing helpful information without overwhelming them with technical details.

Joining Up with Ross Dunn and WebmasterRadio.FM
I met Ross back in February of 2007 at SMX West and he’s a great guy, great SEO and a wonderful father. If you aren’t already keeping tabs on what Ross is doing – please connect with him on Twitter and check out his company, StepForth Web Marketing Inc..

Here’s a photo of [L to R] Darrell Long, myself, Ross and David Wallace at Bennigan’s last February in Santa Clara. [Photo Credit to David Wallace]

Ross and I have obviously been pretty excited about this, and here’s his take on the development:

SEO 101 on WebmasterRadio.fm has been a big success since Jennifer Evans Laycock and I took it on and revived it but when Jennifer left due to time constraints I needed someone to fill some big shoes. Thankfully I have been around long enough to know just how good a guy and SEO Eric Lander is. So when he agreed to be my co-host on the show it felt great to know the show was once again whole. Now it is time for us to plan out some great shows!!

To compound on that, I also want to thank Jennifer for her contributions to the show (and our industry) and wish her the best with all that she has going on. We’ll certainly try to steal her back and get her back on the air for our listeners, too. :)

About Webmaster Radio
Founded in November, 2004 – WebmasterRadio.FM is a free, 24/7, interactive Internet Radio Network focused on the B2B digital marketing world. We have burst on to the Internet Media scene as a community destination with 100% original programming, live broadcasts, archives, and podcast shows. We are proud to boast that our shows are hosted by the most respected names in the business world!

WebmasterRadio.FM Links of Interest:
• WebmasterRadio.FM Website
• SEO 101 Program Details
• Searchbash
• Advertising on WebmasterRadio.FM
• WebmasterRadio.FM’s Blog
• WebmasterRadio.FM on Twitter
• WebasterRadio.FM on Facebook

No related posts.

First, let’s talk about the new layout.

Since I’ve been using it a few days – I actually have forgotten what the old one looks like. I know the size and contents of the sidebar have obviously changed to include trending topics as well as a new replies type option. The @username link now appears where “Replies” once did and it’s now built out to provide you with search results of all instances of your username being tweeted. Now you’ll be seeing when you’re being retweeted or talked about as opposed to just being shown all updates that begin with @username.

Here’s a screenshot:

Click to Enlarge / See Full Screen

Save this Search
This morning I noticed the introduction of the “Save this Search” link as part of the new design. Here’s an “easy as 1, 2, 3…” type graphic to illustrate how it works.

In short, you enter your search term or phrase on the right hand side and either press enter or click the magnifying glass. From there, you’re page won’t refresh – only the tweets appearing under your update box will. This is great if you’re working on a tweet but have to check a couple of things without losing the contents of what you’ve already written up in the form.

Once you like the search results, you can also click on item #2 as I’ve circled – which is the “Save this Search” hyperlink. This link will now allow you to always access that search by adding it permanently in your right sidebar (item #3!). Really easy and intuitive. To remove a search, just load up the search results you no longer care for and in place of the “Save this Search” link you should now see a “Remove this saved search”.

Really easy functionality, but again – tremendously useful. Here’s another screenshot:

Click to Enlarge / See Full Screen

Thoughts?
I’m curious to know what others are thinking about this. I don’t like change so when I saw the new sidebar I wasn’t a fan. The new Save this Search link though seems to be handy and should save me some time loading up TweetDeck or Twitter Search all the time. Got an opinion? Please share it in the comments below!

No related posts.

My goal here is to encourage you to the point of using log files on a regular basis to better tune up your hosting and marketing platforms.

Page Tagging Vs. Log Based Analytics

When you think about web analytics, what do you really think of? The majority of people I know have told me all about their nicely designed dashboards filled with telling graphs, growth charts and tables that just wait online for them to view them whenever needed. While that’s all well and good – those reports are always coming out of page tagging analytic programs. The problem? Page tagging analytics has limitations, and some of those limitations are simply unacceptable for hardened SEOs and webmasters.

That’s not a knock on page tagging analytics either. Page tagging is a popular method of acquiring data as the ease of use and on demand availability of reporting all add up to make these tools a required resource. Log files for me simply help me go that extra mile.

I want you to guess how many of the following analytic suites are providing you with reports and data generated from, in part, log files:

• Google Analytics
• Omniture
• Microsoft adCenter Analytics
• Sitemeter
• Quantcast
• Compete
• HitWise

Ready for the obvious answer?
– Zero! None of the above use data recorded by your server for statistical analysis.

Olivier Amar of CompuCall earned some kudos this morning. When I asked how many followers were not out there checking their logs – he tweeted a reply about ClickTracks – one of the few analytic suites out there for SEOs and site owners that actually integrates log files out of the box.

I don’t want to get into a whole lecture about the differences between page tagging analytics and log parsers (or hybrid solutions for that matter). What I do want you to realize is that no matter the hosting platform, there is some more useful information you could be extracting about your web site and your visitors if you can acquire the logs.

Familiarize Yourself with Log Files

Before we jump too far in it’s probably best for us to review what a server log file is, what it looks like, what data it contains, etc.

What is a Server Log File?
Wikipedia defines a server log as:

A server log is a log file (or several files) automatically created and maintained by a server of activity performed by it.

A typical example is a web server log which maintains a history of page requests. The W3C maintains a standard format[1] for web server log files, but other proprietary formats exist. More recent entries are typically appended to the end of the file. Information about the request, including client IP address, request date/time, page requested, HTTP code, bytes served, user agent, and referer are typically added. These data can be combined into a single file, or separated into distinct logs, such as an access log, error log, or referrer log.

How Do I Retrieve Server Logs?
Each hosting provider or company handles this differently. My hosting company makes it easy for me by keeping logs available via FTP on a 7 day cycle before any logs are removed. I have adapted to just pull those logs down off my server oce a week through an automated application. Set it up once, and now I can forget about the hassle.

I’ve seen other hosts that make log acquisition more… Trying. In any event, server logs have a number of different recording options, structures, formats and file types. This post is focused on using the logs you have available to you – not acquiring them. I highly recommend working with your server administrator or hosting provider to acquire access to logs if you do not have that already.

If you are ever presented with an option – push to acquire Extended Log Files and then quickly hand your hosting provider or server admin a copy of this resource from the W3C.

What Does a Server Log Look Like?
Here are five lines I pulled out of a server log file from my blog as recorded yesterday, March 29, 2009:

85.89.185.215 – - [29/Mar/2009:01:00:09 -0700] “GET /wp-content/uploads/2007/09/100cap006.jpg HTTP/1.1″ 200 46012 “http://www.ironworksforum.com/forum/showthread.php?p=1200167″ “Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/530.1 (KHTML, like Gecko) Chrome/2.0.169.1 Safari/530.1″

38.99.107.141 – - [29/Mar/2009:01:00:10 -0700] “GET /feed HTTP/1.1″ 302 5 “-” “Mozilla/5.0 (compatible; FriendFeedBot/0.1; +Http://friendfeed.com/about/bot)”

193.252.149.15 – - [29/Mar/2009:01:07:24 -0700] “GET /276.html HTTP/1.1″ 200 24655 “-” “Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)”

69.147.112.169 – - [29/Mar/2009:01:11:01 -0700] “GET /feed/rss HTTP/1.0″ 302 0 “-” “Yahoo Pipes 1.0″

66.249.72.136 – - [29/Mar/2009:01:12:28 -0700] “GET /robots.txt HTTP/1.1″ 200 508 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”

In a 12 minute span on my web site, some pretty cool things happened that I would never have known about through any page tagging analytic suites I’m using.

Let’s dissect each of these five lines and I’ll show you what I mean.

Entry #1

85.89.185.215 – - [29/Mar/2009:01:00:09 -0700] “GET /wp-content/uploads/2007/09/100cap006.jpg HTTP/1.1″ 200 46012 “http://www.ironworksforum.com/forum/showthread.php?p=1200167″ “Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/530.1 (KHTML, like Gecko) Chrome/2.0.169.1 Safari/530.1″

Someone is stealing my images! Like a lazy webmaster, I don’t lock down much and prevent other people from using it. In this referenced log file I now have evidence of someone using one of my images (regularly, I might add) on another web site’s discussion board.

This log file entry tells me that this person is using this image on this discussion thread.

Not cool! Now for me, bandwidth isn’t much of an issue and I don’t really mind if someone is repurposing that image. If that were protected photography though – I’d want to keep it under lock and key. More on this later.

The key here though is that the actual “page” being loaded up (the discussion board thread or user profile page) is hosted elsewhere. Since I don’t own that site, I don’t have Google Analytics code on the site and without this log file, I never would have known that this was taking place.

When you consider how much this could happen with a large web site – you can probably see how quicly this can become a big issue.

Entry #2
The next log file entry was this:

38.99.107.141 – - [29/Mar/2009:01:00:10 -0700] “GET /feed HTTP/1.1″ 302 5 “-” “Mozilla/5.0 (compatible; FriendFeedBot/0.1; +Http://friendfeed.com/about/bot)”

As the tail end may suggest to you, this is a FriendFeed bot that’s coming through my web site and pulling a copy of my blog’s feed. Friendfeed’s bot will then see if there’s any new entries and pull them via RSS to use on their own site since I’ve allowed them to do so.

If you’re watching things like page views, this wouldn’t actually count in other analytics since again – the user requesting the data never actually came to my web site. The other issue? The “user” here is actually a bot and my guess is that if it’s like GoogleBot, it probably won’t bother to execute any javascript code that would be required for page tagging analytics to record the hit.

Entry #3
Next up:

193.252.149.15 – - [29/Mar/2009:01:07:24 -0700] “GET /276.html HTTP/1.1″ 200 24655 “-” “Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)”

An old post on my blog on the Internet Marketer’s Charity Party at SES San Jose is being retrieved here by another bot, this time called VoilaBot. Ever heard of VoilaBot before? Sadly, I had not – which is more telling about my failures as an International SEO.

Voila is the provider for Wanadoo, which is a huge portal in France and one of the biggest european ISPs.
Voila itself is one of the best known web brands in France.

Where’d I get that information? From heini, a veteran user on Brett Tabke’s WebmasterWorld, silly.

Entry #4
Still with me? Good, because we’re going to go easy on these last two entries to review! Next is…

69.147.112.169 – - [29/Mar/2009:01:11:01 -0700] “GET /feed/rss HTTP/1.0″ 302 0 “-” “Yahoo Pipes 1.0″

This is the footprint of Yahoo! Pipes, a fairly new RSS / News Aggregator that’s actually quite cool. All that was happening here is that a user of the Pipes program was loading up (or refreshing) my blog’s RSS feed. Again – this would never show up in anything like Google Analytics or Omniture. Why not? You know this. Just read the last three log dissections. :)

Entry #5
And finally… The staple of any SEOs diet… Googlebot!

66.249.72.136 – - [29/Mar/2009:01:12:28 -0700] “GET /robots.txt HTTP/1.1″ 200 508 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”

A well behaved Googlebot, too! The above request is the mark of GOOG coming through and requesting my blog’s robots.txt file for some more direction. It’s always nice when bots do what they say and are supposed to do first, right?

Now, Onto YOUR Log Files…

You don’t really care about what’s happening here on my blog — you want to see what’s going on with your web site. So now we get to take a look at how to make these log files work for you!

What You’ll Need
1.) Server Log Files
2.) Server Log Parsing Application
3.) Curiosity

Again, I’m not helping you with item number one.

With item number two, I’d recommend WebLog Expert. It’s an application that I’ve been using for years and bought the professional version of some time ago. Considering the low cost, I’d recommend it – but there are certainly other log file analyzers available to you.

Just check out download.com or directory listings on the Yahoo! Directory or on DMOZ.

Since WebLog Expert offers a free BETA version with some filtering options though, I’ll use them for screen shots.

Here are some report ideas I’m going to demonstrate for you…

1. Google, Yahoo & LiveSearch Spidering
2. Stolen Content
3. 400 Errors
4. 300 Server Redirections

For the purpose of this demonstration I’m going to use WebLog Expert as the log file analyzer because it’s a free solution and provides some easy filtering options. The key here is in using these filters to look at very specific data.

Google, Yahoo & LiveSearch Spidering

Log files record the user agent of each request. When a human visitor visits your site, their web browser will be recorded as it is labeled. Refer back to log entry #5 from above to see how Googlebot identifies itself. It tells my web server that it’s user agent is Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html).

In order to report on the spiders, you need to set up a filter that excludes all activity outside of the spiders. I’m going to take this one step further and show you how to set up filters in WebLog Expert that only pull activity on the big three – Google, Yahoo and LiveSearch.

Using the filters dialogue, you will need to add a new filter that includes activity based on spider name, and then select each of the appropriate spiders from the drop down list. To do this, you’ll need to set up three filters like so:

Once you set up three filters, one for each, you should see this:

And if you do, just click through the Finish button and then run you’re report. You’ll now get a wealth of information on your spidering activity.

Want to see what data is available? Click here to download the resulting report in PDF format. Here’s a hint to what you may find out… Yahoo! Slurp is sometimes a little more… aggressive than you may think:

Stolen Content

This report, even though I’m using it as an example for this post – really is one I need to take action on. Our goal here is just to find anyone out there who may be using my CSS or images on their own sites or for their own needs. If you were to run this same report, I’d suggest that as an action step – you take measure to prevent images from being hotlinked and so on.

With WLE, you’ll want to create the following filters:

Replace ericlander.com with your own domain, and, add or subtract any files you’d like to see in there. Other popular files to be stolen and reused? mp3, pdf, swf, avi, mpg, mov, and css lead the way for me.

Again, a sample report output of the above can be found here in PDF format.

400 Errors

One of the most useful reports for me over the years has been this report that only looks at 400-type responses. Now, any 400 error from your server indicates that something hasn’t been found. The most popular of which is the 404 error we’re all used to seeing – but there are other useful not found errors to note, including the following table from HTML Goodies:

• 400 : There is a syntax error in the request. It is denied.
• 401 : The header in your request did not contain the correct authorization codes. You don’t get to see what you requested.
• 402 : Payment is required. Don’t worry about this one. It’s not in use yet.
• 403 : You are forbidden to see the document you requested. It can also mean that the server doesn’t have the ability to show you what you want to see.
• 404 : Document not found. The page you want is not on the server nor has it ever been on the server. Most likely you have misspelled the title or used an incorrect capitalization pattern in the URL.
• 405 : The method you are using to access the file is not allowed.
• 406 : The page you are requesting exists but you cannot see it because your own system doesn’t understand the format the page is configured for.
• 407 : The request must be authorized before it can take place.
• 408 : The request timed out. For some reason the server took too much time processing your request. Net congestion is the most likely reason.
• 409 : Conflict. Too many people wanted the same file at the same time. It glutted the server. Try again.
• 410 : The page use to be there, but now it’s gone.
• 411 : Your request is missing a Content-Length header.
• 412 : The page you requested has some sort of pre-condition set up. That means that If something is a certain way, you can have the page. If you get a 412, that condition was not met. Oops.
• 413 : Too big. What you requested is just too big to process.
• 414 : The URL you entered is too long. Really. Too long.
• 415 : The page is an unsupported media type, like a proprietary file made specifically for a certain program…

The filter setup here is super simple. Just create this one:

And the resulting report looks like this (again, PDF!).

300 Redirections

Every SEO needs to have a grasp of 301 redirects, and reporting on the ones your server dishes out is super simple here. Just like the 400-responses, you’ll need to set up a quick filter that only pulls 300-level response codes. Easy!

The value here for an SEO is pretty obvious – so I’ll let you run with why this report is useful. To check out the sample in PDF format, just click here.

Wrapping Up…

Hopefully this post has given you some more insight on how you can begin analyzing server log files. If I’ve confused you at any point, please do drop a comment below and open up a discussion for us as others may have similar questions or hangups.

Don’t be afraid to get creative with the use of filters too with WebLog Expert or any other application that you may find yourself using. It’s very easy to use filters to extract in depth metrics like time spent on site by visitors viewing movies, path of visits referred from Digg, bounce rate for StumbleUpon referrals, etc.

Finally, this isn’t meant as a knock on page tagging analytics and the information they offer. Every successful web site marketer should rely on both regularly – but when it comes to running a clean site, don’t just assume the logs have nothing to provide to you.

No related posts.

While I was certainly happy to see search marketing involved, I don’t much agree with Kim’s suggestion:

Use local paid search. When your prospects search online, who will they find first, you or your competitors? The best way to guarantee your company appearing at or near the top of search results is through pay-per-click advertising. Being at the top of the page is critical to getting noticed, as most searchers rarely go past the first page of results. And the good news is that local paid search is often quite affordable and is readily available through Google, Yahoo and other search engines.

Paid search is an affordable option for those looking to drum up business. If you’re on a tight budget though, why not look at spending no money to capture that same audience?

I often fear that too many all in one marketing types simply suggest paid search because it’s effective before giving other forms of search a solid look. As enhanced listings mature and search engines’ users adapt their patterns to use these universal results – more free opportunities for listings appear.

If you were to perform a search on a major engine for any keyword phrase that combines geographical modifiers alongside a product or service name – you’re likely to see something like the Google OneBox results. It’s that familiar map with nice red icons that plot the nearest vendors that provide exactly what you’re after.

Best of all – Few businesses are out there taking the time to claim and enhance these free listings. That leaves you with some great opportunities to to clean up without spending a dime on search referrals.

My other issue with the author’s suggestion of local paid search is that she emphasizes reaching your targeted audience. Going into any paid search marketing campaign blindly will cost you far more than necessary. Paid search campaigns are optimized over time by evaluating clickthroughs, conversion rates and so on. You can’t afford the luxury of buying that information when you’re truly on a tigh budget.

Now, to the author’s credit, advertising in Google’s Local search results is certainly affordable. In my experience the analytical data for these campaigns suggest that having a unique icon on the map along with a more enhanced business profile will instigate more productive clickthroughs.

My last gripe is that no one reading the referenced article would ever know that such a search advertising product even exists. Furthermore, even basic local listings can be pushed offline and result in your customers generating favorable reviews, relying on Google for coupons, and ultimately generating more free business opportunities for you.

I’ve already gone through how to make the most of your local search profiles on Google in an old article on Search Engine Journal. If you’re interested, please check that out along with David Mihm’s local search ranking factors research.

I’m not against renting space. I just don’t understand why you’d want to rent space you could own for free.

No related posts.

While away from blogging I have become quite reliant on using my Twitter account [@ericlander] for keeping up with news, trends and information in the search engine marketing industry. My interaction there is apparently paying off too, as I’ve been gaining lots of new followers. I was surprised to see this when I checked in on my TwitterCounter stats:

My 3-Month Twitter Stats as of March 29, 2009

Now, I know the blog is bare. I’m running a barebones version of Thesis while I work on customizing things to the point that I’m happy with it. There’s no need in making it look all ugly here while that takes place, so pardon the dust.

In the spirit of giving readers what they want – I need to know what you’d like to hear about here on my blog. Please take a moment to cast your vote on which topics you’d like to read more about. Ready? Go!

[poll id="1"]

No related posts.

Ironically ADP gets a good plug here – as they did throughout the conference materials. Too bad they didn’t see the trip as being important enough to cover. I paid my way and was forced to also use my vacation time.

Oh, and hat tip to Oilman for posting this on his blog ages ago. Since he and I are a lot alike (postings once per millennium) you should thank him. Not me. :)

No related posts.

Why? As the title suggests – I care more about the people in my network than I do the number of connections I have. Specifically, I care about ethics, personal accountability… and honesty.

Let’s dive in.

Ethics in Search Engine Marketing

The subject of ethical business is less important in house painting than it is search engine marketing. Your house has either been painted or it hasn’t. From there, it either looks good or it doesn’t and the job could have been done a lot better. The bonus here is that the average Joe knows when their house has actually been painted.

Search engine marketing, as an industry though, will always be open for interpretation. SEO will, for some small group even fifty years from now, conjure up images of snake oil salesmen and get rich quick with AdWords personalities. It sucks, but it’s true. And it’s true because legitimate SEOs are both artists and scientists.

Enter again, the topic of ethics.

The underlying problem here for me is that the search engines are forced to govern themselves and do so in a way that leaves the artists effectively blind. Home builders don’t build houses and finish them with a polished metallic exterior just so they can sit back and laugh at a painter who tries to make it look good.

Search engine marketers have two options. They can either work with their clients to make web sites better and force a true business relationship to be formed. Or, they can opt for the quick buck and be content screwing someone over with some regurgitated piece of “best practices”.

Remember my LinkedIn Network? I’m not going to mention names, but I will say this. I removed some of the 53 because of these common ethical problems:

• Lack of knowledge
• Inability to prove their effectiveness
• Reluctance to share transparent analytics
• Consistent outsourcing of projects only to mark them up

Personal Accountability and Search Engine Marketing

This topic has become more important to me now that I’m on my own as an independent consultant. While employed at ADP, I wouldn’t mind carrying dead weight for the good of the team. Similarly, I wanted to show support for those I couldn’t vouch for because they shared the same team as I did. If this were a sport team, it’d be like routing for someone because of the laundry they happen to wear.

After all, if I can’t stand for my own teammate – what am I saying about myself?

I’m over that.

I’ll connect with people who’s work I can’t see the impact of directly. To not do that would be stupid for me and arguably anyone on LinkedIn. There’s a level of accountability that I want to hold my connections to. That accountability starts with a willingness to actually work for the good of the client.

Here are some accountability related reasons people once in my network may have gotten trimmed:

• Possessing a typical salesmen mentality
• Laziness /Unmotivated / Unwilling to Learn
• Ignorant
• Plagiarism
• Refusal to acknowledge failure

The Importance of Honesty

In case you hadn’t noticed yet… Each argument for trimming my connections piggybacks off the previous. No surprise then to see honesty, right?

For me – honesty is critical. That’s because the people in my network are a reflection of me. Likewise, I hope that they have added me because they genuinely believe in a value that I have either provided or proven to them.

I found that my network was littered with folks who simply weren’t honest. That could be a dishonesty towards me, towards clientele or simply to thmselves. As a consultant now in the SEO field, I need a concise network of people I can trust.

Maybe my network will continue to build over time. I hope so, but some of the removed individuals failed me in ways like:

• Intentionally providing inaccurate data
• Wrongfully sharing privileged or protected industry information
• Placing clients or projects at risk for personal gain
• Claiming to have been more active than they were
• Making excuses for their lack of effort

The Twitter Feedback

When I found myself compelled enough to actually remove some LinkedIn Connections, I twittered about it and invited the replies…

@ericlander: Just *removed* 53 connections from LinkedIn. Now that I’m going solo, the quality of my network matters more than bullshit numbers.

kid_disco kicked off replies with…
I don’t see how keeping those 53 connections would’ve hurt…?

@tonyadam replied in support…
Quality of connections and followers always matters over quantity! Well done man! :)

@debramastaler disagreed…
on LinkedIn? Hmmm…. quantity

@monicawright saw both sides with her reply…
both play a factor, but I’d say quality. I’m skeptical of those salesy types who have a ton of connections.

While @chris_hart offered advice to both ends…
quality for the trusted opinion and quantity so you can sample and verify

@cyandle then backed up my thoughts…
quality; so what u have 1k connections that barely know ur name; having quality ppl know who u r & what u’re about is important!

And the last reply cam in from @kgamache…
I think it is a combination of building quality connections with a lot of people to build a stable work base. My opinion.

Some Closing Thoughts…
Not everyone is in a position where they care about their LinkedIn Network. Similarly, some really don’t see any value at all other than watching their numbers and form-letter inspired recommendations increase. I’m taking a different approach with my network. I’m opting for quality over quantity.

I look at it this way… If you were to mention anyone on my network and then ask me for an off the cuff recap or recommendation of them – I’d want to be able to do it.

<rant>I don’t need to be “connected” with that wannabe SEO who can’t own up to making mistakes, can’t find an original thought, can’t develop an actual strategy or even begin to explain a damned thing about why he does what he says he does.

So here’s the deal… Having 500+ followers does not make you successful. Befriending everyone in every little pretend SEO “group” on won’t do it either. A number next to the people’s names who don’t and won’t ever know you can never represent a quality of work you do not truly represent. And, it sure as hell doesn’t mean you’re networked.</rant>

No related posts.

And I’m sure it showed.

I followed up that appearance with more SMX panel at SMX East while covering SMX Advanced as a press member for Search Engine Journal. I spoke at a couple local meetups and finally Affiliate Summit West in January, 2009…

But it’s time to head back to Santa Clara and do it all up again.

So I’m doing what anyone would logically do. I’m hopping on a panel with Jessican Bowman and Laura Lippay. While I mean no disrespect to Anthony Long – I somehow feel like he and I are bout to get pwned by two of the most awesome women in the search biz.

So stoked for that. And I’ve loved talking with Laura and Anthony on the phone recently. I think we’re going to hav a great, great panel.

So here’s the thing… Jessica has worked with Third Door Media to put together an entire day of In House SEO topics and it’ll be capping off the SMX West show and running all day on February 12th… Here’s the current lineup:

Session 1 — I’m responsible for SEO. Now what? (This is the session I’m in on…)
Once the company decides to start SEO in-house, what’s next? In this session you’ll learn what you’ll need to start up a successful in-house SEO program that runs smoothly and reels in profits. Even if you’ve been doing SEO for a while, implementing these tactics now will increase the productivity and reduce the frustrations you face on a regular basis.

Session 2 — Training the Company on SEO
In-house SEO isn’t easy, and often short staffed. In this session, you’ll learn how to bring in an SEO novice and work with the entire development team to help you execute SEO campaigns.

Session 3 — Get SEO Into All The Right Places Of The Development Life Cycle
SEO doesn’t work well unless it’s addressed in every aspect of the development life cycle. This session will show you what typically happens and what needs to happen as you build a new website and manage ongoing maintenance. You’ll walk away with a clear understanding of what you need to do to get SEO weaved into the right places of the development life cycle at your company.

Session 4 — Reporting & Scorecarding for Management
Management loves numbers. They build your business case and help get SEO high in the priority list. You’ll learn how to give a number on potential lift when management insists, which reports to deliver on a consistent basis to management, and how to use reports to help your cause when traffic growth isn’t going as expected.

Stay tuned… But if you haven’t gotten yourself registered yet for SMX West, you best do so now.

More info to come… :)

No related posts.

Today is a truly strange day.

While winning a SEMMY is fun and all, the day will forever be remembered with a feeling of loss and disappointment. Last year on this day my grandmother passed which has made me learn some very important things about myself. Here in 2009 though, I’m really feeling back on track, but I’m also losing the first real SEO I have had on my staff since 2004.

Ok… So… As many of you know,  I’ve resigned from ADP’s Dealer Services Division and the BZ Results Marketing Company. I’m out on the 6th of the month.

Today though, just two days before I leave – William Hunter Satterwhite – Or simply Hunter as well all call him, will be taking his own walk out of ADP for much bigger and better things. To say that he has the skills to be successful in this field would be a colossal understatement.

Hunter reminds me of another great friend of mine, Andrew Gerhart. Gerhart and I fought tooth and nail to understand this industry through the early 2000’s before going our seperate ways. Since then, Andrew has been phenomonally successful in all areas of life.

There’s no doubt that the same won’t happen to Hunter with where he’s heading.

So here’s the deal.

Rather than go on and on about what this guy will likely do in our lifetime to help change the online marketing industry – go connect with him. He’s on Twitter and on LinkedIn, too. Take the time to know him and appreciate his wiseguy sense of humor too.

I’ve been in search since late 1999 or early 2000. Hunter is without a hesitation of doubt, the best true SEO I’ve ever had work on my staff. Period.

No related posts.