A Quick Primer on Google Megasitelinks

Aside from having a completely uninspiring codename, here’s what we know about Google Megasitelinks:

• Megasitelinks are part of 30 search quality improvements Google worked on in December, 2011.
• Being touted as an improvement to the existing Sitelinks display algorithm.
• Said to use visitor’s location in search results to emphasize most important content in Sitelinks area.

Known Problems in Google Sitelinks

Since these new Megasitelinks are so new, there’s a lot that remains to be seen with how these conceptual changes will change what Google users see. At the core of it all, it makes sense – Google wants to improve user experience and geo-relevant information has been proven to do that in the past.

One of the most frustrating things about the existing Google Sitelinks platform has been the lack of webmaster input. In Google’s Webmaster Tools, there are certainly “controls” where a webmaster can offer up suggested suppression lists to prevent certain URLs from appearing in the Sitelinks area. In most cases though – those “suggestions” that come from a webmaster are disregarded.

No joke, I have more than five clients at this point in time who are experiencing frustration with the Sitelinks controls and the lack of influence their suggested changes actually have on the SERPs.

Naturally, this leads me to ask one thing…

Is Megasitelinks a Step Forward or Backward?

I’ve logged in to a few Google Webmaster Tools accounts this evening and have not seen any UI changes in the controls or suppression lists.

To be fair, I didn’t really expect to though. I guess you could say that I was hoping for something new to be in there for us as marketers and site owners to work with.

Here’s the thing for me. If Google Sitelinks is now a program that uses an intelligent algorithm to display a subset of pages relevant to a searcher’s query, then there needs to be functional controls in place for them – or no controls at all. The grey area where Google asks for suggestions but ignores them is an old act.

In other words, Google – please don’t waste our time.

Changes in Sitelinks We Could Actually Use

If Google is committed to making the most of the Sitelinks tools, it has to be a two way street where they focus on user experience and search quality and they allow us as webmasters to permit certain content from being displayed, hidden, etc.

Other items that would be great to see include:

• Sitelinks Analytics: Where were Sitelinks displayed? When were Sitelinks used over traditional organic listings? For what queries? Which Sitelinks have appeared in which position for which terms? (Without incredibly deep analytical reporting) We’re flying blind here, Google.
• Enhanced Geographical Support: Here’s a practical use case. A franchise has a store locator that performs well in Sitelinks now. Using the enhanced geographical inferences Google can make, opportunities should exist where said franchise can promote locations, offers, coupons, etc. within their Sitelinks display area.
• Sitelinks Suggestions: Google’s Webmaster Tools does a good job of making suggestions on crawl issues, content issues, etc. Nowhere though are indicators as they pertain to Sitelinks improvements. If Google can’t give us the ability to block out content from appearing in the Sitelinks area – why not provide us with some additional insights on why the links that do appear are always there? (Hint: They’d have to share too much about their linking values to make this happen. Still, it’s wish list.)

Megasitelinks and Online Reputation Management Issues

I’ve witnessed a number of branded search results that provided companies with a free pass from a lot of negative reviews. In particular, the Sitelinks 12-pack growth has done a lot to push negative press in the SERPs below the fold. That’s good for brands, bad for searchers. I won’t get into the entire RoR-style-site issues — but it’s clear that geographically relevant Sitelinks will sway ORM issues one way or another depending on what’s shown.

What do you think about the news of Google’s Megasitelinks? Are you expecting them to prove as an innovation – or flop? Share your thoughts in the comments area below.

Jim was careful in his commentary, indicating that others have steadfastly defended Google’s innovation to him. I found that a bit funny since that perspective was shared just a month after they cloned Facebook and called it Google+.

So, I want to know what YOU think.

Is Google a true innovator or just another tech industry bully?

– Please provide your thoughts on this in the comments section below!

My take is this…
I’ve been critical of Google in the past and I have no intentions of letting off the gas. I said to Jim through the course of our initial discussion…

Google has done a tremendous job of innovating monopolization.

Today, just one month later, the once promising community has become useless to me. It’s just another proverbial flash in the pan on Google’s stove-top.

The only difference is that more than 25 million users have tried their hand at Google+. While impressive in their acceleration towards this scale – there’s little else to celebrate.

One month ago today I decided to bring my blog back from more than a two year hiatus. The reason I brought it back was quite clear — Google+ had created enough momentum that I couldn’t ignore it. My opinions began to form and I felt compelled enough to publish them for the purpose of initiating some discussions.

My posts then have been pretty direct. I started by sharing Tom Anderson’s words on Facebook’s inherent advantages in the social space. I then followed up by wondering aloud — How would Google+ be monetized? Then, as I forced myself to use Google+ more and more I realized that there were some serious issues.

I then made a (strong) case for Google+ Stream Modifiers and later proved how “power plussers” are killing the vibe.

After all that writing, thought, analysis and discussion, you’d think I’d check in on Google+ regularly, right?

Wrong.

I have to set a calendar item to remind me to check Google+ Notifications.

Ridiculous, right?

This is the monster that Google has created with Google+. It’s almost a perfect clone of Facebook. That alone is laughable considering Google has cried “foul” accusing Microsoft of stealing their search results. The only thing that Google+ hasn’t figured out that Facebook has nailed? Users. Community engagement. Branding. Games. Discussions. Hell, even noisy chatter.

What they have done is alienated early adopters time and time again.

The icing on the cake for me is that Google has gone out of their way not to integrate Google+ with their paying Apps for Domain customers. Rather, they have required those customers to go out and create separate accounts to continue using the Google+ service. So now, rather than getting instant notifications on updates, users have to juggle multiple logins or rely on the annoying amount of email notifications just to stay connected with Google+.

I know I’m not the only one annoyed by all this. I’ve seen many ranting and raving on the issue and Rand Fishkin of SEOmoz has shared his displeasure directly on Google+.

On top of all that the core community and experience in Google+ hasn’t changed at all. The Brogans, Pirillos and Scobles of the world still dominate Streams for all of their followers. Everything “social” about it has become old, stale and cumbersome.

With that in mind, the ball is squarely in Google’s court. What they do from here on out is up to them.

As the first couple of weeks wind down to a close, Google+ seems adamant on being more than just another flash in the pan. 10 million strong have joined the ranks at Google+ and Google co-founder/CEO Larry Page was certain to highlight these and other impressive numbers on the company’s quarterly earnings call Thursday.

The search giant though, known to favor large and reputable brands in the past, could be lending the same benefits in their new social network.

Identifying the Problem

In my previous post I made mention of how Google+ Streams can become overrun with updates coming from the same series of people.

The cause for this behavior in Google+ is due to user interaction on posts.

When someone with little to no following posts an update on Google+ it is often swept down the Stream because there is no one there to interact (+1, comment, share) the post.

When someone with a much larger following posts an update though – there are thousands upon thousands of people there to interact with the post. When they interact, the timestamp on the post is effectively changed, and the post then continues to stay at or near the top of your Stream view.

The Usual Suspects

Before I get into the list of offenders here, I want to say that I love these people. They are what make the Internet, our industry of search and social communities what they are. They’re not the problem themselves, they’re simply the poster children for it.

That said… Anytime I load up Google+, I’m guaranteed to see updates from Matt Cutts, Chris Pirillo, Robert Scoble, or Danny Sullivan.

I chose to see updates from these people in my Stream so if there’s anyone to fault, it’s me. The problem caused by the existing technology though is something I firmly believe should be changed or improved upon.

Class Structures & Celebrification

Intentional or not there’s a class structure created by the Google+ Stream system. On one hand, that’s okay. Online communities mimic offline communities and whether we like it or not, there’s certainly a class system in our offline society. On the other hand, I don’t believe that I should miss out on my connections’ updates and posts because more popular users are also active in the community.

Yet, that is exactly what happens.

In one of my Google+ posts tonight, Brian Chappel of Ignite Social Media presented the fact that celebrification in this social setting allows Google to push more traffic to Google properties – but at the cost of greater relevancy.

How the Rich Get Richer

The stats don’t lie.

Cutts, Scoble, Pirillo, Sullivan… These are all successful authors and their contributions to online industries are far reaching and have been for years. In other words, they’ve worked hard to get where they are today. They’re successful in life, in business… and in the Google Search Results.

The SERPs are where things come full circle.

Google+ and the +1 Button are all about personalized search results. If you’ve logged into Google Webmaster Tools recently, you’ve likely noticed that there’s some handy reports on the +1 button and how it has been used in conjunction with your site’s URLs.

Barry Schwartz describes these reports and highlights how popularity plays a role.

The Search Impact report gives you an idea of how +1′s affect your organic search traffic. You can find out if your clickthrough rate changes when personalized recommendations help your content stand out. Do this by comparing clicks and impressions on search results with and without +1 annotations. Google will only show statistics on clickthrough rate changes when you have enough impressions for a meaningful comparison.

The Activity report shows you how many times your pages have been +1′d, from buttons both on your site and on other pages (such as Google search).

Finally, the Audience report shows you aggregate geographic and demographic information about the Google users who’ve +1′d your pages. To protect privacy, Google will only show audience information when a significant number of users have +1′d pages from your site.

Note “significant number of users” and “how +1′s affect your organic search traffic.” In other words, the more +1 interaction, the more positive the influence on organic traffic. Rest assured, the most popular users in Google+ are also benefiting from having a more “significant number of users” accessing their content.

Highlighting this late Thursday night was Jason Calacanis who noted how much more traffic Google+ sent compared to other networks when it came to his shared content:

Google+1 is crushing Facebook and Twitter as a traffic referrer to the LAUNCH blog. It’s just insane the difference!

With 100k twitter followers I’m getting half the clicks as from Google+ with 10k followers. That’s 20x the effectiveness.

I’m getting 10x the traffic from google+ than Facebook and i have 5k facebook friends. That’s 15x more effective.

Same Old Battles Wage On
Five years ago webmasters and site owners put their energy into building quality backlinks. Today, those same people are putting more energy into the social marketing of their content. It’s all for the same reason, too – to climb the organic rankings and gain more exposure.

Unfortunately, larger sites are led by more popular people. In both cases – backlinks for sites and social interaction on Google+ posts – quantities will be higher for those already having that celebrity stature.

The end result remains the same.

Google employs an algorithmic imbalance that rewards bigger brands with broader reach and discards smaller organizations and individuals.

–
I’d like to thank Matt Crouch of SEO Video Corp for his observations and comments that helped lead to this post being written in the first place.

With an incredible start out of the gate my mind has continued to race with thoughts as to how Google could innovate social experiences online through Google+.

The late Dr. Carl Sagan once said, “You have to know the past to understand the present.”

In the spirit of this quote I would like to present a case for how Google+ could use something successful from their past to improve upon the Google+ experience… Search modifiers.

The Background on Modifiers

Years ago when search modifiers were introduced on Google, they were primarily used by marketers and webmasters trying to perform research and analysis. Over time, savvy searchers also began using modifiers to shape the SERPs and find what they were looking for more quickly.

In no time at all, search modifiers were being used to do a number of things, including:

• Locate Top Pieces of Content on Domains
• Research Changes in Domain and URL Records
• Check Indexing Status on Content
• Locate Backlinks & New Link Opportunities
• Find Duplicate Content and Plagiarism
• Search for Specific Phrases & Strings
• Exclude Keywords/Phrases from Results
• Hone in on Domain Types (.gov, .edu, etc.)
• Search for Specific File Types (PDF, DOC, Images, etc.)

Over time, Google’s Advanced Search page adapted to fit these needs and now anyone has access to those modifiers with the help of Google’s interface.

Google+ Streams

As new users jump onboard with Google+ they may find that their streams are different than they’d expect. It’s not a direct clone of the Facebook “News Feed” contents and in many cases old content you’ve already seen can bubble back up to the top. This happens most frequently if you’ve chosen to follow people with larger numbers of followers like Zuckerberg, Page, Brin, Scoble, etc.

As an example, my connections on Google+ are predominantly from the search or social industry, so I see Danny Sullivan and Chris Brogan appearing most often in my Stream.

For the sake of this discussion the Stream in Google+ is our search result page. It’s great, it works, and it provides with access to the top pieces of content. It just doesn’t provide us with refinement opportunities – and I can see many cases where users will want that.

Examples of Stream Modifiers

Here are some examples of Stream modifiers that I can see being introduced:

Search
I’m always amazed when Google products are lacking search functionality, so let’s start there. Streams are populated with all of your circles by default, so it’s easy to see what your connections are up to when you’re checking out the default Stream.

Wouldn’t it be nice to see what your users were saying about particular topics though? Using current headlines as an example, it would be great to search for these items across all of my circles to find and participate in discussions on certain topics:

• World Cup
• Hugh Hefner
• Obama
• All Star Game
• Murdoch

Having a simple search query box would be an excellent way to provide this to Google+ users.

Time & Date Filtering
Twitter is all the rage when news is breaking. Real time hash tags and trending topics allow users to see that something abnormal is happening and actively being discussed. Think about how many news items Twitter has made you aware of. From celebrity deaths to impending natural disasters, Twitter makes it easy for people to see what’s happening now.

Google+ doesn’t quite do that, but it’s not because Google doesn’t know how. Even logged out Google SERPs allow you to customize the date range of the results you view when you search on Google – so why not provide something similar in Google+ Streams?

Display Toggles for Update Type
When it comes to Streams, there are five types of things you can share:

• Text Updates
• Photos
• Videos
• Links
• Places

It would be great for Google+ to allow users to see all links shared in their Stream, all photos, videos, check-ins, etc.

Likewise, it’d be nice to filter out some of those updates to reduce the amount of noise when you’re trying to see what your friends and connections are up to.

The Users Interface

Here’s an example of what the Google+ interface could look like with integration of search and Stream modifiers:

Modified Google+ User Interface (Click to Enlarge)

What Stream Modifiers Would You Like to See?

Share your thoughts on what you’d like to see below in the comments.

Google+ invites have circulated for a couple of weeks now and as the exclusivity begins to wane, curiosity of Google+’s evolution is on the rise. One thing that seems incredibly common is that Streams are too frequently populated with updates on Google+ itself – something that many users have begun to find quite annoying.

Suspiciously absent in all the Google+ talk is an active discussion of how Google plans to make money with their social network.

Let’s start that discussion here.

The unavoidable comparisons between Facebook and Google+ have been tackled from every conceivable angle, but the balance sheets tell the true story as to why these networks exist in the first place. Facebook’s 750 million users will help to generate an estimated $2.2 billion in annual display ad revenue, while Google will be lucky to generate $1.2 billion.

It’s not often that Google is the small player in any online game, is it?

When Google first launched GMail talking heads all wondered why Google would bother to invest so much in technology just to give away the service and data storage for free.

The answer was simple — to serve more ads.

If you haven’t made the connection yet, Google+ will most certainly serve up ads all over the place and they will probably do it in dozens of different ways.

While we as users of Google+ are getting acclimated to Streams, Circles, Huddles and Sparks – Google is probably hard at work trying to find the best way to blend in sponsored ads, links and videos of all kind. Cementing this belief for me is that Google has already taken a steadfast approach to preventing businesses and online communities from infiltrating their network, while knowingly attracting some of the most active and savvy online publishers as their core group of users from the start.

What do you think is in store for Google+?

Discuss in the comments below, and let’s get some ideas flowing.

EDIT: Jon Henshaw wrote a post on Monday discussing the same thing, but it much more detail. Please give this a read: Google+ will introduce new ad types and higher prices

I never thought I’d come back to the blog after a two year, one month and nineteen day hiatus to blog about Tom, the awkward looking friend we all popped our MySpace cherry with. But I have, and with that said, I’d like to buy Tom Anderson a beer for providing some interesting perspective on Facebook, Google+ and the topic of social media innovation.

Before the July 4th weekend officially began, Facebook CEO Mark Zuckerberg told the world that “something awesome” would be announced this week. After enjoying the weekend, Americans settled in for a short work week to promptly ho-hum Facebook’s unveiling of Skype video calling and advanced chat features.

The global reaction was equally unenthusiastic.

The insider reaction was harsh. CNet, PC Mag, online marketing blogs and news outlets comprised of connected people had little positive to say about Facebook’s “awesome” innovation. The reaction was a bit predictable though considering that Google+ just hit the scene with an exclusive invite-only social community.

On one hand Google drops invites off to thousands of highly connected technology insiders and rewards them with an entirely new, engaging social community experience. On the other hand Facebook takes one of their least appealing features in chat and tries to innovate it by adding a technology that has been available to everyone directly from Skype for years.

The Role of Exclusivity in Google+

Earlier I referred to “insider” reaction of Facebook’s new features. Insiders, for the purpose of this discussion, are the numerous authors on prominent technology websites that covered both topics from every conceivable angle. These are the people that (even outside of Internet marketing) are early adopters of social web services. They’re the ones that spend their lives online… After all, that’s their job, right?

With Facebook – the “new” video and chat enhancements were rolled out to the masses in an instant. Intense market saturation meant that the majority of people who can now use those features on Facebook are not all white collared tech insiders. They’re blue collared workers, construction crew members, grocery store clerks and students just looking to stay connected with the one large social community that all of their peers are using.

They’re not the users Google+ cared to attract at launch.

Peer Pressure

Now let me ask you one thing: If you could choose to be associated with a group, would you choose to be in a massive group of people with few common threads, or, choose to be among some of the most well connected and influential people in the technology sector?

Since you and I are both “connected” like those industry insiders writing about Google+ and Facebook’s flop, I’ll assume that most of us would choose to rub elbows with the likes of Larry Page, Sergey Brin, Michael Arrington and Matt Cutts.

In other words, we’d like to be alongside a few of Google+’s most popular members.

You may want to be connected to these people because of who they are, what they’ve accomplished, or maybe for the successes that they have accumulated in their professional lives. There’s something about the underlying exclusivity factor and a desire to emulate the success of these potential peers that’s too much to ignore.

Few tech insiders would ever feel excited enough to write about how they’re as connected as Esther Millward, who could be some 84 year old woman from Oklahoma City who does needlepoint when her arthritis isn’t flaring up. For this reason, there’s little fanfare for Facebook’s video chat services.

To help me illustrate this point more clearly, here are the top 10 users of Google+ rated by the number of followers they have:

1. Mark Zuckerberg – 44,871 Followers
2. Larry Page – 30,271 Followers
3. Sergey Brin – 23,426 Followers
4. Vic Gundotra – 19,693 Followers
5. Robert Scoble – 17,109 Followers
6. Matt Cutts – 13,761 Followers
7. Leo Laporte – 11,852 Followers
8. Bradley Horowitz – 10,392 Followers
9. Markus Persson – 9,552 Followers
10. Kevin Rose – 9,504 Followers
11. MG Siegler – 9,409 Followers
12. Gina Trapani – 9,194 Followers
13. Tom Anderson – 6,734 Followers
14. Jeff Jarvis – 6,392 Followers
15. Kelly Ellis – 6,265 Followers

Some very familiar faces up there, right?

The kicker for me is that Zuckerberg is #1. Bonus points to MySpace’s former CEO Tom and his 13th place ranking. Other notables in the top 100 most followed include the likes of Marissa Mayer, Danny Sullivan, Marshall Kirkpatrick, Jason Calacanis, Chris Pirillo and Chris Brogan.

Okay… So, What About Tom?

It’s time to bring this back to Tom Anderson, the former CEO of MySpace and why I want to buy him a beer for his take on these topics.

It’s easy for us in search and the associated Internet related industries to write off Facebook’s announcement and hail Google+ as the next big thing. While that may well be the outcome from these two summer stories, here’s a healthy dose of broader reality that Tom served up (now available on TechCrunch):

“…lots of companies are going to build things like video chat, but Facebook competitors also have to build up their social graph first. Facebook’s job is to just keep innovating.”

It’s a very short comment but one that caught my attention immediately.

It’s easy for us in technology industries to assume that we’re the perfect case study for anything that is born online. The reality is, we’re not, and in my opinion, Tom’s position is one that we should certainly pay attention to.

When it comes to social communities, are there more than a few others more qualified to speak on the topic?

Before Google+ and Facebook ever hit the scene, MySpace was the social community for the masses. Since Facebook’s explosion began MySpace’s quick fall from grace has been well documented. Despite that media coverage, few people could be more aware of the struggles and challenges that took a company once valued at $12 billion (in 2007) to being sold for just $35 million (last month) than our old friend Tom Anderson.

For those keeping track that’s an eleven billion nine hundred sixty-five million dollar loss.

On the surface Tom’s quote may seem innocent enough. Taken in context though, it is clear that he offered his take while being supportive of Zuckerberg’s initial reactions to Google+.

After thinking about it a while I began to suspect that Tom was speaking on a topic that he wished he had known the truth about sooner, though. MySpace, for all of their success, did little innovating while at the top of the social world. That lack of innovation is what allowed Facebook to seize control with it’s simplicity and once exclusive membership.

The Future of Google+ and Facebook

The eventual success or failure of Google+ is already baked into the social network’s framework. I’m not suggesting that Google+ will fail, either. I’ve seen many people including Greg and Dave discuss their immediate admiration for the community as well as seeing some writing on the wall indicating that Facebook’s days may be numbered.

Those are both opinions that I certainly trust.

The only things I would suggest are that the make-or-break features that will propel Google+ toward it’s fate are cards that have already been dealt, and, that Facebook’s grasp of broad market share are the key to it’s longevity.

Google is known for their core search and advertising businesses. They’re also known for a host of products that die slow and painful deaths like Google Wave, Google Video, and most recently, Google Health. Could Google+ be the next in line, or is the exclusive, invite-only release (which follows in Facebook’s footsteps) something that suggests more precision?

For Facebook, will innovations like video chats be enough to keep people engaged? Facebook Places was initially discredited at launch, but more and more are using the feature and are jumping on board with it.

Perhaps we shouldn’t always be so quick to judge.

Late last week I found myself watching a documentary on organized crime in New York City. Focused on the Gambino crime family, this documentary illustrated how the mafia (which included the Gambino and other families that made up the Cosa Nostra) had their hands in on nearly every facet of city life. From restaurants and delivery trucks to corrupting the law enforcement and legal systems – the organization was in control.

Less than an even a decade ago Google was simply a search engine. A player in the growing Internet that consumers and the public were excited to use and would eagerly embrace in their everyday lives. In the years following their cult like launch however, a blend of strategy and brute force has catapulted them to become what I now see as the Internet’s first organized crime family.

I must admit that drawing a comparison like this is a bit of a reach without conducting any research. So that’s what I started to do… And in the past few days I’ve learned that while not quite “criminal” in their everyday actions, Google’s dominance and reach has resulted in their becoming a monolithic organization. Precisely, I’m sure, as planned.

The Foundations of Activity
For the mafia to operate to its full potential there needs to be structure and reach that allows all activity to be seen, heard, felt and eventually capitalized on. Most of what the public knows about true organized crime is that these organization penetrated into labor unions, law enforcement agencies and massive commercial organizations. Unbeknownst to them, the public ends up living life feeding these organizations. Whether it be through paying higher prices or casting favorable public opinion behind devious characters – much of what organized crime controls is done so with a level of transparency that can never be rivaled.

Having worked in the search engine marketing industry, I’m more aware of what Google sees than most others would. Most of my blog’s readers know as much or more about Google’s reach than I do – but the public still remains unaware.

Google’s Bread and Butter
Let’s take a deeper look at the types of information that Google has access to, starting with their core competency – a simple search query on Google.com.

Google Account holders and Google Toolbar users are enrolled in something called Web History. Web History allows Google to track what you search for, and more importantly – what you click through to. Google’s willingness to track what you’re searching for and viewing online is masked at a usability feature. In fact, you can log into Web History and see any of the following types of Google Search results that you’ve clicked through to, including:

• Web
• Images
• News
• Products
• Sponsored Links
• Video
• Maps
• Blogs
• Books

Now, let’s be clear about a few things. Google Account holders have an ability to view their Web History and look at some of the information that Google is recording. Should we be naive and believe that Google’s being completely transparent here?

I vote no. And I hope with some more careful consideration – you may agree.

Google’s Penetrating Reach
Google is no longer simply a search engine. They may have started that way, but have since penetrated deep into the everyday lives of billions of people worldwide… They’ve done that not by forcing themselves on you – but acquiring or fueling the sites, services and networks the public relies on most frequently.

While I don’t like relying on Wikipedia for too much, I’d ask that you take a look of the List of acquisitions by Google article that can be found there. Having acquired more than 50 companies, Google now has the ability to control, monitor and view user activities across an overwhelming portfolio of products and services.

Here’s a comprehensive list I’ve built of known product and service sites. While I’m sure that I’m missing some from my list, the key here is the massive scale and reach that Google now has.

Google’s Standalone Applications
• AdWords Editor
• Chrome
• Desktop
• Earth
• Gmail/Google Notifier
• GoogleTalk
• Pack
• Photos Screensaver
• Picasa
• Picasa Web Albums Uploader
• Secure Access
• SketchUp
• Visigami
• Web Accelerator

Google Application & Desktop Extensions
• Blogger Web Comments
• Dashboard Widgets for Mac OS X
• Google Gears
• Send to Mobile
• Google Toolbar

Google’s Mobile Products & Services
• Blogger Mobile
• Calendar
• Gmail
• News
• Google Mobilizer
• iGoogle
• Maps
• Mobile Updater
• Mobile Search
• Product Search
• Reader
• Sync
• YouTube
• Picasa Web Albums
• Google Latitude

Google’s Web Based Products
• AdSense
• AdWords
• AdWords Website Optimizer
• Audio Ads
• Click-to-Call
• DoubleClick
• Grants
• TV Ads

Google’s Communication and Publishing Tools
• 3D Warehouse
• Apps
• Blogger
• Calendar
• Docs
• FeedBurner
• Friend Connect
• Gadgets
• Gmail
• iGoogle
• Notebook
• Knol
• Marratech e-Meeting
• Orkut
• Page Creator
• Picasa Web Albums
• Reader
• Sites
• Voice
• YouTube

Google’s Developer Tools
• Android
• App Engine
• Code
• Mashup Editor
• OpenSocial
• Subscribed Links
• Webmaster Tools
• Web Toolkit

Google’s Geographical and Mapping Tools
• Maps
• Map Maker
• Mars
• Moon
• Sky
• Ride Finder
• Transit

Google’s Search Based Products
• Accessible Search
• Alerts
• Base
• Blog search
• Book Search (Previously Google Print)
• Checkout
• Code Search
• Directory
• Directory (Google China)
• Experimental Search
• Finance
• Groups
• Image Labeler
• Image Search
• Language Tools
• Movies
• News
• News Archive Search
• Patent Search
• Product Search (Previously Froogle)
• Scholar
• Sets
• SMS
• Squared
• Suggest
• University Search
• U.S. Government Search
• Video
• Voice Local Search
• Web History
• Web Search

Google’s Statistical Tools
• Analytics
• Gapminder
• Trends
• Zeitgeist

Google Hardware
• Google Search Appliance
• Google Mini

Other / Miscellaneous Products
• GOOG-411
• Health

Google’s Key Players are Products and Services
Through decades of research conducted by prying journalists and investigative organizations like the FBI, we know that organized crime families have a strong hierarchy in place. This hierarchy puts the most influential and powerful people in a position to orchastrate specialists beneath them. This differs for Google in the sense that products and services take the place of people. What I mean by saying this is that Google’s brand and services represent leaders and shot callers that monitor and profit from all of our Internet based activity.

I am by no means an expert on organized crime, or Google for that matter. My years of being an Internet professional though suggest that the following products and services are the key players underneath the Google family:

• Search
• Analytics
• AdWords & AdSense
• GMail
• YouTube

These five players have market reach and dominance that is unrivaled by any other non-Google organization. The majority of Internet users rely on these five core services – and as a result, Google stands to profit.

The Profitability of Internet Dominance
What I find most disturbing about this trend though is that like criminal organizations in the past, Google is driving up the price for advertisers and consumers with the incredible control they have over all of the market variables. If you’re a web site owner, you’re likely providing Google access and permission to analyze what your visitors are doing on your site. If you have a GMail account, they’re able to parse the content of your emails and tailor advertising accordingly. If you’re an AdSense advertiser, you’re paying a premium through a veiled cost per click system that Google developed in a proprietary way to maximize their profit with. And if you’re relying on AdSense, or “Webmaster welfare” as Sugarrae has termed the program – you’re simply fueling Google’s greed further while minimizing your own profitability.

When Google first went public, their IPO priced each share of the organization’s stock at $85. At one point in late 2007, GOOG shares traded at $711.25 – an increase of more than 736% over the IPO’s value.

This cult like following, both in technology sectors as well as in public appeal forced Google to not only become a household name, but even a verb. To this day you can go practically anywhere and overhear someone having “Googled” something, or even worse, be willing to “GMail” you something. Which leads me to…

The Public Appeal and Fascination
Any staunch Google supporter already thinks I’m both crazy and stupid for suggesting that Google’s operation mirrors that of a criminal organization. Keeping with the comparison of Google’s brands mirroring various mafioso of the past, I’m now going to go out on a limb and state that to me, Google is the modern day Teflon Don – John Gotti.

John Gotti’s following came through two methods. First, he struck fear in the eyes of the opposition and second, he served as public figure for people and the media to fall in love with.

CNN once wrote that Gotti murdered his way to the top of the Gambino crime family. I think Google did the same exact thing – and here’s why.

Back in the days before the “big three” search engines, Google results were so accurate that they were shown to anyone conducting searches on major online properties like Yahoo!, AOL and dozens of others. While most simply appreciated the accuracy of clean and relevant search results, Google was deviating from their “do no evil” mantra that many foolishly believe exists today.

One by one, Google effectively sniped the competition either by becoming a provider of materials essential for business (as was the case with the search results I referenced above) or, by simply buying out entire organizations and overhauling or eliminating them. Since their creation, Google has acquired more than fifty leading Internet based organizations… These acquisitions were entirely strategic and have gone on to help feed the Google family of operations. Some of the properties simply stood in their way and were later shut down. Others, often commanding billion dollar buyouts like YouTube and DoubleClick primed Google to reach the majority of Internet users across all demographics and gain insight into every little nuance of their online behavior.

While Google was either eliminating the competition or forcing them to rely on Google’s materials to survive, they placed an emphasis on winning public opinion. John Gotti was famous and untouchable in his Queens neighborhood because he often threw parties, picnics and gave handouts to those that surrounded him. Google is no different, and as an example of that – I’d simply point to GMail being the leading free email service or AdSense and Google Analytics as both free and market leading tools for webmasters to run an online business.

Comparing Google’s Services to Criminal Activities
It was suggested to me that Google’s services actually mirror a number of ways in which a criminal family or organization would work. Google’s AdSense program for example was compared to prostitution, their free applications likened to the distribution of drugs, and Google AdWords being similar to Google’s offering of “protection” for web site owners. I can certainly see and understand these comparisons but am interested too in how readers of this post will react to those comparisons being drawn… So please do leave comments to further the discussion.

Thoughts from Other Search Marketers
I tend to be anti-Google when it comes to my opinion, so I asked a few contacts from the search marketing industry for their thoughts and opinions on my comparing Google to an organized crime family. Here’s some of the feedback I got that was worth sharing:

David Wallace (Twitter: @davidwallace) offered the following:

“It’s apparent that Google wants to get into everything, at least wherever they can stick AdWords. For the company that has made statements to the fact that “search is core to them” they have gone a long way from search… Google Docs, GMail, Calendar, Talk… Shall I go on? http://www.google.com/intl/en/options/ clearly shows that although the Google Search page remains simple, the company is involved in tons of stuff and upsetting many business models…In fact I’m just waiting for the day that we will hear the announcement, “Google acquires Wal-Mart.”

Ben Cook (Twitter: @Skitzzo) had this to say:

“Follow their rules or else, they pay you but can up their take without notice and you get away with more if you produce for them… That’s Google for you. Mobsters have to stay under the radar & deal with cops & other families. Google answers to no one.”

Greg Boser (@GregBoser), while not offering any direct comments commended the comparison and said he looked forward to reading the post when it was completed. Social media specialist Brent Csutoras (@BrentCsutoras) and optimization expert Andrew Gerhart (@andrewgerhart) both thought that Google is more like a government than a crime family.

Chris Hooley (@ChrisHooley) later added:

“Well, a crime family works under the law. Google works above the law. If you are going to compare them to a corrupt organization I would compare them to a corrupt government who puts the koolaid in the water supply.”

Michelle Robbins (@MichelleRobbins) provided a lot of assistance in creating this article. She offered up an editorial review and some recommendations, as well as this quote:

“Even though, in effect, Google has a stranglehold on internet advertising and marketing options – much like a crime syndicate can control and dictate how businesses operate and do or do not succeed in a given location or market – there is one key difference. The mob is self aware. They don’t pretend to be anything other than what they are.”

Please Help Further The Conversation…
Anytime someone takes interest in making such a harsh comparison, strong opinions will be formed. I am not saying that I am right or wrong for drawing or writing of the comparisons made. The above is simply my opinion – and I want to hear more from you, the reader. Please help the discussion continue through the commenting feature below.

Photo Credits: Header Image “Anybody There” by mooranguinho, licensed for use through Creative Commons. In Post image “Yes? Hello?” by rossinabossio also licensed for use through Creative Commons.

My plans for 2009 continue to become more exciting. I just got word back from Ross Dunn, and the folks at WebmasterRadio.FM have made it official… I’ll be Ross’ new co-host on the SEO 101 program that airs each Monday at 5:00PM ET.

What is SEO 101?
SEO 101 is weekly program that speaks to Search Engine Optimization, or “SEO” from the very beginning. The goal of the SEO 101 program will be to interact and teach listeners about the many aspects of SEO by sharing helpful information without overwhelming them with technical details.

Joining Up with Ross Dunn and WebmasterRadio.FM
I met Ross back in February of 2007 at SMX West and he’s a great guy, great SEO and a wonderful father. If you aren’t already keeping tabs on what Ross is doing – please connect with him on Twitter and check out his company, StepForth Web Marketing Inc..

Here’s a photo of [L to R] Darrell Long, myself, Ross and David Wallace at Bennigan’s last February in Santa Clara. [Photo Credit to David Wallace]

Ross and I have obviously been pretty excited about this, and here’s his take on the development:

SEO 101 on WebmasterRadio.fm has been a big success since Jennifer Evans Laycock and I took it on and revived it but when Jennifer left due to time constraints I needed someone to fill some big shoes. Thankfully I have been around long enough to know just how good a guy and SEO Eric Lander is. So when he agreed to be my co-host on the show it felt great to know the show was once again whole. Now it is time for us to plan out some great shows!!

To compound on that, I also want to thank Jennifer for her contributions to the show (and our industry) and wish her the best with all that she has going on. We’ll certainly try to steal her back and get her back on the air for our listeners, too. :)

About Webmaster Radio
Founded in November, 2004 – WebmasterRadio.FM is a free, 24/7, interactive Internet Radio Network focused on the B2B digital marketing world. We have burst on to the Internet Media scene as a community destination with 100% original programming, live broadcasts, archives, and podcast shows. We are proud to boast that our shows are hosted by the most respected names in the business world!

WebmasterRadio.FM Links of Interest:
• WebmasterRadio.FM Website
• SEO 101 Program Details
• Searchbash
• Advertising on WebmasterRadio.FM
• WebmasterRadio.FM’s Blog
• WebmasterRadio.FM on Twitter
• WebasterRadio.FM on Facebook

My goal here is to encourage you to the point of using log files on a regular basis to better tune up your hosting and marketing platforms.

Page Tagging Vs. Log Based Analytics

When you think about web analytics, what do you really think of? The majority of people I know have told me all about their nicely designed dashboards filled with telling graphs, growth charts and tables that just wait online for them to view them whenever needed. While that’s all well and good – those reports are always coming out of page tagging analytic programs. The problem? Page tagging analytics has limitations, and some of those limitations are simply unacceptable for hardened SEOs and webmasters.

That’s not a knock on page tagging analytics either. Page tagging is a popular method of acquiring data as the ease of use and on demand availability of reporting all add up to make these tools a required resource. Log files for me simply help me go that extra mile.

I want you to guess how many of the following analytic suites are providing you with reports and data generated from, in part, log files:

• Google Analytics
• Omniture
• Microsoft adCenter Analytics
• Sitemeter
• Quantcast
• Compete
• HitWise

Ready for the obvious answer?
– Zero! None of the above use data recorded by your server for statistical analysis.

Olivier Amar of CompuCall earned some kudos this morning. When I asked how many followers were not out there checking their logs – he tweeted a reply about ClickTracks – one of the few analytic suites out there for SEOs and site owners that actually integrates log files out of the box.

I don’t want to get into a whole lecture about the differences between page tagging analytics and log parsers (or hybrid solutions for that matter). What I do want you to realize is that no matter the hosting platform, there is some more useful information you could be extracting about your web site and your visitors if you can acquire the logs.

Familiarize Yourself with Log Files

Before we jump too far in it’s probably best for us to review what a server log file is, what it looks like, what data it contains, etc.

What is a Server Log File?
Wikipedia defines a server log as:

A server log is a log file (or several files) automatically created and maintained by a server of activity performed by it.

A typical example is a web server log which maintains a history of page requests. The W3C maintains a standard format[1] for web server log files, but other proprietary formats exist. More recent entries are typically appended to the end of the file. Information about the request, including client IP address, request date/time, page requested, HTTP code, bytes served, user agent, and referer are typically added. These data can be combined into a single file, or separated into distinct logs, such as an access log, error log, or referrer log.

How Do I Retrieve Server Logs?
Each hosting provider or company handles this differently. My hosting company makes it easy for me by keeping logs available via FTP on a 7 day cycle before any logs are removed. I have adapted to just pull those logs down off my server oce a week through an automated application. Set it up once, and now I can forget about the hassle.

I’ve seen other hosts that make log acquisition more… Trying. In any event, server logs have a number of different recording options, structures, formats and file types. This post is focused on using the logs you have available to you – not acquiring them. I highly recommend working with your server administrator or hosting provider to acquire access to logs if you do not have that already.

If you are ever presented with an option – push to acquire Extended Log Files and then quickly hand your hosting provider or server admin a copy of this resource from the W3C.

What Does a Server Log Look Like?
Here are five lines I pulled out of a server log file from my blog as recorded yesterday, March 29, 2009:

85.89.185.215 – - [29/Mar/2009:01:00:09 -0700] “GET /wp-content/uploads/2007/09/100cap006.jpg HTTP/1.1″ 200 46012 “http://www.ironworksforum.com/forum/showthread.php?p=1200167″ “Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/530.1 (KHTML, like Gecko) Chrome/2.0.169.1 Safari/530.1″

38.99.107.141 – - [29/Mar/2009:01:00:10 -0700] “GET /feed HTTP/1.1″ 302 5 “-” “Mozilla/5.0 (compatible; FriendFeedBot/0.1; +Http://friendfeed.com/about/bot)”

193.252.149.15 – - [29/Mar/2009:01:07:24 -0700] “GET /276.html HTTP/1.1″ 200 24655 “-” “Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)”

69.147.112.169 – - [29/Mar/2009:01:11:01 -0700] “GET /feed/rss HTTP/1.0″ 302 0 “-” “Yahoo Pipes 1.0″

66.249.72.136 – - [29/Mar/2009:01:12:28 -0700] “GET /robots.txt HTTP/1.1″ 200 508 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”

In a 12 minute span on my web site, some pretty cool things happened that I would never have known about through any page tagging analytic suites I’m using.

Let’s dissect each of these five lines and I’ll show you what I mean.

Entry #1

85.89.185.215 – - [29/Mar/2009:01:00:09 -0700] “GET /wp-content/uploads/2007/09/100cap006.jpg HTTP/1.1″ 200 46012 “http://www.ironworksforum.com/forum/showthread.php?p=1200167″ “Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/530.1 (KHTML, like Gecko) Chrome/2.0.169.1 Safari/530.1″

Someone is stealing my images! Like a lazy webmaster, I don’t lock down much and prevent other people from using it. In this referenced log file I now have evidence of someone using one of my images (regularly, I might add) on another web site’s discussion board.

This log file entry tells me that this person is using this image on this discussion thread.

Not cool! Now for me, bandwidth isn’t much of an issue and I don’t really mind if someone is repurposing that image. If that were protected photography though – I’d want to keep it under lock and key. More on this later.

The key here though is that the actual “page” being loaded up (the discussion board thread or user profile page) is hosted elsewhere. Since I don’t own that site, I don’t have Google Analytics code on the site and without this log file, I never would have known that this was taking place.

When you consider how much this could happen with a large web site – you can probably see how quicly this can become a big issue.

Entry #2
The next log file entry was this:

38.99.107.141 – - [29/Mar/2009:01:00:10 -0700] “GET /feed HTTP/1.1″ 302 5 “-” “Mozilla/5.0 (compatible; FriendFeedBot/0.1; +Http://friendfeed.com/about/bot)”

As the tail end may suggest to you, this is a FriendFeed bot that’s coming through my web site and pulling a copy of my blog’s feed. Friendfeed’s bot will then see if there’s any new entries and pull them via RSS to use on their own site since I’ve allowed them to do so.

If you’re watching things like page views, this wouldn’t actually count in other analytics since again – the user requesting the data never actually came to my web site. The other issue? The “user” here is actually a bot and my guess is that if it’s like GoogleBot, it probably won’t bother to execute any javascript code that would be required for page tagging analytics to record the hit.

Entry #3
Next up:

193.252.149.15 – - [29/Mar/2009:01:07:24 -0700] “GET /276.html HTTP/1.1″ 200 24655 “-” “Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)”

An old post on my blog on the Internet Marketer’s Charity Party at SES San Jose is being retrieved here by another bot, this time called VoilaBot. Ever heard of VoilaBot before? Sadly, I had not – which is more telling about my failures as an International SEO.

Voila is the provider for Wanadoo, which is a huge portal in France and one of the biggest european ISPs.
Voila itself is one of the best known web brands in France.

Where’d I get that information? From heini, a veteran user on Brett Tabke’s WebmasterWorld, silly.

Entry #4
Still with me? Good, because we’re going to go easy on these last two entries to review! Next is…

69.147.112.169 – - [29/Mar/2009:01:11:01 -0700] “GET /feed/rss HTTP/1.0″ 302 0 “-” “Yahoo Pipes 1.0″

This is the footprint of Yahoo! Pipes, a fairly new RSS / News Aggregator that’s actually quite cool. All that was happening here is that a user of the Pipes program was loading up (or refreshing) my blog’s RSS feed. Again – this would never show up in anything like Google Analytics or Omniture. Why not? You know this. Just read the last three log dissections. :)

Entry #5
And finally… The staple of any SEOs diet… Googlebot!

66.249.72.136 – - [29/Mar/2009:01:12:28 -0700] “GET /robots.txt HTTP/1.1″ 200 508 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”

A well behaved Googlebot, too! The above request is the mark of GOOG coming through and requesting my blog’s robots.txt file for some more direction. It’s always nice when bots do what they say and are supposed to do first, right?

Now, Onto YOUR Log Files…

You don’t really care about what’s happening here on my blog — you want to see what’s going on with your web site. So now we get to take a look at how to make these log files work for you!

What You’ll Need
1.) Server Log Files
2.) Server Log Parsing Application
3.) Curiosity

Again, I’m not helping you with item number one.

With item number two, I’d recommend WebLog Expert. It’s an application that I’ve been using for years and bought the professional version of some time ago. Considering the low cost, I’d recommend it – but there are certainly other log file analyzers available to you.

Just check out download.com or directory listings on the Yahoo! Directory or on DMOZ.

Since WebLog Expert offers a free BETA version with some filtering options though, I’ll use them for screen shots.

Here are some report ideas I’m going to demonstrate for you…

1. Google, Yahoo & LiveSearch Spidering
2. Stolen Content
3. 400 Errors
4. 300 Server Redirections

For the purpose of this demonstration I’m going to use WebLog Expert as the log file analyzer because it’s a free solution and provides some easy filtering options. The key here is in using these filters to look at very specific data.

Google, Yahoo & LiveSearch Spidering

Log files record the user agent of each request. When a human visitor visits your site, their web browser will be recorded as it is labeled. Refer back to log entry #5 from above to see how Googlebot identifies itself. It tells my web server that it’s user agent is Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html).

In order to report on the spiders, you need to set up a filter that excludes all activity outside of the spiders. I’m going to take this one step further and show you how to set up filters in WebLog Expert that only pull activity on the big three – Google, Yahoo and LiveSearch.

Using the filters dialogue, you will need to add a new filter that includes activity based on spider name, and then select each of the appropriate spiders from the drop down list. To do this, you’ll need to set up three filters like so:

Once you set up three filters, one for each, you should see this:

And if you do, just click through the Finish button and then run you’re report. You’ll now get a wealth of information on your spidering activity.

Want to see what data is available? Click here to download the resulting report in PDF format. Here’s a hint to what you may find out… Yahoo! Slurp is sometimes a little more… aggressive than you may think:

Stolen Content

This report, even though I’m using it as an example for this post – really is one I need to take action on. Our goal here is just to find anyone out there who may be using my CSS or images on their own sites or for their own needs. If you were to run this same report, I’d suggest that as an action step – you take measure to prevent images from being hotlinked and so on.

With WLE, you’ll want to create the following filters:

Replace ericlander.com with your own domain, and, add or subtract any files you’d like to see in there. Other popular files to be stolen and reused? mp3, pdf, swf, avi, mpg, mov, and css lead the way for me.

Again, a sample report output of the above can be found here in PDF format.

400 Errors

One of the most useful reports for me over the years has been this report that only looks at 400-type responses. Now, any 400 error from your server indicates that something hasn’t been found. The most popular of which is the 404 error we’re all used to seeing – but there are other useful not found errors to note, including the following table from HTML Goodies:

• 400 : There is a syntax error in the request. It is denied.
• 401 : The header in your request did not contain the correct authorization codes. You don’t get to see what you requested.
• 402 : Payment is required. Don’t worry about this one. It’s not in use yet.
• 403 : You are forbidden to see the document you requested. It can also mean that the server doesn’t have the ability to show you what you want to see.
• 404 : Document not found. The page you want is not on the server nor has it ever been on the server. Most likely you have misspelled the title or used an incorrect capitalization pattern in the URL.
• 405 : The method you are using to access the file is not allowed.
• 406 : The page you are requesting exists but you cannot see it because your own system doesn’t understand the format the page is configured for.
• 407 : The request must be authorized before it can take place.
• 408 : The request timed out. For some reason the server took too much time processing your request. Net congestion is the most likely reason.
• 409 : Conflict. Too many people wanted the same file at the same time. It glutted the server. Try again.
• 410 : The page use to be there, but now it’s gone.
• 411 : Your request is missing a Content-Length header.
• 412 : The page you requested has some sort of pre-condition set up. That means that If something is a certain way, you can have the page. If you get a 412, that condition was not met. Oops.
• 413 : Too big. What you requested is just too big to process.
• 414 : The URL you entered is too long. Really. Too long.
• 415 : The page is an unsupported media type, like a proprietary file made specifically for a certain program…

The filter setup here is super simple. Just create this one:

And the resulting report looks like this (again, PDF!).

300 Redirections

Every SEO needs to have a grasp of 301 redirects, and reporting on the ones your server dishes out is super simple here. Just like the 400-responses, you’ll need to set up a quick filter that only pulls 300-level response codes. Easy!

The value here for an SEO is pretty obvious – so I’ll let you run with why this report is useful. To check out the sample in PDF format, just click here.

Wrapping Up…

Hopefully this post has given you some more insight on how you can begin analyzing server log files. If I’ve confused you at any point, please do drop a comment below and open up a discussion for us as others may have similar questions or hangups.

Don’t be afraid to get creative with the use of filters too with WebLog Expert or any other application that you may find yourself using. It’s very easy to use filters to extract in depth metrics like time spent on site by visitors viewing movies, path of visits referred from Digg, bounce rate for StumbleUpon referrals, etc.

Finally, this isn’t meant as a knock on page tagging analytics and the information they offer. Every successful web site marketer should rely on both regularly – but when it comes to running a clean site, don’t just assume the logs have nothing to provide to you.