Enterprise SEO Blog RSS From re1y.com

Occupy Google

I noticed a few days ago that "site:domain.com" searches no longer go any deeper than 1,000 urls - many sites show significantly less, even when the number of indexed urls is high. Not yet seeing any push back from the seo community, but this is really huge for those of us responsible for the ranks of our clients. If your site has a large number of urls, this change means that you no longer can view what Google has cached beyond a limited set that is no greater than 1,000 results. This is not a good thing.

Over time, Google has become less and less transparent regarding the data that it chooses to make available to us. The recent uproar over Google's announcement to prevent access to referral data from logged-in searches has appropriately angered the seo community and is only the latest in a string of harmful pullbacks.

"Last week, the search giant said that it would begin encrypting logged-in searches that users do by default, when they are logged into Google.com. This further integration of a Secure Sockets Layer (SSL) will prevent search marketers from receiving referral data from the websites consumers click on from Google search results." seos were not buying google's privacy motive for encrypting search 2011-10

So Google's walling off and withholding the data on the search behavior of its user base, and claiming they're doing it for security/privacy reasons. They're downplaying the small numbers this involves, but at the same time, they're making it easier, and encouraging users to remain logged in - you can choose to do this from every Google property (eg. Gmail, YouTube, Google Maps, Google News, Google Video, etc.). So these numbers are not going to stay small, and the more it grows the more valuable the data you're not getting. Apparently, you can get access to the https data by advertising. How can you buy the privacy excuse when the changes do not impact advertisers? I see this as a way for Google to compete with Facebook - isolating the community into a vertical, then monetizing the data. It's their right, but it's bad for us.

In July 2010, Google removed our ability to see all inbound links they had discovered. Webmaster Tools now shows us only a 'sample' of the total they store. I have 2 posts on this: Caffeine May Have A Hidden Cost & Coping With The Loss Of Link Metrics. So Google penalizes sites for certain kinds of links, yet we may not be able to discover those issues when our ranks are harmed.

The withholding of data has become a pattern over the recent past. Here's a summary of the important ones I'm aware of:

- Index searches now incomplete - limited to 1,000
- Logged-in search metrics no longer available
- Link data from Webmaster Tools incomplete
- Supplemental results still exist but are no longer labeled as such

And while we're on the subject of harmful decisions, a client had an experience with Google that is the epitome of unethical and unjust behavior from the enterprise that claims to do no evil. His site was penalized and in order to keep his business functioning had to turn to Adwords (Google undoubtedly gains advertisers by penalizing sites). After a short time, the Adwords account was suspended, with no explanation. It turns out that a competitor had complained falsely that my client was selling counterfeit merchandise. Google would not reveal who the complainant was, and the only way to recover the account was to have the manufacturers confirm that my client was indeed a legitimate distributor of their goods. In Google's eyes, my client was guilty based on the claims of unnamed accusers, and the burden was on him to prove his innocence.

I'm aware that Google has the right to do all the things being discussed here, and in spite of its pr machine touting how people friendly the place is, it is a corporation that has morphed into a huge monopoly with agendas that reveal them to be more of an adversary than ever before. And lets not forget that every major industry, oil, telephone, railroads, steel, etc. all evolved via a free market to the point that one giant monopolized the market, and regulation was required to force ethical behavior upon them. Google is that monopoly right now, without the regulation.

Occupy Wall Street is the result of people finally saying, "Enough!" to the bad behavior. Wondering when that threshold will be reached with Google's continuing slide, and will enough people be willing to camp out to motivate change. Kind of scary to think about.

Google Has Lost The War Against Paid Links

This post is a follow on to Enterprise Search Manipulation, which discusses problems Google has with enforcement actions against major players like JCPenney.

Like the JCP fiasco, once again Google proves that it is incompetent with regard to enforcing its own guidelines with respect to the buying of links to push rank.

For the second time this year, the NY Times calls Google out for being unable to detect black hat techniques associated with the purchasing of PR. This time, it's not just one company, but the top 4 businesses (measured by their search results) showing up for searches for "Mother's Day Flowers" - search for "Trying to Game Google on 'Mother's Day Flowers'" to read the article without triggering the paywall/registration. The article is here. Relevant excerpts:

Internet marketing experts say Teleflora, FTD, 1800Flowers.com and ProFlowers are trying to elevate their Web sites in search results with a strategy that violates Google's guidelines.

The flower companies deny it. But all four have links on Web sites that are riddled with paid links, many of which include phrases like "mothers day flowers," "mothers day arrangements" and "cheap mothers day flowers." Anyone who clicks on those backlinks, as they are known, gets sent to the floral retailer who paid for them.

The links have now been evaluated by several independent experts (including us) and the article sites several examples that demonstrate the links were paid for.

Google denies that these websites were advantaged by their link buys, and further claims their automation is robust enough to detect and discount paid links - something we know to be false.

On Wednesday, The New York Times sent Google representatives a list of roughly 6,000 links to the flower companies that were built in the last month. After Google's spam team studied the list, a company spokesman, Jake Hubert, sent this statement:

"None of the links shared by The New York Times had a significant impact on our rankings, due to automated systems we have in place to assess the relevance of links. As always, we investigate spam reports and take corrective action where appropriate."

In essence, Google said that these companies tried to game its algorithm, but for the most part, their efforts failed.

We doubt the claim that the links were fully discounted, but because of the nontransparent nature of Google's algorithm, no independent entity can confirm these claims. And although we can confirm the paid links exist in each case, there is so far no consequence for the clearly black hat strategy employed by these players:

Google is not saying whether it plans to demote any of the companies, but as of late Friday, it had not. A search of "mothers day flowers" had Proflowers at No. 1, 1800Flowers at No. 2, Teleflora at No. 3 and FTD at No. 4.

Once again, important questions are raised concerning not only Google's ability to enforce their own guidelines, but also the fairness of the existing search results. We know for a fact that the top of the most competitive search results is very often populated by sites using black hat techniques, especially the use of paid links. Knowing this, it becomes impossible to advise clients to always stay within Google's guidelines - because those that do will be disadvantaged by those who don't.

And while we know that Google can impose harsh consequences for those who step outside their guidelines, we also know that these consequences are not dealt out consistently. In both this and the previous article on JCP, the Times leads readers to believe that JCP suffered a penalty as a result of their link buys. This is untrue. Google only removed their ill gotten gains - and did not impose the kind of penalty we see when smaller businesses get caught doing the same thing. A real penalty would have made JCP unfindable even for their trademark.

And there's clearly a quality of too big to fail present here. Google's own reputation would probably suffer if they had to penalize the 4 top players in any business, since searches that did not include them would appear to be much less relevant, and their absence would be very noticeable even to the novice searcher.

Now that violators have so much to gain and so little to lose, gaming Google has become a mainstream activity not just for the large players. It's an astounding failure on Google's part to have put us in this position where their rules are not able to be consistently enforced, because the consequence is that bad behavior is being encouraged. The model has shifted in a perverse way to actually favor black hat strategies.

The fact that the discovery of questionable ranking strategies have to be revealed by a newspaper, rather than by Google's automation only compounds the failure. And the unconfirmable denials of Google's enforcement team that these obvious paid links have no bearing on ranks is truly laughable. Does anyone still believe they would all be doing this if it didn't work? The top 4 national flower websites are laughing all the way to the bank.

The real story here is the fact that the big boys are all flaunting Google's guidelines with impunity, and that tells us that Google has lost the war on paid links.

Google Penalties Now Called Manual Actions

Are we starting to see some transparency from Google in their responses to reconsideration requests? So far, we've only seen 5 examples of this version of a new response, which denies "manual actions" (read 'manual Google penalties'), suggesting that fixing the issue will auto-correct the ranking losses. We welcome this change because it includes some real information regarding the rank loss, even though we can't tell whether they're acknowledging a penalty with it. This suggests that there is at least one other response that acknowledges 'manual actions' or denies automated actions. Note how the word 'penalty' is not present.

If you've seen a different response, please send it to us in a comment.

*********************

Reconsideration request for http://www.xxxxxxx.xxx/: No manual spam actions found

April 22, 2011

Dear site owner or webmaster of http://www.xxxxxxx.xxx/,

We received a request from a site owner to reconsider http://www.xxxxxxx.xxx/ for compliance with Google's Webmaster Guidelines.

We reviewed your site and found no manual actions by the webspam team that might affect your site's ranking in Google. There's no need to file a reconsideration request for your site, because any ranking issues you may be experiencing are not related to a manual action taken by the webspam team.

Of course, there may be other issues with your site that affect your site's ranking. Google's computers determine the order of our search results using a series of formulas known as algorithms. We make hundreds of changes to our search algorithms each year, and we employ more than 200 different signals when ranking pages. As our algorithms change and as the web (including your site) changes, some fluctuation in ranking can happen as we make updates to present the best results to our users.

If you've experienced a change in ranking which you suspect may be more than a simple algorithm change, there are other things you may want to investigate as possible causes, such as a major change to your site's content, content management system, or server architecture. For example, a site may not rank well if your server stops serving pages to Googlebot, or if you've changed the URLs for a large portion of your site's pages. This article has a list of other potential reasons your site may not be doing well in search.

If you're still unable to resolve your issue, please see our Webmaster Help Forum for support.

Sincerely,

Google Search Quality Team

*********************

This response suggests that this specific rank loss is the result of an algorithmic action, or automated Google penalty, that can be unwound by simply fixing the issue.

We know from previous posts by Matt Cutts & this video that manual penalties come with a clock - a time frame that determines the length of time you'll be punished. From the penalties experienced by our clients, we suspect those time frames somehow line up with the perceived severity of your non-compliance. We've seen the time frame on newly compliant sites range from 90 days to over 6 months.

This is controversial by itself - so you get put in jail for some period of time, even if your rank loss was triggered by an inadvertent error and you've corrected it. We'll be looking at this much more closely now to determine whether that time frame is started from the fix, or from the start of the penalty, and exactly what the time frames may be. We strongly suspect they start once the site is compliant, and a reconsideration request is filed. We doubt a manual action is going to self correct once the site is compliant.

For automated penalties, we suspect you're not getting out until you're compliant, and then have to wait out some additional period before release as the bots update the index.

Google Bomb Today

Looks like Google issued their biggest update yet today. It affected us, and a lot of people we know, what are you seeing? I came up with my own list, but #1 on the list is very specific to today's update. Some of the others are well known, some of them need to be explained and are not well known.

Things I am seeing with the Google updates:

1) They cranked up the dial big time for % of exact match anchors to the target page (very bad for us) all the old blog links we bought were 100% exact match anchors, 3 per article. None of the ones we bought in the last year were like that, and often times had no/random anchors. XXXXXXXX alert got pinged for it in the last Google update, they turned up the dial which means bad news for our top interior landing pages. Overall domain authority was a negative benefactor, so even pages without any links were dropped a few spots as well.

2) Social shares is a much bigger factor (good for us)

3) Brand signals is a much bigger factor (ok for us, not good) That pushes Amazon up and other huge sites

4) Number and % of no followed links to the target page plays a very big role

5) They could stopped rankings from passing of paid blog networks

6) Number of ad units on the page plays a big negative role. (good and bad) We don't have ads/affiliate offers but a lot of our old blog links had tons of low quality ads

7) Scraper sites and low quality directories were zapped (good for us)

8) Local sites are showing in the organic results for certain queries (bad for us). I think Google will unwind this do to low quality

9) Exact match domains and long domains were devalued (good for us) bad for our affiliates and some competitors.

10) Click-through rate of organic is a big factor now (Great for us)

11) Site Speed is a bigger factor (I just popped an email to Strangeloop and EC to get Google to see our enhanced version of our site this week)

Update: 12 April 2011

The update hit mid-day yesterday, I expect most people don't know what happened yet. Anyone who had pages that were too heavy on exact match anchor text got pinged bad. Anyone with a low percentage of no-follow links got pinged. Since so many links were just devalued completely, domain authority on many sites really got hurt.

This was definitely an algo update. Keep me updated on what you see and hear in the next week.

Penalized Site Seeks Help: papofurado.com

My site www.papofurado.com was created in 2008 august, e disappeared disappeared results of Google searches on 18/01/2011, my difficulty in finding the cause is through intentionally did not do anything that might have caused this penalty. Not working with purchases or sales links, I have no links to partners, not applied any black hat technique and did not change on the site today.

The only question I could think as a possible reason would be a considerable increase of about 70% on visits in 2011 caused by visitors coming in the original articles and no attempts at manipulation, but that generated a spike in the graph analysis(which put the attachment to view) at this time the site fell dramatically around 90%.

I therefore ask for help diagnose possible reasons for this penalty from
Google!

I am available for Further Information!

Did The Hammer Come Down On Content Aggregators

You might have heard that the hammer was about to come down on a part of the web universe occupied by white hat link builders. Lot's of screaming was taking place surrounding the pending Panda or Farmer update to Google's algo. Some hints from within Google lead to rumors about the target being sites with poor content, with finger pointing in anticipation of losses for Demand Media, eHow, EzineArticles, and other 'content farms.'

And then it just happened. Sometime around the end of February 2011, the switch was thrown, and comments from Matt Cutts suggested that the change would impact about 11.8% of the search. That's actually a pretty gigantic impact. When Google makes a change of this magnitude, the search world kind of shakes a little.

The measurable hit to these "content farms" in the first few days had some suggesting that this change would be a fatal blow to the link strategies involving them. SISTRIX showed some metrics that WebProNews picked up on along with searchenginewatch and others showing massive traffic drops for some selected sites. Very scary stuff for anyone with resources in this game.

The media is playing the algo change up as a massive overhaul that is wreaking devastation and upheaval, and some of that is unavoidably being carried into the seo community as if it were true.

This doomsday frenzy is currently being followed up with individual horror stories. Some include sites with proven records of quality content - sites that are so good that their content is always copied verbatim. One of note is fonerbooks.com - having seen the heavy hand of Google before, this is completely expected. We hope some adjustments will be forthcoming in the immediate future to roll back some of the harsh treatment improperly imposed on some good sites. Algo changes always leave victims in their wake.

We, like many other professional optimizers, build content on aggregator sites for the links, so we have an important stake in how this shakes out. In principle, we believe Google has to set standards that are high enough so that the garbage doesn't rank, no matter where it's posted. We know that's not happening - garbage is everywhere and ranking. But does it make sense to drop the ranks of genuinely useful content from certain sites, just because of where it's posted?

One of the sites taking the biggest hit according to SISTRIX was suite101.com. This is a site that has a uniqueness requirement on content posted there. In other words, if you post the same content elsewhere, suite101.com takes your content down. Yet they took one of the biggest hits. How can this be? When they look carefully at exactly what happened there, I suspect they'll discover that uniqueness is not enough to determine quality. Garbage can be easily be made unique.

A check on some of the aggregator content we recently posted for clients does not reveal any significant losses. Like any content on any site, these articles are subject to some of the same mercurial forces - meaning that even before the algo change, our forensics would reveal much of this content not indexed, or indexed in the supplemental results. Over time, well written, useful articles from some of the aggregators gradually make it into the index,and actually hold rank. So far this is not changing. One of the critical pieces of this strategy is to spread your content across many platforms. Because not all aggregators are the same, and finding the ones that have their act together is something that requires constant vigilance.

And my guess is that the value of the content aggregator link building strategy is not going to go away any time soon - and it's not going to change very much for those creating high quality work. Because if Google were to begin to use a standard other than quality, no matter where the content resided, it would lead to the collapse of relevance within their search results. I also noticed that content copied from our sites are still holding ranks for the copiers, something I was hoping to see change. But this still makes sense - the fact that people copy and reuse quality content only proves its stature, and that stature very likely will continue to be rewarded if the evaluation is really based on perceived value.

Our recommendation is to continue to put resources into content aggregator programs used for relevant link building, provided the content created meets a quality standard that is rigorous. The key is and always was quality. Content is still king.

Update:
We are finding other sites that support similar views, like nichebot and potpiegirl

Enterprise Search Manipulation

Whatever your view of seo, one thing is true - "search engine optimization" is simply a more acceptable term for "search manipulation." Advancing the search performance of a website regardless of the technique is essentially manipulating the ranks higher.

I think it's important to remove any distinction because many seos mistakenly believe they serve a higher cause connected to an ethical responsibility to abide by search engine rules - and that when they comply, they are more 'optimizing' than they are 'manipulating.' Clearly there is always an ethical responsibility to others, including one's clients, to act only in their interests and do no harm. But be careful not to let search engines form the basis of your ethical standards, because they are already failing in that regard.

Consider the ethical appropriateness of:

- imposing harsh punishments based on secret laws
- an unwillingness to even acknowledge when a site is penalized
- being the sole arbiter of justice with a stake in the decision
- permitting no recourse
- creating victims through Google's own frailty
- penalizing sites in Google for the actions of 3rd parties
- rules that change during the game
- the absence of an effective warning mechanism

It's pretty clear that Google is not setting the ethical standard we can believe in.

Of course, Google's ethical problems don't excuse unethical behavior, but there are times when acting in your best interests (or the best interest's of a client) may conflict with the imposed guidelines of a search engine. Consider this recent NYTimes story involving JCPenny's very high ranks on consumer terms just before Christmas 2010, obtained by some very robust link buying strategies. Here's an excerpt:

The company bested millions of sites - and not just in searches for dresses, bedding and area rugs. For months, it was consistently at or near the top in searches for "skinny jeans," "home decor," "comforter sets," "furniture" and dozens of other words and phrases, from the blandly generic ("tablecloths") to the strangely specific ("grommet top curtains").

This striking performance lasted for months, most crucially through the holiday season, when there is a huge spike in online shopping. J. C. Penney even beat out the sites of manufacturers in searches for the products of those manufacturers. Type in "Samsonite carry on luggage," for instance, and Penney for months was first on the list, ahead of Samsonite.com.

With more than 1,100 stores and $17.8 billion in total revenue in 2010, Penney is certainly a major player in American retailing. But Google's stated goal is to sift through every corner of the Internet and find the most important, relevant Web sites.

Does the collective wisdom of the Web really say that Penney has the most essential site when it comes to dresses? And bedding? And area rugs? And dozens of other words and phrases?

By running a SEMRush report showing the lagging results in the top 20, you can still see how huge this manipulation was. #1 for category level targets like dresses, lingerie, home decor, bedding, swimsuits, dress, slipcovers, bedspreads, lamps, comforters, sofas, quilts, cribs, skirts, womens clothing - download to view all 97,000 ranks in the top 20 here.

As a result of the reporting of the NYTimes exposing an outrageous breach of Google's guidelines, all these incredible ranks disappeared, but only after a "manual action" by Google. JCPenney denies responsibility:

"J. C. Penney did not authorize, and we were not involved with or aware of, the posting of the links that you sent to us, as it is against our natural search policies," Ms. Brossart wrote in an e-mail. She added, "We are working to have the links taken down."

The traditional seo community is buzzing with this story, and whether they're pointing fingers at JCPenney, Google, or the seos involved, one thing is very clear: This was an amazing success - a truly impressive feat that will be emulated over and over again. Because there is no doubt that this was also a huge financial success - absolutely worth bearing the very minor consequences of the "manual action." The lesson for seos is that when your client is big enough, the risk/reward analysis for black hat tilts toward the reward.

Further, while most professional seos may be seeing this as proof of the dark side, there's an even darker side that's being ignored. If this massive search manipulation went undetected until exposed by the NYTimes, isn't it obvious to everyone that there are sites right now, holding lots of very valuable ranks for similar reasons - using strategies outside of Google's guidelines, and remaining undetected? And succeeding at the expense of the search compliant players.

And what about this 'manual action' that Google claims to have implemented as punishment? MANUAL action? Isn't this proof of failure? Here's a company with massive resources aimed at automated rank detection and guideline enforcement. But if even they can't catch the biggest guys breaking their rules, doesn't that harm those who play by them? As JCPenney's seos manipulated all their targets onto page 1, that pushed the same number of other sites off page 1. Another example of harm done to sites playing by the rules. We see this as Google's responsibility, but when it happens, and it happens frequently, Google usually punishes the perpetrator.

I ran the forensics on JCPenney.com to see if there was any evidence of a domain level penalty, and found no obvious suppression. All trademark and related searches show #1 - so the "manual action" was merely a slap on the wrist. Kind of reminds me of the big banks getting away with fraud with the help of our government refusing to prosecute. Of course, there's no proof (is anyone even looking?) that JCPenney was involved, and they're denying responsibility for it. For all we know, it very well could have been a third party trying to get them penalized by pointing a boatload of links at them, right? Except for how rewarding it was, and the absence of any penalties, some might even believe that. But that's basically what they're claiming.

This leads right into my final concern about this story as it relates to the size of JCPenney. I see a lot of penalties, and there is definitely a correlation between the offense and the punishment. If you successfully scam Google's system, you will be harmed upon discovery, so the wrist slap here to JCPenney just doesn't cut it. If one of my clients were to be nailed for this strategy (as they have been) the consequences would almost certainly be much more severe, much more long lasting, much more painful. But like the too big to fail banks, JCPenney gets a pass. Anybody surprised by that?

Google Has A Huge Cloaking Problem

I've run into this so many times - often connected to penalized sites - that I know it needs some kind of extra push to get Google to fix it once and for all. Maybe embarrassment will work. Please, someone, put this post in front of Matt Cutts.

Here's the problem. It's very easy to get cloaked pages ranked highly in Google. So easy that there are syndicates selling advertising presence in the searches of some high profile terms.

A quick review of what a cloak is - that's when a site shows different content to the search engines than to other visitors. It's very easy to identify a search engine bot by ip/name and conditionally serve files. Basically the cloak fools the search engine into thinking the content is something other than what it really is.

Now since this is a penalizable offense (will get you banned), the perpetrator never wants to risk his own sites. So most of these cloaks involve hacking an insecure infrastructure (like almost any .edu) and using that as the platform for the cloak.

Today (21 January 2011) you can see a perfect example of a successful cloak insertion into Google's search results. Do a search for "high roller online" and notice there are 2 .edu sites in the results (see screenshots below). The one that's #1 is from columbia.edu and is being pulled by the cloaker (only shows a php file at this time, but held the #1 rank for over a week). The result at position #3 is from ucsb.edu and is the money maker for the moment. (click images to enlarge)

Here's what Google sees - what the cached link reveals: (click images to enlarge)

So Google thinks this is a page from the history department at the University of California. But the visitors who click the search result go to one of six different casino sites - better promotion than any normal rotating ads presentation because of where this is occurring:

Keep in mind that this is happening at the TOP of searches - in this case claiming 20% of the page 1 organic search real estate. But even more valuable searches are often showing 3 or more such results on page 1.

Now why should we care that this is not only possible, but also happening with great regularity? We think all legitimate sites should be outraged at this, and therefore at Google. Because when 2 sites come onto page 1 via cloaks, 2 legitimate sites fall off of page 1.

Not only that, but often the rank is actually hijacked from some innocent site. Since it would be prohibitively difficult to insert a high ranking result from scratch, most cloaks actually take the rank away from someone who legitimately holds the position.

And when that happens, often the site that lost its rank also gets penalized by Google. Let me repeat this: victims can not only lose their ranks in Google as a result of the cloak, but can lose all their legitimately held ranks because Google blames them for the problem. In the past year we observed 5 separate instances of this. Up until this year, the victimized sites got penalty free almost immediately after reporting the hack. Right now, we know of at least one instance where the victim site remains penalized, even though the cloak was removed. The hack, by the way, is of Google's infrastructure via the search, not a hack of the victim's site. So it's a kind of indirect attack that has the effect of harming the business directly.

My biggest complaint about this is that the fix is so obvious and simple if Google chose to act. Just grab the page one results only (as a way to limit the resource outlay) two ways, once as Googlebot, and once as an anonymous user agent from the same geo area. If the two files don't match have a human review it. I believe this would catch every instance of this kind of embarrassment for Google. The question is why hasn't this happened already? The answer is obvious - they're only going to act when there's enough outrage. Victim sites should consider this: There's an argument to be made that Google is complicit in the malfeasance occurring within their search if they are permitting this to continue when they clearly must know about it. My clients have already reported this problem countless times via reconsideration requests and spam reports. So there's really no excuse any more.

It's very likely that by the time you read this the cloak I'm showing here will have been taken down. But it's also very likely that it will have been replaced by another.

24 January 2011 update:

I'm getting a lot of questions about how to protect a site from this. The answer, unfortunately, is that you can't - this is Google's problem. The attack is not to your site, so no amount of local robustness will prevent it. The fix is to get Google to make their search results more robust, and especially to pay more attention to cloaks in general. If you think this has happened to your ranks, point to this post in your reconsideration request. We need to get their attention onto this problem.

24 January 2011 update II:

uscb.com has found and removed the hacked page on this search ("high roller online") and on "high roller gambling" where there was a similar cloak result. The important point here is that the school took down the hack. It wasn't Google that fixed the problem - the problem still exists and will be evidenced again, count on it.

A Sorry Tale of a Google Penalty in Action

(And what we are doing to help us and others do something about it)

Last May, my new online news aggregation business www.onenewspage.com, suffered a steep loss of traffic. In the space of just a few days traffic collapsed, falling from about 18,000 daily visits to around just 700, a fall of over 95%.

Our Google page rank took a similar tumble, going from PR 4 to PR Zero. And a search on Google for 'onenewspage' saw the results move from the first page of its SERPs, to pages five or six.

In short, our site had been penalized by Google. Traffic from Bing, Yahoo! and other search engines remained unaffected; in fact it continued to grow nicely.

But our problem was that we didn't know what had triggered Google's penalty. Thus began many months of sheer frustration as we tried fruitlessly to find out.

To cut a long and sorry story short, we were given an entree into the gloomy, Kafkaesque world of Google, where sites with severe traffic loss make endless tweaks and changes to their content and site architecture in the vague hope that one of these tweaks or changes might release the site from its penalty.

Let me be clear, we saw no reason why our site had suffered a penalty. We had read and understood the Google Webmaster Guidelines and made sure our site complied. But to be absolutely certain we hired an SEO expert to give the site a thorough review. He gave us the all clear, having found no technical causes for a penalty.

Every change we made to the site to help reverse our situation we'd document in detail, and then file with Google via its reconsideration request mechanism, pointing out why we thought our changes might release us from penalty prison.

Each email received a standard short reply with words to the effect that Google had noted our request and the site had been reviewed. If only we could have spoken to a Google representative and had the true low down from the horse's mouth. But shadowy Google doesn't give out numbers or enter into a dialogue like this with its users, more's the pity.

After each request to Google we waited in hope - but precisely nothing changed. Our Google traffic remained in the doldrums, our PR rank stuck at zero, and our own name search still marooned to pages 5 or 6 of the SERPs.

After five reconsideration requests and a lot of heartache, we finally asked our SEO expert to put the request to Google on our behalf.

You can probably guess the outcome. A long and detailed (and expensive!) email was drafted on our behalf and received exactly the same automated reply from Google.

Ultimately, after seven long months we decided to go public with our problem and did an interview with the enterprise editor of the Daily Telegraph, one of the UK's most respected national newspapers.

Whether coincidence or not, a month and a bit down the road we could see our Google traffic returning.

Because Google refuses to define or even acknowledge its penalties, we felt that there should be much better transparency from Google when the changes it instigates lead to dramatic collapse in a site's traffic.

Taking a deep breath and plunging bravely in, a month ago we decided to launch an international campaign. We are appealing to others in the same boat to swap stories. If enough fellow sufferers were to come forward, we could collectively exert pressure on Google to be much more open about its penalty system. Our campaign has a new website - www.haveibeenpenalized.com - which is collecting case studies of web based businesses that have suffered penalties but have been unable to enter a dialogue with Google to find out the nature of the penalty.

We are calling for Google (and in fact all search engines) to introduce three simple measures:

* First, to bring in a simple mechanism that tells the site owner that they have suffered a search penalty

* Second, to establish a communication mechanism which allows site owners to find out more about the nature of the penalty

* Third, to instigate a fast and efficient appeals process if site owners wish to challenge the penalty

We are already getting some momentum and we are starting to build case studies. But we need many more to help convince Google to change the way it operates its penalty system.

If your site has suffered a similar fate as www.onenewspage.com, please visit www.haveibeenpenalized.com and share your experience. Remember, the longest of marches starts with the first step.

A New Google Penalty

There's a trading technique used frequently by professional traders (high frequency trading - HFT), where computers exploit both price movement and the trading rules to gain an edge. And the big money has spawned trading firms whose sole strategy is using technology to generate massive numbers of automated trades to gain advantages over their competition.

There's an seo technique used frequently by professional seos, where computers and low paid offshore employees post massive numbers of links on blogs comments, forum posts, etc. And the big money has spawned seo agencies whose sole strategy is using technology to generate massive numbers of spammy links to gain advantages over their competition

In both of these worlds, the common thread is automation gone amuck. In the traders' marketplace, the individual investor is disadvantaged, and the markets appear to be manipulated by the monied players. In the search world, the sites that pursue genuine relevance are disadvantaged, and the search results appear to be manipulated by the monied players.

It's a sad state for everyone, when the best strategy becomes the one that games the system the best.

And for us, at least for the moment, it means that Google is in desperate need of a change - both in ranking protocols and in disincentivizing garbage links.

These two stories are really related in a much more serious way. Both the markets and the search are model environments where honesty and truth should be rewarded. In both worlds, the system is disadvantaged if monied players can buy an edge, and the if the existing rules are enforced, that should not happen.

But it has already happened in both worlds. And the citizens are waiting for enforcement on both fronts to correct the corruption of these systems. The markets will require a revitalization of the currently captured regulatory bodies like the SEC - read bureaucratic, long lasting problem.

Unlike the markets, the search only requires a single entity to act, and we think we've already seen the opening salvos from Google in the form of a new penalty.

In a sense, Google is responsible for the problem. Once the world learned that links pushed rank, entire industries were born to generate links. How many link schemes have you been sold lately? From paid links to auto submit software that can get beyond the captcha wall and automate link insertion on blogs and forums, website owners are bombarded by link building sales pitches.

We noticed that in Europe and especially the UK, there are SEO agencies that do ONLY this kind of link building, and if you read the blogs and brags on their sites, they openly discuss it as a valid technique that's working for their clients. This is why one might say that Google has lost this battle. And it's related to another lost battle, that of paid links - since it is impossible to know the motivation of a link on a web page. Without understanding that motivation, the value, or trust behind the link is unknown. And that trust or vote was a primary motivation behind rank in Google's algorithm.

Starting late 2009, we started seeing penalized sites that did not conform to the typical non-compliance issues. Basically, sites with no technical issues at the domain or server level, and no structural or traditional link issues were receiving penalties that were not removed until their link profiles were cleansed.

This is a clear signal that consequences are being meted out, however randomly. We know of many large European sites that have outrageous link numbers - in the tens of millions, the majority of it spammy, that are still within the index, while many small sites are getting hammered for a few thousand links.

This is from Bruce Clay's 19 Nov post:

"Internationally, link spam is often just "how SEO is done". Google grudgingly turns a blind eye without a serious attempt to detect it and levy penalties, because to enforce their link guidelines would be to leave their index barren, or because the language and technology issues are too expensive to warrant action. In many countries, link spam is openly discussed by the local SEO firms; in some cases, it is openly advertised by publicly traded SEO firms as the only SEO activity that is necessary - these firms do not even try to improve on-page quality because it is not necessary. Google is losing the international link spam battle and the top ranked site is often the site with the best spam instead of being the most relevant. So far Google has chosen to let the international sleeping dog lie but that will not last."

That dog has actually mauled quite a few websites this year, all over the world.

And we believe that risk of harm is greater in the US. We oversee seo agencies here and abroad for our clients. Our own overseas clients are often reluctant to cease the spammy link builds, because it has been working for them. But we're seeing more penalties over there lining up with our suspicion, and we're advocating for better link practices among all clients just to get it on their radar. In the US, you put your enterprise at risk if you initiate a spammy link campaign on a young site, because one of the penalty triggers appears to be related to the relative numbers of garbage links in the your overall link profile. On older, larger sites, here in the US, these spammy links tend to do no harm but really do not push rank even when indexed in large numbers and included in Google's Webmaster Tools. So there's no longer any incentive to use them - yet the marketing of them powers on.

The enterprise is required to act conservatively, so we have always steered clients away from these kinds of links. But for many site owners, the message is coming late. They either still have teams in place obtaining these links, or have a large inventory already pointing at them. If this is you, there's a new risk to your business - a risk due to the nature of your link building team.

To preserve search integrity, Google has to act. Either it migrates trust away from links, or it devalues certain kinds of links - profiling as it were. This would no doubt create harm to the communities devalued. Whatever the consequences, and believe me they are unknown, changes are already underway. And the accompanying changes Google must make to preserve the search value of its index are making charlatans out of previously successful seos.

There's a new Google penalty you might be at risk of if you buy the charlatans' talk.

A Brute Force Attack May NOT Be A Brute Force Attack

This post references my previous post, where a client's dedicated servers were hacked.

The security team at the host claimed to see numerous brute force attacks that were successful. In other words someone put a bot on the log in form and cycled through ALL the possible characters for username and password until the right combination was hit. Supposedly, they could do this because cpHulkd - which looks for multiple log in failures and blocks the ip - was not enabled.

But upon thinking and reading about brute force attacks, we are scratching our heads. An eight digit password should take 2 centuries to cycle through all the possibilities, and ours were at least 10 digits. So it's not possible, unless the hackers has incredible technology, or we got faked out

This is where our connection to the dark side pays off. A long time ago, we got a very big hacker client out of a Google penalty, and to show us his appreciation this client has kept in touch with us, explaining the hacker perspective on all kinds of security issues. When asked if it is possible to speed up a brute force attack, he responded:

"No one uses brute force except idiots. Since the logs don't lie, you are misreading. Probably someone scored a bunch of username/password pairs from your desktop or emails and just hit your servers until they got in. If they have enough ips they will always succeed. The first failures will make it look like a bfa but it's much worse than that. You have a security breach somewhere."

Got it? Just because you see a bunch of failures preceding a successful hack, does not mean a brute force attack. In fact if your password was at least 8 digits, and the hack succeeded you can pretty much rest assured that it WASN'T a brute force attack.

The problem with misidentifying a successful hack as a brute force attack is that you put your security in the wrong place. If someone is able to hack your admin level entry, it is most likely stolen username/password pairs that got them in.

What these hackers do is look only for the username/password pairs - they don't spend time looking for what they're used for, although I'm sure they'll take that as well. All they need are enough pairs, enough resources (ips), and your log in urls. It's the ENDPOINT you need to protect - your email, your desktop, your files and data.

We Suspect That Someone Paid dz.z3ro To Hack Our Clients Servers

In the very early hours of Saturday, 21 August 2010, several dedicated servers related to one client were hacked.

For starters, there was an avoidable vulnerability that was exploited. The servers were newly implemented and did not have cpHulkd enabled. This is the "brute force manager" that locks you out after several failed attempts to log into WHM (WebHost Manager). Repeated failed attempts from an ip results in that ip being blacklisted. If you're running Apache servers this should be a default.

Because of the nature of this client's work, there are powerful, monied interests that have become his adversaries, and who stand to gain by crippling his sites. While we are aware that paranoia runs deep on all things internet, the stakes involved in this one are so high that it is in our client's interest not to be complacent. So yeah, a little paranoia is a good thing.

For this reason, and because we love this work, we spent the weekend reverse engineering the attack and running forensics on the trail.

We woke up to this message posted on many sites this client owned, and some others that were related:

At first sight, it's lol. Nice prank, frat boy. This guy spent a lot of time on this display, and although we only saw this logo, there was an audio/video component that we didn't see - because the French site they had hacked to store the flash & MP3 files had taken those assets down already. This is a proud hacker, folks! Wants the glory.

In case you can't read the banner, here's the text right from their code:

Contactez-Moi A:
DZ.Z3R0 [AT] GMAIL [DOT] COM
DZ.Z3R0 [AT] YAHOO [DOT] COM
DZ-Z3R0 [AT] HOTMAIL [DOT] FR
Pseudonyme de Skype: DZ-Z3R0
Site Web: www.dz-z3r0.com

He or she openly posts a French "Contact Me" message (appropriate for Algeria), a business card, with contact numbers and email addresses on the three search engine's mail systems - and that's where we get paranoid again. These guys are hackers for hire! And we suspect someone hired them.

Because of this text, the hacker actually ranks the attacked sites for the search for "dz.z3ro" thereby getting even more publicity. Assuming these sites have not yet detected the hack, and if enough sites are hacked, there's bound to be plenty of them.

The attacks were coming from these ips
41.200.172.33
41.97.64.186

The ips are Algerian, but always anticipate proxies and a head fake pointing in the wrong direction.

(If you ever have to run forensics, use reliable tools. There's a lot of garbage out there that ranks high: search for "find country by ip" - #1 is selfseo.com/ip_to_country.php, which says our ips are from Japan.)

Do a search for "dz.z3ro". The #1 result takes you to www.sharmakay.com. Searches on the whois data look innocent - the address exists, Sharma appears to be a real person there, so it's likely a hacked site:

Registrant:
AUD Family
17150 University Ave Ste 300
Sandy, Oregon 97055
United States

Domain Name: SHARMAKAY.COM
Created on: 24-May-00
Expires on: 24-May-14
Last Updated on: 18-Mar-06

Administrative Contact:
Kay, Sharma
AUD Family
17150 University Ave Ste 300
Sandy, Oregon 97055
United States
5037300203 Fax -- 5036687722

Domain servers in listed order:
CWPRO1.CROSSWINDS.NET
PRO.CROSSWINDS.NET

And it displays this:

The Algerian Hacker Team seal of approval, bar codes and all. And look at all the hacked sites ranking because of the hack!

Server log tells the story: They brute forced the password on WHM and then did different things on different sites. The constant was the posting of logo.png, and flag.png, plus the script on both index.php and index.html. In some instances they also renamed the index file and left backup.php, a little toxic code we recommend you not download. So far, it appears that's all that was done, and recovery is just replacing a wiped out index file, and removing the detritus.

Technically, they did more than just hack the password, delete, post & rename files. They also did some neat tricks - like running wget to directly pull & post the files from another server they had hacked. So there's command line knowledge behind this.

The script also appears to disable left and right click, but I haven't put time into that yet. Will get to that - we already know way more than we can reveal here about the players and their connections.

But some of what we can reveal is still very interesting. For example, this hacker's very busy - just looking at all the hacked sites in the above search tells you that. Here's another trademark image from a June 2010 hack:

This is a report of a different kind of attack by the same entity, and supposedly tracks back to Nigeria, according to the victim's post http://www.nairaland.com/nigeria/topic-469034.0.html. So it looks like dz.z3ro has at least 2 tricks up that sleeve - brute force and one other - possibly RFI (remote file inclusion) or SQL injection. Not high level chops by any means, but enough to practice graffiti.

Not sure how fearful to be about a flagrant hacker who demands such attention - my gut says this is a bozo, but any bozo who can hack a server is someone to watch, especially if someone is paying for services rendered.

Update I 23 August 2010: Gmail Intrusion

Today, our developer working on the compromised accounts discovered a notice in his Gmail account - warning of an intrusion into the account originating in Algeria:

Algeria (41.200.164.37)
Algeria (41.97.64.186)
Algeria (41.200.173.123)
Algeria (41.200.165.201)

Of these ips, 41.97.64.186 is also where the attack on one of the servers originated.

Couple of things here - One, did you think your Gmail accounts were secure? This is a big question, and the answer here is "NO!" And secondly, although we were told a brute force attacked succeeded on the server, this suggests is was stolen credentials. Looking into this and we're still trying to tie down the timing.

Update II 23 August 2010

dz-z3r0.com is not hosted.

Just tried to contact hacker using the 3 email addresses posted in the hack:

Got failure to deliver on all three email addresses

[DZ.Z3RO@gmail.com]:
74.125.65.27 does not like recipient.
Remote host said: 550-5.1.1 The email account that you tried to reach does not exist. Giving up on 74.125.65.27

[DZ.Z3RO@yahoo.com]:
98.137.54.237 failed after I sent the message.
Remote host said: 554 delivery error: dd This user doesn't have a yahoo.com account (dz.z3ro@yahoo.com) [0] - mta160.mail.sp2.yahoo.com

[DZ.Z3RO@hotmail.com]:
65.54.188.94 does not like recipient.
Remote host said: 550 Requested action not taken: mailbox unavailable. Giving up on 65.54.188.94.

Given this, feeling less paranoid - the email and domain now seem to be just empty brags. DZ, dude, like just when we were starting to respect you it turns out you're just talk...

And you're just like the rest of the graffiti artists I found in this nice big archive of hackers and their victims, along with some very cool hacker art - click the mirror links: (this list is growing fast)

http://www.zone-h.org/archive/published=0/page=1

(note that on 2010/08/24 ViRuS Qalaa is listed as having hacked Google.ae - United Arab Emirates - but they're now back online.)