Without naming the blog (this is now under further investigation, so if you know the site, please don’t mention it in the comments, etc.), I’ll say it’s a popular, revenue-generating blog with a very respectable following and technorati ranking.

We didn’t build the site but were familiar with its guts - we do a lot with similar blog software and had recently upgraded the site to WordPress 2.5.1. The warning, though, was new to me. I don’t think we’ve ever had a client that got flagged by Google.

1. To make sure we knew what we were chasing, we verified the site and checked its status in the Google Webmaster Tools dashboard. The site overview told us that day’s crawl found intermediary links to malware. The list of pages it encountered with the link included the front page, so the issue didn’t seem to be coming from a comment.

2. Viewing the generated source (consider this another endorsement of the Web Developer extension for Firefox and Flock) showed a suspicious iframe being included just inside the body of any blog page. The source of that frame was an offsite link to a malware drop. In other words, the site had been hacked.

What do you do if your site is hacked?

3. I’m not the world’s best hack0r chaser. In fact, when I took off to find the source, I thought I was looking for a simple document.write or crazy hex string somewhere in the hundreds of files / thousands of database entries involved in making the site go.

I checked the wp-posts and wp-comments tables. I disabled the plugins. I turned off the javascript ad calls. All this assuming someone was leveraging a WordPress security hole or injecting code through an ad server. Incorrect.

4. We downloaded all site files and did a local search on the project files starting inside the theme and moving outward. We didn’t find anything.

5. After talking a bit more to the site owner, we decided to just carpet bomb the javascript - remove all scripts on the live site that weren’t already checked. The iframe disappeared.

6. After going through each file manually, we found the hacked code buried inside a script that called Flash to handle some typography on the site. It could have been any where, but that script probably seemed about the least obvious place to attach it. And he / she was right. We missed the eval(unescape(’%64%…’)); on the first pass.

How the hell did that get in there?

7. It’s probably worth mentioning we didn’t have shell access to the server, and we had to ask the web host to send us the access log. When we got it, I was a bit surprised to see the hacker got in via SFTP. They had the password.

8. We changed the passwords to the site, the databases, and to WordPress. Then, 9. went through each file uploaded by the ip address the hacker used. A contact form needed to be re-written. Some WordPress includes were replaced. A few other files were trimmed. A robots.txt file had to be corrected.

10. Now that the site was clean (we hope), we had to take care of the Safe Browsing flag at Google. We requested another review in the Webmaster Tools dashboard. It took about 10-15 hours, but the crawl came back clean and the warning was removed. That was early Sunday morning.

11. The next step is a fresh WordPress install (it’d be great if 2.6 came out of beta in the next day or so). This will remove any doubt that we missed anything nasty outside of the theme or content folders. The web host will need to get involved as well. I’m not sure what kind of monitoring is currently in place, but it can always be improved. All passwords on any related applications will need to be changed.

What they were trying to do.

I’ve seen plenty of malicious code injections. The beta launch of Truemors alone resulted in 3 or 4 different ways clever kids can drop some text into a site. But this one was nasty. Judging from everything we found, it’s pretty clear the perp’s were trying to nuke the site’s ranking in search engines in addition to everything else.

1. The iframe sourcing malware was a blatant flag to Google.
2. An additional page was set up in an attempt to host malware. (another flag)
3. A contact form was hijacked.
4. A script was added to allow a quick password change. (to lock out the site owner.)
5. A robots.txt file was modified to disallow bots.

That last bit (blocking search engine crawls with a robots.txt file) is pretty telling. And scary. Someone was really trying to damage the popularity of the blog, and, from what we can tell, this was just the start.

My point in writing this rather than just twittering a bunch of zomg blurts was to show how quickly someone with proper motivation can damage your livelihood.

In this case, the site owner didn’t have a dedicated team that could immediately correct the issue. He was able to call in a few favors, but it still took him about 36 hours to get turned around (including the review by Google). It could have been longer had Firefox 3 not shipped with browsing alerts turned on by default, and it definitely would have been worse had he not had a few pals familiar with his setup. (Not that I’d rank high on the list of heroes to call.)

The take away here is this: you need to lock down anything you rely on. Keep your passwords clean. Keep your software current. But, more importantly, make sure you have a few emergency contacts. And, pay attention to Google.

This could happen to you. Especially if you’re kind of a big deal.