Hours prior to to Geocities closure, Reocities was born a project to archive its content, it got a lot of twitter time including myself and they manage to scrape a hefty chunk. However unknown to them Yahoo had been working to save Geocities through archive.org

For legal reasons Yahoo felt they could not wholesale dump Geocities content to archive.org but did provide the site with every assistance to allow them to store and and access all publicly available sites at time of closure, they also allowed their bots unlimited connections and bandwidth to grab as much content as possible.

Why didn't they publicise it?

Good question, but they were quite keen when I asked that the message was made clear they had worked with archive.org and that this was a long term plan, as David said back in June. With archive.org not being indexed however it does mean a large part of the web has still vanished from day to day searches and so potentially their still is a place for projects like reocities.com.

The posts are now all being stored in an archive on timnash.co.uk and both the original blog and the domain name venture-skills.co.uk redirect back to timnash.co.uk sadly I haven't had a chance to go through them and tidy up the code so they don't always look fantastic. I have also turned off comments on the posts as they are meant to be truly an archive, its sad but true that the Venture Skills Blog received more hits then timnash.co.uk even after nearly a year without an update so it will be nice if some of the new visitors to the site stick around.

If you had a favourite post what was it and why, I hope I brought it across to the blog. When deciding what to save I chose the top 10 posts based on visitors and then the top 10 of my favourite posts from the rest followed by the most posts by comments.

Last week I released some of the statistics from a project we worked on earlier in the year that revealed 92% of people use the same password for their email as they do for other sites. One of the more interesting stats was actually from the follow up survey where almost a third of Hotmail users believed their accounts had in the past been hacked.

Over the weekend many of them were, along with Yahoo and Gmail accounts, and the emails and passwords were published. They are easy to find on the internet with a few well chosen google searches.

This led me to a great way to promote my message of change your password.

Password Searching Service

My idea is to create a small app that lets people search to see if their email has been compromised, the application asks for their email and for security (and double opt'in for possible future mailing) requires they log in and confirm their email address, when they click the link. The system searches using a couple of google searches for possible passwords, and retrieves any it thinks are passwords that are associated with the user, it then displays these along with some randomly generated passwords on the screen.

The system will never know if it got the password right (the biggest issue with it) but it would provide user with extra confidence. Regardless the page would also leave a message telling them to change their password. If the system returns no results then it tells them such but suggests changing password to be on the safe side.

The question is would such a system be legal?

Expanded, reason I ask is because while the passwords are floating on teh web, to extract the users potential password would mean the system would have to access and parse the contents which are "stolen", of course the quick way to do this would be to store the lot but thats a quick way to a cell I would think.

Sometimes watching television can be frustrating and I don't mean because of the silliness of CSI awesome computing and forensic capabilities No what I find silly is the way advertisers try to leverage the web. So I would like to introduce you to a case example.

Park Hampers – Park Hampers are a Christmas saving scheme based in the UK they work by you paying for your hamper in the weeks months prior to Christmas when it comes to Christmas time they deliver your hamper. Its aimed more at low income families or people who are terrible at saving. Their SEO team should be shot!

Their recent TV campaign tells you to visit park11.tv this is of course a redirect to a truly terrible landing page, but it is not the landing page that concerns me but the fact other then this URL there is no mention of Park11 or Park11.tv anywhere on there site this results in a Google Search for Park11 being:

Christmas Hampers? No. Parkisons disease for Christmas? No thanks

What has gone wrong for Park?

Park obviously want to track number of hits their TV campaign is bringing to their website so have created a unique url for that campaign the aforementioned park11.tv this I assume logs the user before redirecting to the landing page. Park have numerous other URLs leading to various landing pages and mini sites so this seems to be part of an overall strategy the problem is that as the unique url has diversified away from the core brand they haven't laid any foundation on their site to cope with it.

So they don't show in a Google search for Park or Park11 so what?

How many people put urls into Google not the url bar on a web browsers? I would suggest quite a lot especially those people who perhaps are not as web savy which are in part the customers Park Hampers are aiming at. So by having a sponsored listing they are simply losing traffic and what traffic they are picking up they are paying for via PPC. Not only are they paying for an advert on tv they are then paying to pick up the traffic from that advert!

In the case of Park they have a small reputation issue as well, while badly researched by them Park11 association with Parkinsons can't be helping as users arrive at Google but find no results for hampers they do read a pile of scientific jargon if that doesn't put the user off they may well visit the wikipedia page on Parkinsons. Which to be fair I think every one should so here is the link again.

How do you solve the problem

First off all sorting out the rest of Park Hampers site so that its pages are ranking, then reference Park11 campaign create a dedicated landing page for it with the keywords in use and at least internal navigation pointing to the site. Just to be safe surely you would optimise for keywords such as Park11 & "Park Eleven"

Park are not the only company not to do this, but since they are on tele right now advertising I thought they were the perfect example.

Update: Myself and Carolyn were bored over the weekend and so have been playing with look at Park Hampers redirect strategy to say its a mess is an understatement, they own park1.tv through to park40.tv and seem to be using most in various tracking, about 20 of the Google results for park1-40 actually have Park anywhere in the top 10, examples of other Park redirects include park5.tv and park15.tv. This itself wouldn't be that bad but in addition Park own another 8 or so domains including myparkmag.co.uk, gopark.co.uk etc these are not minisites but rather badly designed redirects. In short a mess, don't fall for the same mistakes.

92% is a scary statistic here how we got it

Over the last 6 months we have been working with two clients to experiment with their security authentication methods our first issue was to see if what the issues was, one of the questions asked was do people use the same passwords across multiple sites.

So we set up a very simple test using several websites registration processes, we identified users who's email address were yahoo, gmail, hotmail and a few of the smaller free email providers. Next we added a special terms and condition box (this was in addition to the normal terms and conditions) which they needed to opt into but was not defaulted to nor a requirement. We tried very hard to make sure what we were going to do was utterly transparent, to our surprise the opt-in rate was nearly 70% we can only assume people were blind clicking.

Why do we think they were Blind Clicking because they just agreed to:

Give xxxxxx permission to attempt to login to your mail account using the details you provided, no mail or contact details will be collected and no personal identifiable information will be stored about this attempt. Please be aware allowing this action maybe in breach of the terms of service for your email provider and could cause discontinuation of your service. xxxxxx will not be held liable in such cases.

We then provided a link with more explicit information on what we were planning on doing and all the legal arse covering bits. now we had just over 2000 "volunteer" that link was clicked just twice and one of them then went and agreed to it! Sadly as the data is dissociated with the account we have no way of knowing if that person did it because he knew it would be a failed login or not.

Once we had their permission we used a simple bot to attempt to login, storing successful logins in a database identifiable only by an ID and the mail server we never stored who's email it was which is just as well as it would have been a privacy nightmare.

Our Registration pages were split into two types, one that required at minimum a weak password and ones that required alpha numeric password of more then 8 characters.

Finally we surveyed 1 in 10 about their email and password habits.

Gmail users the worst at password protection

With a little over 93% of the passwords working with their Gmail accounts, it would appear Gmail password users are the laziest of those we tested, though the figure dropped to 89% when a stronger password was required.

Yahoo Mail users have shocking memories

That's the conclusion we reached as they were the only user group where the secure password sites had a higher % of successful logins then the weak account 91% vs 90% which is strange because it has only been recently that YMail has had any decent requirement for password strength.

Hotmail passwords most secure, surely not!

It's true Hotmail users came out best but we are pretty sure we know why, in the follow up survey almost a third of Hotmail users claimed they believed their account had been hacked.

Oh and users lie about their password habits

We all know security and regularly changing and different passwords is important which is probably why only 42% of people asked admitted to using the same password on both email and the site they registered on.

So couple of take away points, people really really do not read terms and conditions and for god sake use a different password for your email to the one you use to register at free sites!

Go change it now...

update As if to reinforce the point news at thousands of Hotmail passwords being posted online is announced.

The new API allows:

• Retrieve, create, modify, and delete pages and content.
• Upload/download attachments.
• Review the revision history across a site.
• Display recent user activity.

While at first glance this doesn't sound breathtaking and to gain multiple Google Page sites you need multiple sites and multiple API keys this is the first time a "free" provider has made it quite so easy to automate parasite sites not hosted by the SEO company. And while Google may well de-list individual Google Pages or ban that user they are not going to ban their entire system meaning that its one of the few sites where you are sure to be able to from the off create nice clean sites in the eyes of Google with no worries about past history.

Ideas for content Network Developers

My first thoughts was to grab all those nice article directories and republish them, or indeed a bit of desert scraping, automate the search, Xpath the content and automate the uploading after the initial setup you could simply insert keywords and let it go, for multiple Google accounts Mechanical Turk provides a simple and cheap way round the CAPTCHAs.

Just food for thought, of course these low quality sites would be terrible for users and near worthless for search engines so expect poker, viagra and other competitive niches to be abusing them soon.

The SEO blog of course died or should that diversified into a blog about "stuff" but it was a little over two years ago I accidentally got myself blacklisted by Google on my .me.uk domain name and moved the blog over to the .co.uk a move which I'm very pleased I made.

Some Interesting lessons in stuff from history

Well history of this site anyway!

• Use the best tool for the job, I remember the early days of using Drupal to run a blog never again, its Wordpress all the way.
• Don't be afraid to go where your passion is even at the expense of some audience, likewise blog because you have something to say not because you feel you have to
• Don't get down when people don't comment on your awesome piece
• Never use Sphinn or similar cess pits as the guage to public opinion (indeed add twitter to that)
• It's your blog do what you want with it!
• If you opening Pandoras box have tissues and thick skin
• Comment spammers really will never stop
• Making things bold and in red does not mean people will read your polite notices
• Do not call your readers guinea pigs out loud!

My Favourite Posts from Tim Nash SEO Blog?

In no particular order here are some of the ones I'm most fond of

• Am I an advanced SEO?
• Link Worth (I spoke on this at Think Visibility 2)
• Flash SEO revisited
• So Long and thanks for all the fish
• Profiling Landing page Users

So what's the future for Tim Nash the Stuff Specialist blog?

It's evolving I keep meaning to add a mailing list and in doing so demonstrate a rather nice double opt in method for emailing comments to people. Expect the posts to become more technical and less SEO based covering more practical behaviour modelling rather then simply theory. Oh and I might even get a video camera out and try some of this video melarky out.

Am I going to monetise the site?
Strangely a question I get asked alot, at the moment their is the occasional affiliate link and some bits like my recommended reads (which are genuine suggestions not just random books) but no "direct" advertising I do have space for potential 8 advertisers on the site I just never considered selling ad space on this site. Also up until recently I felt any advertising of an SEO product would be a conflict of interest things have changed slightly so it could be a case of wait and see. If you want to advertise then drop me a line and we can discuss it.

How about for Tim?

In will continue my quest to educate people in stuff and hopefully will be able to engage more with the community both online and off. I will also be off to a few more conferences though none of them SEO related I suspect the first of which will be Paypal Innovate in San Francisco in early November will you be there?

At the time I first double checked our own bots don't do anything quite so stupid before suggesting that I thought it unlikely but would happily test it. Dave suggested a wager oddly enough one I never took him up on and I'm glad I didn't!

How we crawl robots.txt file

The need for speed is paramount when crawling a site, a bot is taking up server resources and you want it to complete its required action in as short a point as possible. If your bot follows REP (what's REP See Post notes for details )it's first action should be to download the available robots.txt file, on average these files are 2-4kb in size very small and take no time to download, however a file sent with a 404 is closer to 22-40kb assuming it also sends the associated html. a much larger size given the majority of sites do not have a robots.txt file this means if you are not careful your robot will spend more time downloading a useless file then anything else. The method we use is to simply ask initially for packets if the return is a status 200 we proceed to download the file, anything else and the status is stored and is ignored.

Is the way you do your crawler the correct way Tim?

There is no "official" recommendation within the RFC governing REP that covers how you should treat status codes and which you should follow to only follow Status 200 is by far the most efficient method but it comes at a cost as you could be ignoring the file! It also doesn't totally protect against downloading 404 pages as some servers send out a status 200 not 404 when a page can not be found.
A draft proprosal did suggest that other status codes should be followed including 3xx related to moved documents temporary or permanent it did not explicitly mention dealing with cross domains.
I have started to make changes to our own bots See Post notes for details

How does Google deal with cross domain 301 of a robots.txt file?

It reads the file at least according to webmaster tools, in Bronco follow up post they show Google Webmaster tools accepting bit.ly/robots.txt file nice should we be alarmed potentially though only if your allowing custom urls to your user at a root level on your domain with dots in them so if your running a URL shortener then yes perhaps something to check.

Did you do your own tests?

yes I had already done some tests last night which backed up what they did here is how I tested.

Experiment 1 Cross 301 oh please say this doesn't work!
We created two domains domain A and domain B with a robots.txt on Domain A and 301 to a file on Domain B the robots.txt dissallowed access to /test/ folder, a test folder was put on both domains and index file was put in both, each domain was given a root index cross referencing each and each of the test files.

If Google crawled the robots.txt then Domain A should have 1 indexed page, Domain B 2 when finished.
with a monitor attached to the logs doing reverse DNS looking for a Google IP so we could watch the interaction some links were thrown at Domain A.

Result: Domain A - 1 page indexed, Domain B - 2 page Indexed

In Webmaster Tools a status 200

Experiment 2 - Let's give google the benefit of the doubt
Ok so maybe they have indeed adopted the 1997 draft and are therefore obeying redirects it will ignore a Status 666 right?
Fresh domain this time our robots.txt file will be in the correct location but will send a http status of 666

Result: Domain A - indexed 1 page

In Webmaster Tools - status 200

Experiment 3 - given you a robots.txt file regardless
Ok so what if we tell you our server is broken i.e 500 but we give you a correct robots.txt file?
Fresh domain, correct location but headers sent are http 503 - Service Unavailable we are telling it we are not available the server is buggered in effect.

Result: Domain A - indexed 1 page

In Webmaster Tools - status 200

Tim - If you think about this it actually supports the belief google have actually programmed in the ability to follow 3xx as otherwise it would have for the 3xx returned a 404 or a 200 and blank file

Experiment 4 - I'm not here even though I'm here
Final test send http status 404 but also a valid robots.txt file what you going to do Google!

Result: Domain A - indexed 2 page

In Webmaster Tools - status 404

Only in the final test did Google behave as if it was paying the blindest notice to http status codes, can we assume 404 is hard coded and it will accept anything else?

Why should you care?

While the potential for abuse is small unless you run something akin to a URL shortner what happens when your site is producing an intermittent 500 error. From playing with status codes it would seem Google if shown a invalid or unreachable robots.txt will continue to use the old file could this be a potential for abuse what about a sneaky redirect only google on a 301 from your robots.txt by a very mischievous hacker. food for thought, and I'm glad I didn't take that bet of with DaveN.

Whatever the reason only two weeks before the event is not enough time to find a suitable replacement so we have had to cancel the event. Myself and Dom are still commited to HackNorth and plan on finding a venue to run it in the new year.

We want to apologise to people who have tickets and who have paid trainfares we are as you can imagine utterly gutted!

The website will be updated shortly with

Dear All,

Earlier today we were told by the venue owners AQL that the venue for HackNorth would not be ready in time to host the event.
This close to the event starting we are utterly devastated as we do not believe we will be able to find a venue that can hold 200+ with WiFi in time and so have had to cancel the event.

At this point we want to thank the amazing sponsors, Yahoo Developer Network, Sun StartUps, Paypal, Haribo, Yorkshire Forward, 4IP, BBC Backstage and eBuyer who have been amazing
We plan to source a new venue and look to plan HackNorth 2 for early next year so don't despair HackNorth will happen just with a little delay. For people who have gotten tickets fear not they are safe and should you wish to come next year will be valid. For those who have booked trains etc we are so sorry that we had to cancel so late in the day but it really is beyond our control.

While we would be happy to run such an event in an unfinished space we did need it to have basic requirements such as sanitation that could stand up to 200 people use over 24 hours.

If you have any questions please do not hesitate to ask

Cheers
Tim and Dom

Morte info on Doms post HackNorth Cancelled

Think Visibility

Think Visibility is a conference unlike any other its informal style but high calibre speaker list plus low price makes it a real winner doubled with it being in Leeds and not London its no wonder its so popular. This time around I had a slightly larger role over 2 days rather then just the day itself and while I was feeling under the weather it was unrelated to the conference.

I really wish I could have seen all the talks, and hopefully will when the DVD comes out (barring a couple) but the two that I did get to see bits from which really stood out were Fiona from Simple Usability eye tracking talk and Karyn from Tinderbox on Business bloging talk.

Behaviour Modelling Seminar

For Think Visibility attendees on the Friday before, you may well have seen me pushing this on the blog and twitter but the seminar went really well. It was very much an experiment to see if the format and content would work and the feedback was positive. Hopefully the attendees learnt a lot and indeed so did I.

The "Secret Panel"

Image Credit: sk8geek

Originally envisioned as QI meets the house party the lunch time panel was scaled down due to my other commitments and the previous days seminar. Instead I moderated a panel consisting of ChrisG, Al Carlton, Kieron "share my playlist" Donoghue and Dom Hodgson. The original brief was to make it fun but interesting and I think we did so with a range of audience questions.

Link Worth

My main talk of the day was on the value of links, not in terms of link building but actual metric values. Which covered some basic ideas for valuing individual links on various factors including some future where search engines could be working in particular through page segementation. While a few complaints were made that I declared relevancy to be irrelevant (apparently it was sensationalist and contreversial me I thought it was a rather obvious given) the talk did seem to go down well with people asking questions after.

Sadly due to feeling unwell I wasn't really able to mingle much and went and hid in my room for most of Drink Visibility which from the tales sounds awesome. Overall the event was a success I think with people seeming to be enjoying and learning at the same time and hopefully will see everyone at think Visibility 3 in March 2010!

Charity Hack 09

So this weekend I went along to CharityHack a Hack Day sponsored by Paypal in their headquaters down in London and it was great. Now to be fair I could be saying that because I'm writing this while wearing a black warm fleece provided by them but I think I would be saying it regardless!

Photo by cyberdees - I managed to lose my Netbook Charger but thankfully a replacement was found

Sadly the turn out developers wise was a little underwealming but in many ways that was better because teams inter mingled and the atmosphere was great. The guys from Paypal kept everyone fed and watered, the office has what appears to be a seemingly endless vending machine which never ran out of coke.

While the name gives it away CharityHack was all about coming up with ideas and applications for charities and given Just Giving, Mission Fish and Paypal all released new APIs most hacks submitted were donation based ideas using one of the apis. Our hack which won the overall "grand" prize (a trip to Paypal Innovate) was a First Person Shooter game where users donate to the charity of the person who "frags" them. Run out of money the system kicks you off the server. It was quite addictive so much so that at 4am when the development team went to bed our "beta testers" will still providing valuable data. You can see the complete winning list on Paypals Lets Talk site.
Photo by cyberdees
The team consisted of Tim, Dom, Kevin, Leeky and Tom

The weekend was probably one of the smoothest running hackdays ever and kudos to John for managing to unblock a gazzilion ports on the network even if he did have to wake rather important execs up to do it. Likewise Alison spent the day running around and generally keeping us all fed. I'm looking forward to next years!