I’ve uploaded my new book to the interwebs so you can now download and read through it, if the fancy takes you, completely free of charge (SEO Truth – A Bible For The Next Generation Of Search Engine Optimisation).

There may be a few blank pages because it’s been laid out for print; you can get yourself one of these hard copies from this website here.

Let me know what you think and submit some feedback over on lulu by all means! Cheers.

Edit: The download link works correctly now, oops.

Hello!

As usual I’ve been spending a horrendously long time without writing anything on my blog – and for that I apologise. However, I have spent some of my time writing an SEO (Search Engine Optimisation) handbook, covering the importance of next generation techniques and practises.

I’m sure there are those of you who are all too familiar with the increasingly backwards approaches used by a few ’special’ SEO agents and individuals out there and perhaps for you this will merely reinforce what you already knew to be true. For those of you who don’t know what I’m talking about -then please read the book and have a good laugh at yourself for being such a silly.

You can order print copies of the book – just not yet… more details on that coming soonly!  I’ll be publishing online chapter by chapter (honestly I have finished writing it, but as an SEO, if I didn’t serialise it then it would look bad).

Enjoy the read and let me know what you think, if the first edition is terrible and you order, of course it’s going to be valuable in 200 years!

Preface

First off, I’d like to introduce myself. I’m a Search Engineer, a developer and programmer. I’ve worked with clients throughout the advertising industry at many different companies. My specialty is developing software that works with the search engines of companies like Google, Yahoo and MSN and attempts to influence the rankings of my client’s websites, as well as report on those ranking changes. I’ve never been to a lecture on computer science, read a book on development methodology and yet I’m in demand. My skills lie in understanding the technology of a search engine and how to capitalise on their ranking algorithms, web crawlers and content filters and it’s the ideas I generate in this area which have kept me in gainful employment.

SEO (Search Engine Optimisation) used to be a fairly simple task where you’d make sure every page on your client’s site had Meta tags, descriptions and content unique to that page. You might then try to analyse the keyword density of your key terms to keep them somewhere between 4 and 7 percent. More often than not most SEO companies wouldn’t even attempt that.

What most SEO companies would never tell you, and this is the industry’s most well kept secret, is that they’re intrinsically lazy. If you had a good client, with good content and a product of interest then their SERs (Search Engine Rankings) would climb entirely naturally to the top spots, you’d have nothing to do but sit back and reap the benefits of your lack of work.

This is of course a sad state of affairs which no real SEO company would allow and part of this book will help you to spot the difference between a professional outfit and rank amateurs and define the widening gap between the two camps.

As the title suggests I’m writing about the next generation of SEO. It’s becoming more difficult to increase the rankings of a particular website and it will only get more difficult to manipulate a website’s ranking without any understanding of how new search engine technology works. Lucky for you, my field is semantics (how to correlate the relationship between one word and another essentially) and you’re in for a whole chapter in manipulating a semantic index similar to those increasingly used by the major search engine players.

Chapter 1 – The Past

In order to proceed correctly in the future, the most important lesson is for us to understand what happened historically. There’s no shortage of information on the internet and amongst SEOs and webmasters about how Google’s original PageRank system worked. This is in large part thanks to a paper written by Google’s founders, Larry Page and Sergey Brin, whilst they were still studying for their PhDs at Stanford University. Not long after that they received their first investment from a company called Sun Microsystems which enabled them to build upon the hardware they had in their university dorm room and create the international phenomenon we know today.

PageRank was essentially a very simple system. It counted each link from one site to another as a vote for the destination site. By voting for another site the original gave away some of its own PageRank. The idea came from Salton’s Vector Space Model, which is a mathematical principal known to most Computer Science graduates today. This simple method of calculating which websites had the most votes, and therefore deserved higher rankings, is key to all search engine algorithms as it’s extremely fast to calculate. The most important factor in any search engine is its speed in returning and ranking results, especially when you’re dealing with an index of billions of pages.

The Anatomy of a Search Engine, based on the work of Larry Page and Sergey Brin whilst at Stanford.

If you understand that all calculations undertaken by a search engine must be as fast as possible, it allows you to draw logical conclusions:

·       Thinking about a page as a machine would (which struggles to actually understand rather than just read), rather than as a human, is key to analysing your websites content for SEO value.

·       Is every single underlined heading, keyword color, font size, image location, keyword relationship and page title length analysed when a page is crawled? It’s highly doubtful that anything too in depth is going to be indexed, when the crawler has another hundred thousand pages to visit and rank as quickly as possible, use some common sense here. Of course as processor speeds and bandwidth increase more in depth analysis will become possible in a shorter space of time.

·       The search engine needs to maximise two things: the speed of its calculations and its measure of quality relevancy. Occasionally one is going to suffer at the importance of the other, if you were going to choose between indexing a page poorly – or not at all – which would you do?

SEOs in the past were able to capitalise on this speed issue by choosing to concentrate on areas of a page such as the Meta tags, description and page title. The content itself gradually became more important as time went on but still was subject to the speed of indexing. SEOs quickly realised that keyword density (how many times a keyword appears on a page out of the total number of words) was a very quick way to determine some kind of relevancy, and that the search engines were using it too.

Once the search engines got wise they implemented filters that stopped SEOs from flooding a page with keywords. Arguments in the SEO community followed over exactly what was the ideal keyword density for a term, and this usually settled somewhere between 4 and 7 percent.

Of course the PageRank model meant that agencies were keen to build as many links to their client websites as possible. To make matters worse however they were after links that already had high PageRank values to gain the maximum ranking as quickly as possible and this sprang up a cottage industry of people generating high PageRank links, purely to sell on. Google of course were unhappy about this and their anti-spam team began its work. Blacklisting of websites which ‘farmed links’ was becoming fairly common and this moved on to other aspects of ‘black hat’ SEO behavior – where an unfair advantage was being made by some nefarious companies and individuals.

Most SEO agencies at this stage relied heavily on staff who’d be subjected to some extremely tedious and repetitive labour. Going through page after page of a website and adjusting the number of keywords on a page, slightly changing each page title and Meta tag was a boring job and not well paid.

Directors and CEOs didn’t have a whole stack of problems though, if they kept building up link relationships with ranking websites and making sure their Meta tags were in place, their job was done. Often enough they’d have clients who already had an interesting product which did most of the work itself, spreading links around the internet as people registered their interests.

This natural traffic increase was what Google was looking for as they wanted sites which progressed on their own merits rather than trying to beat the system.

I’ve said many times before that so-called SEOs out there need to stop believing every word that spills from Google’s overactive pen. Google is a business just like any other and they feed information that’s deliberately misleading to stop people from gaining an unfair advantage with their search rankings.

It now appears that, in fact, Google assigns an estimated worth to each ranking on their pages – visible to members of their AdWords sales team they use the information from your PPC campaigns and analytics package in order to figure out whether you’re worth it.

I’m sure many of you will draw your own conclusions from this and in time we may see a Google press release, from that department which again knows as much about how their technology actually works as most of the SEOs do. Take it with a pinch of salt is my advice and invest the time to understand how a search engine really works.

This story was broken on the french blog Zorgloob, much credit to them for a brilliant find.

One of my favorite blogs, that I read just about every day is readwriteweb, a sterling tech, web 2.0 and search blog. Not so long ago their AltSearchEngines regular article was turned into a fully fledged blog in its own right headed by Charles Knight who knows about the existence of more search engines than probably anybody else on the net.

I checked it out this morning and spotted an interesting article:

Today we launch Part I of our 3 Part Series

Part I: What is a Search Engine? by Nitin Karandikar (Mon)

Oh glominy! I thought, glibbily. This is right up my street so I settled in for a powerful, thought provoking read.

Alas, the writer was a complete nitwit and I felt compelled to post this raging comment:

You’re completely wrong, I don’t know why on earth you’d try to reclassify what a search engine is when we’ve known what search engines are for a long time.

A search engine is simply “an information retrieval system designed to help find information stored on a computer system” (Wikipedia).

1. It enhances findability of relevant web content for the user

It doesn’t need to have anything to do with the web. Findability is not a word, even in italics.

2. It searches the entire web or a large subset thereof
(this excludes publisher search engines that search only a single site or group of sites)

No search engine searches the entire web. Don’t listen to the Google PR machine so much, and again, it doesn’t need to touch the web to be a search engine. Plus you’re on AltSearchEngines here… how many verticals do you guys cover?

3. Searches are specified using a keyword, phrase or question, or using input parameters, without the need for undue navigation
(I don’t consider pure directories like dmoz to be Search Engines)

So you’re saying you need an input to get an output? That’s genius.

4. It provides search results on demand, not periodically

I don’t even know what the hell you’re trying to say this for. It’s still wrong. Why does it have to do as a person asks it?

5. It provides some kind of unique or special processing of its own: either in the search algorithm, or in UI improvements, or both
(this excludes pure Rollyo or Google Coop-based search engine subsets)

This is far and away the worst thing you’ve written, you’re clearly grasping at straws. That is until you said:

The criteria described above will not remain static; as technology progresses, Search Engines will need to support increasing levels of functionality to be taken seriously.

No, i’m afraid a search engine, will always be a search engine. No matter how technology progresses it will still be a search engine.

The article you should have written is, “What search engines should have on my holidays”.

Yakov: A search engine doesn’t need to have its own index of the web or build it. A crawler of some description is responsible for building an index – that can take many forms and is often included in the search engine software itself. If you want examples of search engines without their own index, then take a look at the recent Digg API contest for some examples.

I’m hoping Charles gives you a massive kick up the backside and stops you writing what essentially is a load of bollocks.

Yes, it was a little scathing, but I get extremely irate when I see article written by someone who clearly is just trying to write for the sake of saying something. Especially on a source I have a lot of respect for because I don’t want to see them letting it through to the front page, that’s their role as editors – to weed out the rubbish and go with the quality content right?

Is the name of my new blog on the slightly odd spam that Akismet catches for me.

We need to relate to the spammers in order to understand their needs, and you can do so right here.

In previous versions (for those of you lucky enough to see the Alpha of the world’s first search engine to run directly from the user’s own desktop) Sunbeam would ask you to input your favorite websites as a starting point for its indexing routines. This was a problem for two reasons:

1. Nobody ever wants to enter anything they don’t have to, especially when that information exists somewhere on their machine.
2. It limited the ‘profile’ of the user initially available to Sunbeam and how quickly they’d be able to retrieve information actually relevant to them.

It also meant that the semantic engine that appeared in the earliest release was not capable of returning accurate matches for a period whilst the engine cranked up and had indexed at least a few hundred pages.

I’d been musing over these problems for a while, I wanted an experience where the user would be able to just install the program, let it do its work without going through any configuration screens, which they may not understand or that might put them off the install completely.

The solution as it turned out, was fairly simple. Using the browsing history of the user we can track down the urls that are visited most frequently and most recently without damaging privacy. After all these are just starting points to build a profile of interests. Data like this is a goldmine for Sunbeams advanced statistical algorithms and will enable it to deliver the results that mimic the language used in the websites in your browsing history.

It doesn’t stop there though, also added are routines that scan your outlook sent messages, tracking the semantics of your own typed words. These again, are not stored as complete messages anywhere in the system, are not tied to email addresses or even subject lines and privacy here is key. What is most important here is that you as a user will never have to go through a slew of irritating questions when you install Sunbeam, that inadequately attempt to locate and disect your interests.

Seeing as I expect privacy to be such an issue here, let’s turn to another reason to use Sunbeam over Google or Yahoo:

• Your searches are your own.
• Your data will never be sent anywhere else (there isn’t the server space for it!).
• If you choose to share your search database with anyone else (as easy as emailing the one file), then that’s completely up to you and not something you have to ‘opt-in’ to.

This software is entirely your own to play with, these are the things I’m really loving about it:

• You can play with the open source search algorithm.
• You can swap, share and amalgamate databases with friends or download one from the web.
• There are no adverts, no pop ups and no interruptions.
• If you don’t remember the exact word you’re looking for, just put in a similar one, or a descriptive phrase.
• If you want to use the same database when you get home, just mail it to yourself.
• If you don’t like the results you’re getting, run a seperate database for work and for home to match your corporate and downtime moods.
• If you have to do market research on teenagers, just use the database your nephew compiled.

I have just watched a video of the most exciting user interface ever seen. It’s not of the forthcoming iPhone nor is it any kind of Apple product. This is Microsoft Surface and it promises a revolution in how we interact with our computers and mobile devices, I’m completely blown away by not the technology behind the system, but how well it’s used to produce a product that will potentially devestate Apple’s market share.

If you wondered why Bill Gates was suddenly agreeing to do an interview with Steve Jobs, then I’m pretty sure this is the reason. It doesn’t matter if he does badly in that discussion because as soon as Surface was on show then Steve Jobs had lost out anyway. Will Jobs have a rebuttal product that we haven’t heard about? I doubt it.

Pricing And Availability

You’ll be able to get Surface from winter 2007 for between $5000 and $10000. I know that’s a lot of money right now but they aim to bring the price down to a consumer level quickly and this is the first device I’ve seen that really will fit right in your living room, instead of just attempting to hide in a corner. Designer coffee tables go for far more and I know which I’d rather have.

The New Standard In Interaction

For me, as a search and user interface developer, this fits in extremely nicely with my view of tiling search results as images. An application using Windows Live Search in this way for not just searching but RSS feeds and bookmarks would be highly intuitive and allow the user to see what they want straight off the mark.

One of the most ingenious features they’ve integrated right off the mark is the ability to interact with your mobile devices. We all have phones now; they started with IR then Bluetooth, now some feature WiFi. How many of you actually use these connection abilities reguarly though? I’d guess it’s a low percentage because the hardware and software we have to connect with doesn’t make it simple and easy enough to use frequently in most cases.

What Surface lets you do is put your mobile phone, PDA or digital camera directly on the table top and a ring will appear around it to signify the connection. You can then drag media to and from the device with your finger and a bit of wrist movement, it’s so simple it makes me want to cry. I spend a lot of time shouting about the need for simple and intuitive user interfaces and this is the model we should all start building from.

This is the new standard in user interfaces, keep up.

Hi Readers!

The internet is an odd place, as I look at wordpress.com right now I see the top few blogs are I CAN HAS CHEEZBURGER?, passive-aggressive notes from roommates, neighbors, coworkers and strangers and of course Scobleizer.

F or those of you yet to witness the phenomenen of icanhascheezburger then let me summarise for you by saying it’s a blog filled with cute/demonic pictures of animals, mostly feline in nature with captions underneath. The passive-aggressive notes blog is exactly as it says in the title; pictures of amusing passive-aggressive notes.

As a further exercise in demonstrating to you the power of this medium let me give you an example of an icanhascheezburger image (taken of my girlfriend’s cat, yesterday):

If you haven’t been to the site, you won’t understand most likely. The ‘bucket’ is an in joke as these websites often produce. Why exactly though is it so popular over the thousands of blogs that produce well written, quality content?

It’s fast

There are many facets to the speed here, firstly it’s very quick for the authors to add a new post. All they need to do is get an image, put it in the wordpress editor, add a couple of lines about the submitter and possibly the humourous content if they can be bothered and they’re done. This means they can generate hundreds of posts in the time it takes the rest of us to put out one or two (sorry wasn’t talking about you Scoble, or you Winer). The other quick thing they can do when they add a wordpress post is to select categories, this is a very fast way of tagging essentially and means as well as quickly refreshed content they also have targeted keywords. Hello good SEO.

It’s also fast for users; if you don’t get the joke in the first pic you see, it’s a 1 click scroll to the next one. You laugh, it’s funny, you whack the link on an email and send it round the office. They even have a lolcats generator that lets you put a caption on your picture of a cat in about 20 seconds AND automatically submit it to the site. Auto generated content essentially, which is just gold.

If the site updates less often the search engines aren’t the only things that return less frequently. The same applies to all your human users as well. They’re far more likely to refresh if they think the content updates often, and even more if they think their cat might appear on the next post.

What next?

I think very soon, you’ll see an abundance of these kinds of websites arriving if people are smart (often they’re not).

All kinds of non text media will benefit from this treatment and a social voting style system for it will allow a much faster turnaround on content. You’ve seen it with Digg and this is one of the reasons they really should add an images section they’re losing out hugely there.

Other websites have also shown the advantage of fast content generation from any source. Twitter allowing updates by mobile phone for example. I can upload pictures to blogspot from my k800i directly, it’s a shame I don’t like the blogging software.

Urrr.

I completely lost my train of thought I went and read some c# documentation and then all my post ideas ran away. I may finish this later when I regain my mind.

UPDATE:

It would appear Valleywag’s Nick Denton is lacking a sense of irony and unfortunately I seem to have my commenting privileges revoked there now. Shame. He’s thoughtfully left this little nugget seemingly ending the argument with a resounding slap to my pride:

“Hey, Phil, I don’t mind being slagged off. Comes with the job. But you didn’t do it very effectively. One could make the point that mentions of Google itself have become more frequent. But sensationalism? I don’t think you proved your point”

What’s that Nick you can’t hear my answer from all the way over there because you blocked my account? Never mind. Sensationalist articles Nick, seeing as you are unaware, are those that are published without any proof behind them. So I put together my own sensationalist article on your sensationalist article and it appears you lack a sense of humour. Fortunately you’re unable to prove to me you have one because that’d mean you wrote something of substance. Unlike you Nick I won’t delete or remove negative comments even though I rate my blog above a tabloid so feel free to hurl insults from below if you wish.

THE ORIGINAL ARTICLE:

I saw over on Valleywag they’ve written yet another hack piece on the so-called Fear Of Google with the standard sensationalism and lack of humour. They’ve even drawn a pretty graph they collated data on from the Nexis newspaper database showing their spectacular lack of knowledge on current Google events.

Being a bit of a dry and sarcastic git I present to you Fear Of Google: As Seen On Google Timeline! which is a representation of how Google itself sees the phenomenon.

Don’t bother reading Valleywag’s article, go and read what Scoble says instead if I was you.

Personally I have no fear of Google (though I am typing this in the stationary cupboard but that’s because of my love of pens) and instead feel an increasing need to criticise them rather than run in fear. Then again, people react in the same way with governments and it’s surprising that a company can approach that level.

Ignore the numbers in the title for the moment if you will and focus on these keywords: social, networking, search, community.

Web 2.0, by many definitions is all about allowing users to network, interact and the read/write web. Search 2.0 in that context does not yet exist. There are in some instances communities that happen to be built around a search engine such as Yahoo and there are new semantic search engines that let the users tag pages and documents to be found (something I’ve talked about before and pointed out as next to useless). None of these let the users actually interact with which results are returned. There is no networking or interaction that takes place with the search engine itself and this is just plain wrong.

Do you know how many people are on the internet at any one time? I sure as hell don’t but it’s a big number

Who creates all the content that ends up on the internet anyway? It’s not machines, it’s people, human beings are ultimately responsible for all the content on the internet and that’s never going to change. So why are we asking machines about content created by fellow humans when most of the humans are online anyway and know far more about their subject, and where it’s covered on the internet than any machine is ever likely to?

Enough rhetorical questions, I’m going to tell you what the real Search 2.0 is and you’re going to shout at me and tell me I should have patented and I’m a fool. However, if you don’t hire me to build it for you then you’re a fool because I get these ideas on a daily basis and I will crush you at some point in my life. Just kidding, I’m a fan of open ideas as well as open source especially when they’re for the benefit of us all.

Search 2.0

• An instant messenger application or website with a live AJAX interface forms the centerpeice of the front end.
• Users create accounts and select their areas of interest by entering specific key phrases for those topics they feel most knowledgeable about.
• Users can then also select web pages that match those highly specific key phrases if they choose to.
• The search box appears as normal, you enter your query and the fun part of search 2.0 begins.
• Your query is analysed against users on the system, what occurs at this stage is actually a search for users with the best matching key areas against your query.
• If these users are online they can respond directly to your query, either suggesting a web link or entering a chat with you.
• If no users are matched online, then the suggested web pages are searched for the best matching content.

That’s the basis of it, but let’s have a look at the immense social power here.

Firstly, you get to rate the responses you receive, meaning that people can gain a reputation score for specific subjects and topics giving them an online credibility for that topic.

Sorry we just had an office fly-hunting session. Don’t ask.

Right, where was I? This system is by its nature, very low spam, it can’t be manipulated to provide results that are less useful because if you try and peddle a corporate product that’s crap, your reputation will drop very quickly and you’ll be banned. If the product is good on the other hand then who’s going to mind being directed to it if it answers their specific need and that’s better advertising than any money’s going to get you.

This concept is all about the users, no massively complicated algorithms need writing here it’s just using the very advanced and articulate knowledge of the very people who create the content you’re looking for, and to get the best answer you’ll ever get from a search engine is it not worth answering a couple of questions every now and again about the subjects you enjoy?

You can also bookmark people just like in any other IM and make friends with people holding the same interests, who you’d never meet on any other social network, and certainly would never think to find from a search engine.

I’ve previously posted a video of the new search interface I’ve been experimenting with. In order to get some feedback on how it works I’ve put together a small application that uses the Digg API to display the popular news stories on your desktop background.

The application will update for the latest stories every two minutes or so and refresh the tiles accordingly. If you mouseover a tile the window will ‘fisheye’ slightly, similar to the OSX dock. Click on the tile to expand it to a readable size and then if you decide you want to go to the story then double click the expanded image to open it in your web browser.

You can close the application by right clicking the little rss icon in your system tray and hitting Exit, or if you encounter problems then close the process newsview.exe in your task manager.

You will need the .NET2 framework to run this and the installer should point you in the right direction if you don’t have it. Failing that then go here to download it manually.

I stress this is a little alpha level experiment, but if you do encounter problems then I’m only too happy to help, just leave me a comment here and I’ll try and fix any bugs you may find. What I’m really looking for though is some feedback on the interface: Is it simple enough? Can you see the result clearly? Would you use a search engine that delivered results in this manner?

To download ‘DiggTop’ click here and enjoy.

Ever since I was five years old and we had our first home computer – a BBC – I’ve been programming.

No one ever taught me how, in those days I would spend hours trying to learn how the processes worked, happy to sit and debug until I understood.

At school I was no great mathematician I admit but I still spent all my time outside of school programming as much as I could, being fascinated with computer games and the internet in general. I learned that the math happened in my head, and my ability to learn new languages and read patterns was very useful. My first websites in html, and then PHP were shambolic but I persevered despite going to many schools, my parents were in the RAF so travel was a major part of my life.

At university I was unimpressed with my course – with exams on how to use Microsoft Word, I realized my potential wasn’t being reached and I left with the offer of a position abroad in Cyprus on a long term contract.

When I returned I was surprised to find I was in demand and felt in the right place for the first time as a professional when I joined an advertising agency’s new ‘E-Communications’ department as developer ahead of hundreds of other applicants with degrees and more experience in industry. Since then I’ve worked on websites and advertising for companies from Walmart to Sony and across many different market sectors.

In the UK, I can now go for any position I want and feel confident I’ll get it, I turned down a position at a major search engine in order to take my current job. That’s because they can push me and use my to my potential. In the UK there are no jobs in search related fields that aren’t SEO and I don’t want to sell my work to a company who won’t use it effectively.

I do enjoy it, being able to work with designers and with an energetic team enables me to work as hard there as I have everywhere else. I strive for perfection in my work and enjoy the social life – I’m not a typical nerd as defined by stereotype. I’m keenly competitive as a sportsman.

I have ideas I can’t implement, I write down implementations that will never be used and concepts I will never have a response on.

When people ask me; ‘isn’t programming boring, how do you cope’, I tell them that programming is an art to me, it’s not just science, with an end result that can be beautiful and stunning.

This is what I look for in my work, this is what motivates me to write, I need an output for all the ideas that may never happen. People accuse me of not knowing about search, or being too critical of search companies but understand; this is my passion.

I welcome criticism myself, it lets me grow and adjust my ideas and without it what compass would I have. Look around this blog you’ll find comments calling me a f**king moron. Why would I delete a passionate response to an article that I wrote from the heart?

In the coming months, my first commerical search engine, running entirely on your own PC desktop will be released. It has features you’ll have never seen before I promise you that, it may well revolutionise the way you search for documents, web pages and folders. Rather than telling me it’s crap and I’m a f**king moron (if that’s how you feel), it’d be great if you’d tell me why you think that – and how you’d change it. Then I promise to answer you in kind.

I have a lot of respect for the readers who don’t like what they see, because I feel the same way when I write about a lack of innovation in search. If you don’t criticise what you love, it’ll stay the same whilst you as a human being never will. You’re changed by the criticism, praise and ideas of those around you.

Please, keep trolling.

I get a lot of comments, trackbacks and email lambasting me for daring to suggest that Google is anything but the best and most ingenious search engine ever made. Someone even said :

‘I use Google because it’s the best, end of’

Wow, that’s telling me a few things. Firstly you think Google’s the best. Secondly you think that’s the end of the discussion. Thirdly you’re an idiot.

I can say these things because I’m quite sure he doesn’t read this blog because he’s not open to new ideas. That was the fourth thing.

People seem to think that because Google is the market leader, that they’ve revolutionised search that it isn’t right to criticise them. You’re wrong, they should be criticised now more than ever because they are the most visible search engine and should be setting an example in terms of innovation, human rights and design.

What is occurring with how people relate to Google is a fanboy mentality (or fangirl) of a similar vein to the much despised Apple fanboy (at least on this blog). People praise anything with the Google brand on it, regardless of its quality and originality, and that to me is a sad state of affairs because it means Google have increasingly less of a metric to measure whether they actually are producing a worthy product.

I know that in the UK, investors will still not touch search engine startups with a sh*tty stick. This is because of Google, they(the investors) feel nothing will ever surpass it. This means there are practically no search engine startups in the UK and it seems only recently that they are even coming to the fore again in the States. I can’t tell you how much this irritates me, I’ve spent years working around search because I love its fundemental simplicity as a human need, that thirst for knowledge and learning, and it’s only since I spent 2 years of my own time programming a search engine that I was ever able to convince a company in the UK to take on a property that would overlap that of the mighty Google.

So when you read my post in future don’t think that I hate Google or have some spite against them. It’s the opposite, I love Google and want them to innovate as much as possible because otherwise they’re dead in the water. Ultimately though, I love search and whoever delivers me with the most innovation and sheer simple brilliance is going to be my choice.

Capitalism comes after natural selection.

Google I’m sorry, it’s not the way it used to be between us anymore. It’s not that you haven’t cared for me, given me gifts (the digital photo frame broke by the way) and let me watch porn.

But, I expected too much of our relationship I think – which only started as an accident anyway if you care to remember (that’s another thing you never seem to forget any of my personal details, it’s creepy). Don’t get me wrong, we had some great times but you’re just not as exciting as you were.

You used to be spontaneous, suprising and passionate! Now I’m lucky if you do anything out of the ordinary.

The thing is, I’ve started seeing other search engines occasionally. Not all the time, no there’s no one serious… yet. They do things for me you never did though and aren’t afraid of making me a little frightened. You should have seen the pictures Snap showed me, the way Hakia listened to me, and well… to be honest I did have some group fun with Clusty.

I take full responsibility for this, I obviously expected too much and that wasn’t fair on you – just know that you’ll always have a special place in my head.

With sympathy,

AOL.

PS. And Google, please do write me back, I’m open to your views here and I know how personable you are. Heck, if you’re passing by in your awesome ride, just leave me a comment even. (Diggnation t-shirt for the best response!).

I have been writing search articles since November 2006 and as a search engineer it helps me to construct my thoughts and focus on my work. I was never really under the impression that anybody in industry paid a huge amount of attention to my thinking though – maybe they still don’t – but I appear to have predicted many of the features of Google’s new experimental search in quite some elaborate detail.

See results on a timeline or map. With the timeline and map views, Google’s technology extracts key dates and locations from select search results so you can view the information in a different dimension.

Timeline and map views work best for searches related to people, companies, events and places.

You can find the opinion at other levels as well though, and this is where the power comes in in terms of really targeting what the user is looking for quickly and efficiently. All the following mean that this is the first true example of social search:

• Find the opinion over a range of dates, good for current events, modern history, changes in trends.

The text in black is from Google’s experimental ‘Timeline and map views’ search, the text in blue is from this article published February 19th, 2007. I was talking about a semantic implementation used to search through results by a timeframe instead of a straight relevance search. You can even narrow down date ranges by clicking on the timeline to select an era.

Spooky eh? I don’t think anyone else predicted that.

I hear you though, that’s just one example and I got lucky, so lets see… what else did I talk about in the same article?

Narrow Search

• When you begin a search, you enter just one or two keywords in the topic you’re interested in.
• Related keywords appear, which you can then select from to target your search and remove any doubts about dual meanings of a word for example.

The image above is from Google’s brand spanking new ‘Left-hand search navigation‘. Look at the white box (I’ve faded the rest out for clarity) and there you have it, exactly the feature I was talking about in all its living glory.

So what about the other features of Google’s new experimental search? Right hand search navigation… oh hang on that’s just the left hand navigation on the… urr… right side. Revolutionary. There’s also keyboard shortcuts! Oh hang on I just did a Ctrl-S.

In fact all the ‘new’ features of Google’s experimental search are ones I wrote about developing back in February and this I find slightly irritating for a few reasons:

1. It’s very unlikely that they saw my work and copied it. The main worry for me is that they still can’t come up with anything new even in what they call experimental search and that’s what bothers me about a company with so many PhDs. Why don’t they just give me a job? And yes, I am bitter
2. They are afraid to push the boundaries. For instance in my article I write about some ideas far more experimental than these, but they’ve not got anything exciting or new in there. I want to see the things that happen in the Googleplex, and then get binned for being too radical or wacky. I want to see binned.google.com never mind labs.google.com.

Again, Google copying me is hugely unlikely, this I’m sure is a coincidence. However, a coincidence that should never happen. I’m one person developing search technology with no PhD and heck I never even finished my degree I just cram programming languages into my brain and think of crazy ideas until they form into something I can use.

They have all the intelligence in one building, you have no idea what I’d give to use that kind of working environment and the batsh*t insane ideas we’d make a reality.

I’ve decided to put together a list of the material I read on the internet throughout the day whenever I get the time to do so, meaning that you my loyal readers have someone to sort the wheat from the chaff for you.

• DIY 3D Fabrication Machine Makes Sculptures out of Sugar

  • I like ingenuity and this is a fairly interesting read – really though you’ll just want to check out the pictures and marvel at the time that’s gone into this.

• Kevin ‘Digg’ Rose goes for 3rd startup

  • This is a very brief bit of news, because there’s not much substance to it yet; basically Kevin Rose is rumoured to be starting a new company which will be building some kind of new instant messenger client. I’ll keep paying attention to this for any new info.

• George Lucas Announces Two More Star Wars Movies

  • Arg, it’s apparently true the legendary (not necessarily a good thing anymore) god of Star Wars plans to make two further movies that won’t involve the Skywalker family. Read on for the full scoop.

• ROBOT LOVE: How Soldiers Are Loving Their “Bots”

  • I really enjoyed reading this story, it revolves arounds those using robots to detonate safely IEDs in Iraq but is extremely well written and well worth a read.

• THE LONG TAKE – The Greatest Long Tracking Shots in Cinema

  • The longest camera shots in film (and some of the best). My favorites being Hard Boiled and Tony Jaa’s spiral staircase action fest.

So that’s my first roundup, let me know if you enjoy the feature/find it useful – don’t bother if you don’t I like my pride intact.

Over at Valleywag they’ve ushered out a new speculative article on the next move of Jason Calcanis, the Silicon Valley entrepeneur who sold his blog network to AOL Time Warner for a multi million dollar sum:

“several people, in a position to know of his plans, say these schemes are at most hobbies, or pure disinformation; the next venture is a search engine.”

Valleywag, ever the innovators, pin this search engine venture down as a cross between Wikipedia and Google. If I recall, that would be Wikia search right?

Jason has been quick to himself respond in the comments:

“this is almost as good as the don imus stuff… i love you guys–you’ll print anything. ”

Certainly seems to have Valleywag down to me – so called insider tech sites these days seem to think they can invent whatever they like – I don’t know where the buck is going to stop but people are going to have to start properly researching their stories if they want to keep their readerbase, just like any other division of serious journalism. There are those of you I’m sure who’d argue that a blog is just like a tabloid newspaper and that those rules don’t apply.

Go and read a tabloid then.

Jason also briefly discusses why Wikipedia would benefit from carrying advertising – he’s not wrong – just a single well placed ad on each page would make all the difference to their amounting costs and make sure that its fantastic resource would be around for a long time to come.

Some of you seem to be under the misconception that you’re right. Of course you’re not.

I’m right, you’re all wrong and I’m going to whinge about it because this is web 2.0 and I can.

Any of you who’ve ever read and loved Joseph Heller’s Catch 22 will know why you use Google, praise Apple and deify Digg on an all too regular basis:

Because everyone thinks that way, and you’d be a damned fool to think any differently.

What made all of the above unique and intelligent and innovative though is exactly that; they thought differently from everybody else. Apple put design and usability above complexity and a mass of features, Google put simplicity and speed first, Digg put the user first.

These were concepts that were new to technology. These are concepts which are no longer new to technology and without dissent and encouragement of genuinely new thinking then we’re stuck with a Google which no longer returns the results you want, an iPod which will never move far from its original design and features and Digg news from a group of adolescent sensationalist amateurs.

We have a generation of immensely intelligent people now working at these top companies who are producing refinements and spin offs. Very rarely it seems do they ever produce anything new for fear of straying from their original product. The original product is only ever changed by tiny margins at a time for fear of losing their customer base, Google is the prime example of this and the one that most annoys me. They even have another search engine that’s detached from the main company entity, purely for researching supposed new ideas and concepts in search, which is SearchMash.

Have you ever seen SearchMash break? Have you noticed it produce anything other than yet another list of results in that oh so familiar text format or do something even approaching the edges of the box? No you haven’t and you never will because they do all of that in clinical test environments on a developer’s machine so we’ll never know if they’ve produced something groundbreaking, revolutionary or even just cool. The sad thing is, neither will they because the only people who see it are other brilliant PhDd individuals.

So please, when the mob is rioting on Digg, the fanboys are screaming giddily at another gig of memory on an iPod or googledesktop reports on yet another useless pissing widget for ‘personalisation’ then please lean back, take a deep breath and write the following in the comments

Then I’d be a damn fool to think any different.

This morning I came into work, I ran some SEO checks on a few sites and started up FireFox. As usual I browsed speculatively towards Digg for my hit of the overnight news that just wouldn’t be covered in the Metro newspaper I’d browsed through on the bus.

At first I couldn’t work out exactly what had happened. The following code was written everywhere :

09-f9-11-02-9d-74-e3-5b-d8-41-56-c5-63-56-88-c0

Through my sleepy haze I realised it was the code for unlocking HD-DVD protection that I’d seen a couple of times on stories the previous day. It transpires that Digg were actively deleting the stories featuring this seemingly unthreatening code in response to a cease and desist letter.

Jay Adelson (Digg’s CEO) wrote on his blog at 1pm May 1st:

“This has all come up in the past 24 hours, mostly connected to the HD-DVD hack that has been circulating online, having been posted to Digg as well as numerous other popular news and information websites. We’ve been notified by the owners of this intellectual property that they believe the posting of the encryption key infringes their intellectual property rights. In order to respect these rights and to comply with the law, we have removed postings of the key that have been brought to our attention.”

Normally this would have been the end of the matter, but this is Digg after all which, “is all about user powered content. Everything is submitted and voted on by the Digg community.”

Digg users went on to no less than a full out cyber riot.

The community flooded Digg with stories containing the code, making it virtually impossible for the moderating staff to keep up with deleting all the stories – or that’s how it would appear to the mass of the Digg userbase. As a search engineer though, I know how simple it would have been to remove any story containing the code, variations of the code, links to pages with the code on and so on. Very few would have been able to get through if any if Digg was really intent on making sure they wouldn’t have a legal battle to fight.

As I’ve stated, Digg is no standard news website though and offending the userbase would be a poor marketing choice because they are responsible for the revenue by clicking on and viewing the adverts. With Digg’s users so fiercely protective of this story it would seem like the only choice.

Just eight hours after Jay’s post, the founder of Digg and the main public figure for the company, Kevin Rose, posted this:

“Today was an insane day. And as the founder of Digg, I just wanted to post my thoughts…

In building and shaping the site I’ve always tried to stay as hands on as possible. We’ve always given site moderation (digging/burying) power to the community. Occasionally we step in to remove stories that violate our terms of use (eg. linking to pornography, illegal downloads, racial hate sites, etc.). So today was a difficult day for us. We had to decide whether to remove stories containing a single code based on a cease and desist declaration. We had to make a call, and in our desire to avoid a scenario where Digg would be interrupted or shut down, we decided to comply and remove the stories with the code.

But now, after seeing hundreds of stories and reading thousands of comments, you’ve made it clear. You’d rather see Digg go down fighting than bow down to a bigger company. We hear you, and effective immediately we won’t delete stories or comments containing the code and will deal with whatever the consequences might be.

If we lose, then what the hell, at least we died trying.

Digg on,

Kevin”

This post is on Digg’s blog, and is complete with a Digg submission linking to it by Kevin himself. Many have taken this as Kevin and Digg coming around to their readers point of view and allowing them with good grace to post the code as they see fit. I don’t agree, to me this post is a last ditch attempt to reason with the Digg users and say: you lose the whole site, or you get to post your dumb code. It’s also the quickest way out for Digg, by allowing the posting they no longer make it necessary and they can remove them in a day or two when everyone has clean forgotten.

As always in these situations I highly doubt that more than 5 or 10% of the Digg readership were actually involved in this – but if they shout and scream loud enough that’s all that’s needed for them to get what they want over the reasoned arguments of everybody else. Digg by allowing this has opened themselves up to a hundred other groups who will want their own way on the most popular social news site on the internet in the future.

If you’re a Digg user or have any thoughts on this new mob dynamic I’d like to hear from you in the comments, I’ll keep this story up to date as any more comes in.

update at 12:35 02 May 2007

I read this blog piece on Digg’s troubles, it’s an interesting bit of opinion from a female who are in small supply on the popular social news website and I’d have to agree with a large portion of it.

I realise I haven’t made any posts in a while – so I thought I’d attempt an all encompassing update on what I’ve been up to with my various projects and some technology related personal junkage (not a word).

Firstly, big news here, I got extremely bored of being completely unable to buy a Nintendo Wii from anywhere in the UK. I’d like to think that if Lik-Sang were still around I’d have bought one from there but due to Sony’s increasing bizarre death throes that won’t happen. I haven’t been playing with the PC much recently, I don’t have time and I don’t see the urge to spend money on upgrading a system to the current elite state needed for modern gaming whilst it’s in so much flux at the moment. My AMD 2600 system lasted ages, probably a year or two before it needed really upgrading, at the moment with DX10 and new chipset designs emerging constantly it’s just a monetary nightmare to keep up on.

Ah yes, the news, instead of the Wii I went out and bought an Xbox 360. The reasoning for doing so was pretty simple:

• It’s portable enough to shove in a backpack and take places.
• The controllers are wireless, and so is the networking.
• It looks awesome, it has Gears of War (which i’d heard was not half bad).
• It costs about £300 all told with all the gubbins I need to have a good time with a friend on it.
• It has a huge number of titles out already.
• In future, it will plug into my shiny new hd set (long time off).

Whilst I’m doing those, you may like to know why I didnt get a PS3 instead, because they’re also available right?

• I don’t have half a grand to shell out on a machine to play games on.
• Sony are seemingly content to ignore their market and subscribers.
• I don’t believe blu-ray will succeed, and even if I’m wrong, the last ‘in house’ formats they put out haven’t done so well… PSP discs, Minidisc, Memory Sticks, Betamax and I’m not going to take the risk on what will become a virtually useless machine if it fails (plus to Microsoft for having the sense for NOT embedding their HDDVD drive no matter how much people whine).
• I’ve heard rumours of key developers being unhappy with the development functionality, meaning less decent games for me.
• My PSP’s screen broke, whilst sat in a drawer, virtually a month after warranty expired, grrr.
• Sony didnt get an exclusive for GTA IV, and it seems to be their own fault.
• Sony seem content on infighting and letting idiot spokesmen run loose to damage the reputation of a very powerful bit of kit in the eyes of the well informed gamer. Bravo Penny-Arcade by the way.

In a word, my Xbox experience has been excellent. Live just works, Gears of War is fantastic online and the best experience I’ve had since Counter-Strike probably. I don’t care about Halo 3, UT 3 is going to be my first choice seeing what a brilliant job Epic did with the hardware for Gears of War. I still want a Wii, but it’s never going to give the sheer hardcore gaming experience that Microsoft have done so well with 360.

In other tech goodness. I have finally succumbed after years and years and years of repetitive, procrastinating, arduous and never bloody ending grinding from Apple Fanboys. No no, I didn’t get a macbook, that’s a step too far for now.

I bought an Ipod Nano, 2GB. I’m starting small. It’s been good to be honest, very practical, simple and does exactly what it’s meant to do without asking me annoying questions every 10 seconds. It’s also very small and errr…

I…

Well. Underwhelmed really. I think part of the problem with it doing everything neatly and correctly is that I don’t ever find reason to tinker with it and as such it seems like it really is only skin deep. With a PC, or non apple device, it’s always doing something wrong and you begin to understand how it works because you’re always fixing it. I miss that (though it is nice not to have to notice it all the time). Sniff.

PROJECTS! Allegro is no more, well it’s just not called Allegro anymore. Following trademark discussion with the lawyer under dwellers they’ve told us that we probably shouldn’t try and trademark that for fear of inciting the wrath of companies with far more under dwellers than our own. A new name has been conjured from the twisting networks of our over burdened brains but I’ll wait til we launch the product before we unleash it this time for fear of more soothsayings from the lawyer folk.

The project formerly known as Allegro should enter it’s final development phase at the beginning of next week all things going to plan. From there it’ll be released in first beta, for people to test and poke holes in so I can fix them.

Finally, over at www.home-james.co.uk we’re working on putting together our own SEO and PPC blog where you’ll be able to read up on the latest techniques, technology and ask questions about how best to improve your search engine rankings. That should be released at the same time as the project formerly known as Allegro so it’s an exciting Q2 for us.