The essence of SEO is an understanding of how search engines operate…

Fundamentally the search engine is trying to deliver the most relevant match to the end user’s search phrase. As I wrote recently, this led to the original SEO tactic of keyword packing both on the page itself and in meta tags. This use of meta tags was specifically designed to allow the web author to talk directly to the search engine to highlight the particular content focus of each page.

Sensible use of keywords on the page is still essential but the use of the keywords meta tag in particular has been totally discredited. Basically it was too easy and too tempting to try and game the system. To remove the machine-generated keyword-packed spam that was destroying the quality of their results, the search engines had to ignore or even penalize the use of the keywords meta tag. Nowadays all search engines (apart from Inktomi) concentrate solely on the content that the end user sees. This was a necessary step, but it removed what should have been a legitimate and useful way for author and search engine to work together to deliver the most relevant content.

Today though, keywords are back with a vengeance in the form of Web 2.0 tagging.

Use any content management system (cms) such as Wordpress, Joomla and Drupal and all content contributors will be encouraged to tag each item that they add with relevant keywords (the same is true with other Web 2.0 systems such as Flickr and Technorati). These tags are then used to produce a major layer of navigation for the site. They are also used to generate summary pages of related teasers and can be used for other purposes such as automatically generating your site’s URL path structure and breadcrumbs.

As I see it, there are two essential differences between these Web 2.0 tags compared to the old keyword meta tags. First, the tag is no longer hidden but there openly on the page. Crucially this means that it will definitely be added to the search engine index. It also means that it’s not so tempting or easy to game the system with irrelevant keywords so the tags have more credibility.

Secondly each tag appears as a link. Again this is likely to boost the importance/credibility of your keyword as anchor text rates more highly than vanilla text. More to the point, it’s important to realise that the web isn’t built on text but rather hypertext. Google was the first search engine to fully appreciate this and to intelligently index not just content but links so building up a contextual picture of your site as a whole. Web 2.0 tagging is perfect in this regard as it allows Google to understand the content focus of your site as well as your page.

Put these benefits together with those of keyword-based url and breadcrumb structure and well-connected keyword-based summary pages and I think that you largely have the answer to the common question: “how come this guy’s personal blog is outscoring my professional site?”. It also helps explain why adding a blog to a site is generally recognized as a sure-fire way to boost traffic though this should only be seen as a halfway house to a full-blown, Web 2.0, tag-based cms site.

Web 1.0 / SEO 1.0 used to be a secretive oppositional battle of wits between search engine and optimizer. Web 2.0 / SEO 2.0 sees a fundamental shift towards a more open and co-operative user-oriented model. Web 2.0 tagging is the perfect example of this new co-operation in action with author/optimizer and search engine working seamlessly together to deliver the most relevant content to the end user both on site and via search.

This change from SEO 1.0 to SEO 2.0 is perhaps most apparent when it comes to the use of meta tags…

In the distant past, the holy trinity of title, keywords and description meta tags were generally seen as the optimizer’s greatest weapons in the battle for high placement. The basic thinking was: Google only has the end user’s short search phrase to go on when trying to make a match so the best way to make sure that your page comes up on its SERPS (search engine results page) is by packing your meta tags with key search terms.

However look at what Google itself has to say about the meta tags in its own Search Engine Optimization Starter Guide and a very different picture emerges of what constitutes good SEO. To begin with, the keywords meta tag simply isn’t mentioned at all.

In fact this isn’t too much of a surprise. The problem with a list of keywords is that they are too easy to generate automatically and might not have anything to do with the page in question. Clearly Google wants to present the end user with truly relevant content so it’s the on-page use of keywords that really matters (how many there are, where and how they appear and so on), as well as their sensible use in the title tag.

However I have to say that I was surprised to see what Google has to say about the description tag and in particular to read the following on the Google Webmaster Central blog: “meta descriptions won’t affect your ranking within search results.”

So does this mean that the description tag is as irrelevant as the keywords tag?

Absolutely not. The description tag might not help ranking but, as the SEO Starter Guide puts it: “Description meta tags are important because Google might use them as snippets for your pages.”

At this stage you are probably thinking “you have got to be joking if you think I’m going to waste my time writing a description that might or might not appear and which has no ranking benefit. I’m not working for Google, I’m working for me! Let me get on with something more useful like sculpting my page rank or sorting out my 301s. Hey I even used to have a life before I got obsessed with SEO etc etc…”

But hold on. There are other ways to look at this.

To begin with, there’s an important lesson here: appearing on the SERPs isn’t the be-all and end-all of SEO – getting the click-through is. And the best way to do this isn’t via keyword packing but by putting your best case forward for end users to visit your site out of the ten matches on offer. That best case is likely to include the key words that the user is searching on (which are emboldened on the SERPs), but to produce maximum traffic you can do better than that by also including any attractive selling points – “free delivery” – or maybe a hint of humour to indicate that the page was created by a real human being and one that you might be interested in reading.

There’s another important secondary point. You often have to read between the lines with Google and while the description itself might not affect your ranking, that doesn’t mean that it’s irrelevant. In particular it would be very surprising – almost negligent – if Google wasn’t monitoring its own SERPs and using click-through rates to help deliver the most attractive options up-front. In other words, the content of your description might not affect ranking directly, but its click-through rate could be crucial. These days Google is almost certainly monitoring and using visitor behaviour information such as click-thru and bounce back rates alongside web page content and back links.

Both points effectively come down to the same thing and it’s the core difference both between first-generation and current search engine technology and between SEO 1.0 and SEO 2.0: a recognition that the web traffic that really matters is live human beings.

Ultimately you’re not producing your site for yourself, or for Google, but for the end user.