Under the covers

For too long search design has been dictated either by the search engine technology i.e. "Keywords" or by the agregators (Google/Yahoo/MSN) ranking algorithms. Both have their benefits but neither necessarily serves the user the best.

Keywords work perfectly if you are searching for a product. If I want to know about the Treo 750W then typing that into any reviews site or search engine I'll get very specific results. However if I wanted to find out about Smartphones, then things start to get messy. Firstly the term Smartphone means different things to different people. Secondly most product reviews probably won't even mention the word "Smartphone" in them anyway. So keyword searching on a topic is a very hit and miss affair.

Using the main search engine sites, the problems are different but equally vexing for the user. At the mercy of the algorithm, users are now provided with content that has been deemed by some invisible mechanism important.

Google changed the landscape here by working out that what the users thought was important (i.e. Number of links to that content) was actually more relevant than the most recent content with that keyword. Of course, it's not as simple as that and the big three spend millions of dollars tweaking and tuning their "special formula" with as much secrecy and security as Coca Cola.

Natural language search engines have been around for a long time but the problem has always been that it takes around 100 times more computing power to index and search in this way compared with conventional keyword search. Now, with processing power becoming cheaper and cheaper, we are starting to see investment ramp up in these technologies. Cited in this article, there were 47 search start-ups in 2005 raising a combined $260M.

And here on ZDNet, we are starting to experiment with this technology too. Our new search engine has the ability to automatically create relevant tags and topics related to the search term. It's still early days for this type of technology but we believe that this type of search is the only one that considers the user first. Up to 75% of search terms are a single word and our most popular search term is often "Microsoft".

Given the size and scope of Microsoft's market, this would lead you to believe that the user doesn't know how to search for what they are really looking for. And the truth is, most users are working blindfold when it comes to understanding conventional search syntax so they go in one direction until they bump into something, re-adjust and try again.

By creating tag clouds of the most relevant tags and terms, we are narrowing down the search for the user and helping them towards what they were really looking for. Go on, give it a try, take off your blindfold, click here and check out the related tags box.

One of the things we've learned while developing this site is that the old "Web 1.0" model for building web sites doesn’t apply any more.

We used to have a pretty slick production line that consisted of Design building pages, Production turning them into HTML and Tech engineering the code into the content management system (CMS).

As we developed the new site, particularly the Community section, we found that that model doesn't work any more.

In a "Web 2.0" world (whatever that is) we're returning to the age of the developer. The Community area on ZDNet V is all about functionality rather than style. Sure, it looks good but the emphasis has shifted towards more traditional development. Where once the on-line world coveted pure design, what we need now are people with experience in useabilty and Human Computer Interface (HCI) skills.

What's interesting is that these are not new requirements, application development was around long before the web was even thought of. Web businesses are desperately hiring developers as fast as they can and the dangers are clear, application development requires a completely different process to web site development.

Changing that is difficult, as a result of the ZDNet V experience, we are completely re-engineering our development processes to put more emphasis on functionality without losing any of the design ethos that has made us successful so far. It's not going to be a simple process but we've made a start.

The challenge for on-line businesses is strangely converse to those that faced traditional businesses when the world went on-line in "Web 1.0". Those businesses struggled to get to grips with Design and HTML skills compared to the new media upstarts. We were the Cindarella's to their ugly sisters and we had a great time at the ball.

Now us upstarts would do well to look to the experience that the traditional companies have had all along. That is the ability to build robust, functional, easy to use systems and adapt that experience to make them accessible via the web browser.

This is the next big challenge for both traditional and new media businesses but for once, the old school businesses might just have an advantage.

Time for the ugly sisters to get a makeover and pull on their party dresses?

My name is Mike and I Google myself. There, I said it, I feel better already, like a weight has been lifted from my shoulders. The last social taboo has been broken and I'm proud to be the first.

We all do it of course, its the web 2.0, Wikipedia equivalent of an entry in whose who? But nobody admits to it. Well actually I have a friend that delights in telling me that she occupies the top 3 spots when you Google her name but she's a strange girl.

I've never really cultivated much of an external profile as I'm in the engine room most of the time, shovelling coal into the boilers. I speak on the occasional panel, get quoted on silicon.com and here on ZDNet every now and again and I'm sure my bio is on the corporate web site somewhere...

So I thought I'd do a little experiment to see if my blogging here would have any impact on my anonymity. Last week I spent a very unhappy hour typing my name into Google and looking for any references to this particular Mike Barrett. Firstly I have to point out that I'm up against it here. There's a congressional candidate, an NBA basketball player and closer to home, that nice doctor on casualty (wrong spelling though). If you add to that the fact that I recently got invited to join the "Michael Barrett is Awesome" group on Facebook (current membership = 46) then you can see there is some serious competition.

After an hour, and limiting the search to the UK results, I finally located an entry about me appearing on a panel discussion (in 2005) on page 12. My friend would be laughing her pop socks off!

Then early this morning, I got a Google email alert (which is a sub taboo thing to do) which I set up for “Mike Barrett” and lo and behold it was about me! Excitedly I logged on and clicked through to…

My blog posts in ZDNet! Not completely convinced this was some kind of fluke I Googled myself, and after looking through the fifth page, I started to get despondent. Then I selected the “Pages from the UK” option and tried again and…

Excitement is not an adequate description! There I am on the first page! Not just once but three times! I almost call my friend before realising its 5.30 am and she'll just swear at me.

The point of all this rambling? Blogging on ZDNet increases your social standing and boosts your ego. I've gone from being a nobody to a minor somebody in less than two days.

Now I may never be a presidential candidate or play in the NBA, or even rival my friends ranking. But, when those other 46 Mike Barretts on Facebook Google themselves, I'll be up there in third place. Maybe they will elect me king....

So it's nearly 37 hours after launch now and a relative calm has reurned to the Operations area. The first day wasn't without incident, some quirks in the publishing process caused a few problems on the site but nothing that couldn't be fixed.

We had a mini celebration at 4pm yesterday and raised a glass to the new baby, it was amazing to see the number of people that have been involved, over 40 at the last count from the various teams around the business.

Now a long list of P1/2/3 bugs stretch before us to get the site where we really want it to be and then on to the next phase...