NoSQL and the monster mutation

Over in San Francisco yesterday, the brand-new NoSQL movement held its first public meeting. 150 bitwranglers from outfits large and small absorbed ten presentations about how to handle data in the new online world. (Disclosure: the meeting was hosted by CBSi, ZDNet UK's own mothership, and organised by Johan Oskarsson from our sister company last.fm.)

The NoSQL movement – if it's ready to be promoted from notion – is one of those regular reactions to the status quo that make life worth living. Very smart people with big ideas are keen to find ways to store and manipulate data that don't rely on the old relational database ideas, or challenge those ideas in ways that aren't appropriate to the standard methods.

There are many overt motivations behind this. Lots of important database applications have arrived with the Web revolution, and many of those have particular size, scaling, performance and reliability needs that might have optimal solutions outside the mainstream. To take the most extreme case – could Google have prospered, or even worked, had it been designed to run on top of Oracle? But that's just Google, right? The NASA of online commerce? You don't get much insight by noting that you can't get to the Moon on a Boeing 737: it doesn't make much difference to the plane you catch to Alicante.

The difference here is that everyone wants to go to the Moon. To be more precise, they want to be NASA – able to pull off the biggest projects at the same time as doing fantastic things with cheap little robots sent all over the place. If your big idea doesn't work on a tiny phone as well as scaling to the point where everyone on the planet can use it, then you're missing out. And "work" isn't just a question of technicalities, it's as much about economic engineering.

That's particularly interesting, because the mountain of maximum economic pain is nowhere near the canyon of maximum technical difficulty – and it's the technical stuff that tends to attract the attention of the young punks of NoSQL. If you're a very small outfit with big ideas, you evolve something very clever because you've got no choice. If you're a big enterprise, you can use your chequebook to beat your suppliers into compliance. It's the guys in the middle who get shafted. You've got mainstream needs but are too small to make waves: the suppliers are much bigger than you, so you take what they give.

Which is why SME problems with database tech – and technology in general – are centred around getting a fair deal. About not being forced to pay too much on very bad terms. About not ceding control over the company DNA to people whose business objectives are not your business objectives. It's a classic balance of power problem, one illustrated by comparing the balance sheets and profit margins of those very large software companies who supply corporate IT with those of the SMEs who consume it.

For NoSQL to properly change the world, it has to be fit to climb that particular mountain of pain. Like the open source movement from which it comes, it will find this the very hardest task. This particular Everest is frustratingly immune to the weaponry of innovation, lateral thinking, energy and idealism unless harnessed to a focussed, patient, strategic assault. And focus, patience and strategy are not attributes commonly found among bright young punks.

Movements do throw up monsters, though, and its the monsters who devastate. So enjoy NoSQL for the thinking, the arguments, the technology, but watch it closely for its very own Gates, Jobs or Ellison. Sometimes, only a monster will do.