Anyone in the IT industry or even anyone watching anything remotely science, technology or business-ey on the BBC at the moment will have been bombarded by the term ‘Big Data’ just this weekend BBC Click did a piece on it and BBC Horizon a few weeks ago. It’s a very much in-vogue term, connected everything, bits of data being recorded and logged from the whereabouts of busses to the volume of water falling from the sky, you name it, it’s probably on a database…
In my opinion there’s a lot of hope and pressure put on the academics and people working in this area of research and I call it research with purpose as at the moment there’s just not the products out there that I would call substantially new and innovative to bear the ‘big data’ label.
I can’t think how many times I’ve sat in a business and thought of a question that I know the computer system really should know and probably does, but just can’t get the data out in the format that I need it to. Maybe the data went in an unstructured format or maybe it’s just buried in so many different places in nested databases, needing to be linked with another just to make sense of it. Indeed the big deal about big data at the moment, which let’s face it everyone is talking about is not big data – its big information. Any GCSE IT student should be able to tell you that information is data that has been processed with some logic applied to it, in order to give it some value or use.
To me, a big data application should be able to make sense of this and ultimately provide me answers to questions that I haven’t yet thought of asking. At present, what we have is Business Intelligence, which is no different to what we’ve had for years. Business Intelligence is difficult, cumbersome and entirely bespoke – a BI project sets out to find the answers to questions that you already know and simply want answers to. It looks at specific sets of data and doesn’t do anything too clever like looking outside of itself for other factors such as weather, stock markets and vaguely anything that could have an impact on what your business is doing – ultimately this is big data, it should not look at only the data that the business produces but external data.
Don’t get me wrong, I’m very excited about this area – indeed I’d hope one day Unleashed will be selling big data solutions if not developing them. However, if you were to put me in front of a board of directors and ask me to recommend anything – it would be a good database, lots of fast affordable storage (which we can provide oodles of!) and a sound BI tool and a clever person in your organisation, trained to ask those interesting questions and produce the appropriate information in reports. I don’t think I’d let industry hype tell you anything more.
It is ultimately a sexy thing for the IT industry at the moment, we can sell you stuff that you probably don’t really need by putting the ‘big data’ brand on it, getting everyone talking about big data will also encourage government spending into universities and research, which will ultimately benefit in some real useful tools – but not yet. So don’t waste your money, just make sure that you’re recording everything… and have the storage to do it – because that data that may not mean anything to you now, may be something that in the future you can make your next leap forward as a business. That may sound as if I’m hyping it too… mmm…
The other interesting thing about Big Data, is many people believe it’s been around for years – what about Tesco and all that Clubcard nonsense. Yes, well okay, they’re dive-bombing a bit at the moment – so they’re obviously not considering some factors external to their organisation, otherwise they would have known that we would all start shopping in Asda and Morrison’s and updated their offering sooner. What Tesco have been very good at is ‘Data warehousing’, they have a lot of data about us, provide tailored offers but I’d say the one thing that they’ve forgot to consider is ultimately this has changed the way we all shop. Ultimately, when they developed their systems they made a judgement of reactions to their tailored marketing messages at that single point in time.
Confused? Well up until the recession, we tended to hold off purchases until we see a buy one get one free, or get a voucher. That was perhaps good during good times when you could afford for a block of cheese to be sat in the back of the fridge, even though you only wanted one. Now people have seen right through the offers and became smarter than the data-warehouse that had been attempting to alter our buying behaviour in favour of the retailer for some time. Indeed, people have become fed up by having their behaviour modified by the yellow stickers in the store and a clubcard offer coming through the post and have begun to resent it and look elsewhere at single offers. High quality produce, sustainability and all sorts of other factors have become important to shoppers who on the whole are becoming much smarter.
This is no different to the Hawthorne effect, where people changed their behaviour because they know they were being watched. Which is all well and good in the workplace, but how much do people like being watched in their shopping habits?
We’re also becoming much smarter because the technology we have in our pockets has enabled us to – indeed the IT in our smartphone, permanently cloud connected, looking into the outside world is probably a more powerful tool (at present) to what the retailers have. You can wander round s shop, do a price check and walk out in no time – you don’t even really have to go into the shop either, you can be at home, at work or on the toilet! Our phone apps are brand spanking new and the data warehouses that have been borne from these business intelligence tools are nearly ten years old or older – never mind Android, we were only just getting to grips with Google at that stage.
Big data when it becomes a viable proposition will solve the problems with business intelligence and data warehousing (by the way, these are pretty similar – data warehousing is just business intelligence on a grand scale) and won’t be caught out by changes in behaviour – big data should be able to predict a change in behaviour before it happens, tell you why it thinks it and offer you an alternative strategy. That’s ultimately what we need.
Despite my criticisms for the terms overuse – this is indeed the future, and there are important ramifications at the moment to consider for society, businesses and governments. It’s good that there’s buzz and excitement around it, I really believe that this has to be – as ultimately the products will come on the back of the hype. However sometimes I wonder whether we are all plugged into the matrix and only have the illusion of choice. I shall be seeking out the Oracle, I believe she’s a database company.
