Is Big Data the new panacea or just a new recreation of nothing new?

As big data centres move to the so called “Cloud” we are seeing a metamorphosis of other data generally joining  the race. integrating your data to extend its value downstream and upstream is becoming the goal of the giant enterprise business  technology players, to sell competitive positioning. This in turn is spawning the watching software applications world to get in the game with innovative ways to join and manage it all.

It is close to a decade since Business Intelligence became a new concept to get business information into a practical analytical form. That  innovation allowed people to move much faster to make more intelligent decisions and have the confidence for innovative change using new found linked analytics.

We all know the devils are in the detail, but the new emergence of the big data concept to help find them, may actually make it easier for devils to hide. Hence being deluged in cloud bursts of data is now a big risk and we need to progress on that front quite carefully.

With Business Intelligence moving to connect more widely based data, getting robots to crawl around and look for things that are odd, makes very good sense. E.g. detecting fraud and crime or even preventing it etc… But relying on program analytics alone can dull the senses too and render useless even the highly sensitive listening ears of the always alert Blind-Freddy.

The latest hype and new software concept boom underway with Big Data everywhere, is to get people to tap into this using new cloud based software and technology stacks. Vendors now urge us to make the  investment with warnings of the imperative to be step ahead on the so called competitive escalator. Even though its in the early adopter stage, this new way to work in a Doggle-IT style information world is here and wont go away.

I have no doubt it will all come true, but not if the reason for being is just having Big Data to have a better BI, without a culture to use it.

clip_image001Losing sight of the fact that decision making and innovation is the purpose data serves can become an enterprise millstone that could bring you undone. When business takes on a new you-beaut solution, it is important to check it is not just old mutton dressed up in a new lamb skin or worse a wolf in sheep’s clothing.

Other so called new cloud based concepts that go hand in hand within data and also being bandied about as new are “Business Rules Engines” These purport to let you turn your new found big data into a new business modeling wonder. Think carefully before you go off and spend your CAPEX money on new Rules Engine software. Do you really want to do all that so called data mining magic that you imagine will give you an edge? And do you already have that capability? Most do and if tweaked be may do the same job or better.

Big data can also evolve illogical bad processes. When relationships are found to fit some ill conceived political end, to “justify a required answer”. the methods often sustain. We all know figures can lie but lies cannot figure. Hence keeping it simple is even more important to stop these “Clouds” covering good judgment, with incompetence then able to reign over common sense.

A lesson for many is that complex big data analysis capability should be used intelligently as foundation to support a more simple thesis that can be understood.. Conversely having too much data can lose you the game too as you lose sight of the goal.

Some years ago in my local neighbourhood, a town planning hearing judged a community dispute with some greedy developers. The developers had sought to rezone a parkland they claimed was not used. The man in silk on the community team won on the day. When challenged about his small statistical sample of the actual use, he argued the small sample size was of equal value to doing a larger survey. In his case his high price tagged QC credibility may have been what carried the weight but the judge accepted his argument and awarded in the community favour.

Our opponents presented veraciously with a string of expert witnesses to present a great deal of detail. In the end the judge ruled it had proved our QC’s point and we got to keep our park .

It is generally known that the next curve of change starts with s new step change. (E.g. having a big data base was of no use to ice-makers when the fridge got invented.) We must therefore question why we keep big data for many years even down to the SKU level to give us a competitive edge later?

On the other hand we know NASA would have never invented Kevlar if they had not collected data, but in the end it was actually innovation that saw it born. Much like the  breakthrough this week in Australia after 20 years of laboratory research.That cracked the puzzle on how insulin works  Storing all their data was useful to keep it motivated using good historical test information to move up the discovery tree . But in the end it was new innovation that actually got the result.

While data mining keeps analysts busy, I question that we should make the Analyst more a part of the business process and not just in planning. The CEO who is emulated most is the best leader of change in a business and often the best analyst too. Any strategic CEO will know that setting a culture of measured information based  behaviour yields well.  Key driver based measures that get to the payload fast don’t waste people’s time looking for the gold in the tailings.

But the hard work is still turning data into value.  Cleaning data and then figuring out how to use it still requires cognitive effort. Only then do we have capability for new innovation and control. E.g. Even with access to and using mass data environments means marketing still needs the skills to ask the questions for segmented demographic decisions. Now with better community level data they can focus and get better value.

There is no doubt that with the capability to analyse mass data environments using good tools, the opportunities for entrepreneurs are now boundless. We also know big data as the lifeblood of all forms of endeavour is not a panacea unless it is distilled and put to cognitive use. Constantly re-questioning is always the challenge to find understandable information to make our decisions. That is where good efficient tools come in handy to get new innovations to move from a flying in the Cloud to a data driven clear sky.