A great read is “Are You Making The Most of Your Dark Data? By Timo Elliot. In it he translates the concept of finding and using a treasure trove of data hidden away in the dark inside organizations. As I read his working example it defined and placed for me an imperative value on getting to and using it to find hidden treasures.
Curious for many of us when we hear of things like Big Data and Dark Data being spoken about, we quickly dismiss it as a mystery or at best a Jules Verne metaphor of something possible in the future.
Elliot talks about data buried inside organizations disparate sources, like routers. He talks about how to find and use it in a very clear example of witless router log file analysis at Copenhagen airport. This information was discovered and used that to plan based on tracking locations of where most people went as they move about an airport.
In Singapore airport last week I had a three hour wait. I can now imagine my phone leaving a trail of pings as I took the Skytrain to other terminals, in search of better shopping. With Hadoop type technology Changi airport may figure out they need a better merchant’s coverage. The tracing of such habits for market planning makes sense go for any large venue.
These days most multi-tenanted retail businesses typically plan centrally using information collected from all their outlets Point Of Sale systems. The big guys have had SAP and similar for 20 or more years so nothing new here.
In the Oil and Gas Industry too, not only is production information transmitted in real time via satellite etc. but also sent is equipment performance data or maintenance planning. Here dark data is also used here to predict and or investigate failures.
It is quite amazing now that Business Intelligence tools can be updated directly too, with ticker information. E.g. Business Intelligence tools in practical use now can gather in almost real-time all point of sale outlets data at the level where say someone buys a bottle of wine at table 35 in the Opera house Restaurant in Sydney. Looking at the dark data would potentially also tell a lot more too such as frequency the type of customer other customers that person introduced; and so on.
In Small to Medium businesses with high speed bandwidth cheaply available and the overall business Intelligence process costs coming down, this reality is easy to implement in any business.
This cloud type medium is fast replacing the summarized in spread sheets or in out-dated Business intelligence summary systems. For those who doing this they quickly leapfrog forward ahead as they are able to see their dark data while those that don’t still remain in the dark.
It is also a concern too that even if the now omnipresent location transmission tools were turned off Malaysia Airlines MH370.by some felonious person, the widespread ability to collect and interpret similar dark data it should make it possible to show us exactly where and what happened.