Editorials

Big Data – PHOOEY! Just Show Me The Crystal Ball!

Big Data is so many things to so many people. To the boss, it’s predictive information. To database professionals, it’s a lot of things together – velocity, volume, all those things that have come to define it officially.

More and more I see projects struggling because the chasm between what one group defines it as vs what another defines it as is growing, not shrinking. To make matters worse, it’s not only about different types of users, but industries can’t seem to define it in a common way either.

This column isn’t going to go there either. No worries there.

What I would like to do is hear more about how you’re working with these groups to get the work done. At some point, you have to agree on a goal and the source(s) of information to help inform that goal and get busy.

My process thus far has been something like this:

– What do you want to learn? *
– What information sources do you have available?
– When do you need that information? How will it integrate into the company flow? **

These initial questions usually point out a few of the roughest "getting started" points. First, the "*" – this usually breaks the process straight out of the gate. So many people have been told "feed your data into a big data system and it will see new things about your business, predict trends, tell you new insights." People don’t realize in too many cases that they’ll have to determine the questions, the figure out how the data answers those questions.

In talking with customers we work with, and with our own projects it seems to always come down to the same thing. Start small. You know what types of information typically come from the types of data you do have. Go after those answers first. Get the hang of delivering those answers in near-real-time. It’s not the same as building the report and sending it out to the report subscribers. It’s a different process, different presentation.

This is where Ben’s Dashboards (check out his editorials of late too) come in. You need to have a different presentation layer – it’s a different type of information and a different flow and update of that information. As an aside, one of my most iconic BI projects was about a large insurance company and their President. He wanted to have a very simple "good, ok or bad" indicator. We created a single page with a traffic signal on it. Green, Yellow and Red. It did the trick – they could immediately see if the items they cared about, taken together, showed that things were going well, or there were things to drill down into. The simplest of dashboards – but very effective.

That second question with the "**" is a harder one to help people through. It’s another reason to do a small project first. You have to change how and when you provide information to the users of your information. Previously, you provide a reporting share, perhaps even a printed(!) report. A big data system thrives on real-time feedback and information for the users of the information. You’ll want to go in, eyes open, and address the fact that you’ll need to incorporate the new information flow in the work flow of the company. If you don’t, you end up with lots of information, some great dashboards or information bits – that no one is paying attention to. It’s critical that the integration of the new information flows is part of the projects you set up.

What other questions do you ask? How do you approach it?

What do you think? Let me know…

swynk@sswug.org