A little over a year ago I became the CEO of a company in the “big data” space that is developing predictive analytics software. My CTO is the “data guy,” and I’m the “business guy.” While my background is technical, I had very little experience in data or analytics and only knew about big data from repeatedly seeing it in the press.
I thought I’d share some of the insights over the past year that helped me wrap my head around all the big data hoopla. I find it impossible to learn something unless I have a mental framework on which to hang new concepts. Without a good teacher to provide one, half the battle for me is creating that framework. I’m hoping that by sharing what I’ve come up with, others can get up to speed a bit more quickly.
These insights are simplified distillations that allowed me to put the things I was hearing into a useful context. This allowed me to talk to others about the big data concepts in my own words and sound like I knew a bit about what I was saying.
The first question I had was why the boom of big data is happening now. I concluded it is because of a number of technological changes arising around the same time and colliding in a perfect storm:
—Due to the Web and growing “Internet of things,” the number of companies with “big enough data” has exploded into a market big enough to be worth addressing by many solution providers.
—Cloud technologies are available and affordable, allowing development of solutions that require computing resources beyond the scale most companies can afford to manage themselves.
—Leaders in the big data space, such as Google, Amazon, Facebook, and others, have developed solutions for handling big data and made them available as open source software.
My first insight was to realize that today the label big data is, for the most part, a vague marketing term used to describe any product that interacts with data in any way at all. It is important to belong to the big data club, so if a company can plausibly claim that its product is a member, it will do so. One company’s definition of big data may have no relationship to another company’s definition.
Many big data products are “just” traditional business intelligence, visualization, statistics, or database applications modified to scale to work with much larger data sets without really changing their functionality. I put “just” in quotes because making these changes is often quite difficult. However, the conceptual capabilities are not new, despite the claims their marketing departments might make.
I found that mentally replacing
