First of all, let’s get our terminology straight. Depending on whom you ask, “big data” is either:
A. Bullshit (Brad Feld)
B. No substitute for judgment (David Friend)
C. The marriage of corporate data with external data (Chris Lynch)
D. Data that’s growing faster than Moore’s law (Richard Dale)
After this week, my answer would be E, all of the above.
On Wednesday, we had a packed house of 300-plus attendees and a star-studded roster of speakers at our “Future of Big Data” conference. Huge thanks to our sponsors, and to our gracious hosts, the Fidelity Center for Applied Technology, for making this event possible.
In addition to the above insights—and the overarching notion that you have to go beyond the label of big data to get any real value—here are my top 10 highlights from the day, in the form of quotes (some of these might be paraphrased as I was busy moderating):
1. “Twenty years from now, the thing we call ‘big data’ will be tiny data,” said Brad Feld of Foundry Group. He implored us to look beyond the marketing hype of big data and put the buzzword in proper perspective—reminding us that Massachusetts was once home to the minicomputer revolution (with DEC, Data General, Wang, etc.), only to get passed up by PCs and the Internet.
2. “Systems know more about you than you probably want them to,” said Carbonite’s CEO, David Friend, raising the specter of privacy concerns in big data. He emphasized that his company doesn’t allow anyone access to data backed up by its customers. The whole notion of Carbonite (which adds a petabyte per week) and other backup companies reminded me that although the Web never forgets, your personal or corporate data can be lost, mishandled, or breached.
3. “The intersection of big data in the cloud and big data in the enterprise” raises issues of both privacy and security, said Burt Kaliski, CTO of Verisign. (That also touches on a definition of big data above, from Atlas Venture’s Chris Lynch.) EMC’s chief IT architect, Narayanan “KK” Krishnakumar, talked about the importance of “transparency” and “selective sharing” of information in the face of IT threats such as social engineering. What’s more, Andy Palmer (from Cloudant, Vertica, and other companies) said the biggest corporate threat is “disgruntled IT employees.”
4. “The interesting thing is what big data tells you over time.” That was Peter Stern, CEO of Bitly, on the importance of doing long-term analytics instead of just looking at snapshots. Bitly analyzes what people click on, not what they talk about (actions speak louder than words); Stern also emphasized the value of the long tail of Web traffic, rather than the spikes. Bitly estimates that it sees 1 percent of clicks on all new content.
5. “The database people have been asleep, and Oracle has been providing the narcotics,” said Andy Palmer. The big data infrastructure
