“Towards the Quantified Society” was the theme of O’Reilly’s online preview of its Strata conference in Santa Clara, Calif., in late February. The statement does make one tingle a bit – we’re headed in the direction of a data-centric world where everything can be measured, morphed into numbers and metrics, and then crunched through some algorithm or other to discern meaning. Every action is turned into bytes; our lives are broken down into the juiciest bits of information (and cross-referenced for optimal effect).
It’s not that we don’t see the progress already made toward the quantified society – Edd Dumbill, Strata Chair and O’Reilly Lead Architect of Conferences, commented that the mainstreaming of big data is in full effect, with NPR and major (traditional-based) publications giving the concept lip service. Indeed, he called the prior year’s conference a “coming out party” for data scientists – professionals across a plethora of fields discovered that their obsession with data had turned them into the new rock stars. The past year has seen a great maturation in the enterprise data services available, he added.
In his introduction, Alistair Croll, Partner at Bitcurrent and Cochair of the Strata Conference, said that every person is now both an interface and a sensor, processing and sending data out into the digital vastness. He questioned whether we’ll actually be numbed by the constant humming of algorithms.
Because it’s not the road to the quantified society that gives us pause, but the endgame. The data-driven society that we’re moving towards doesn’t really resemble the current world, Croll commented, so discussion becomes necessary to paint a picture of this future.
This is not to say that the digital populace is freaking out over the big data revolution; in a preview of a panel on privacy, John Wilbanks of the Kauffman Foundation for Entrepreneurship suggested that Internet and mobile users are less seeking privacy than data empowerment. People want control over who and what gets their data when and where not to ward off stalking by trackers, but so they can take advantage of their data. It’s becoming clear how valuable information can be. Wilbanks said the panel would argue over the clarity of terms in dealing with data as well as the portability of usage rights.
To give a sense of the rapid growth of data production, Paul Dix of mobile analytics and app recommendation system Flurry shared findings garnered from the 400 million unique smartphones the company receives data from each month – that’s 109 million in the U.S. alone, but he noted the greatest traffic growth was coming from China. The proliferation of data is outpacing the company’s projections – now analyzing 1.2 billion mobile app sessions daily, Dix commented that Flurry broke the 1-billion-session barrier around New Year’s when they expected that milestone to come during tax season.
Lest the name throw you off, size is not the only concern around big data. Dumbill broke big data down into three Vs: volume, variety and velocity (the last one better known as speed). Companies are coming to terms with the volume aspect thanks to management offerings like Hadoop and Google’s data system, which is now in beta. Now speed is top of mind – in particular, dealing with streaming data as it rolls in, real-time analysis.
However, Dix noted that the raw hardware and software necessary for establishing real-time data analysis is not cheap – is speedy insight worth the cost.
Finally, echoing the IAB/Winterberry Group report released last week, Dumbill suggested that the industry is still stymied by what he termed “The last mile.”
“We can grab all the data in the world, but we as people must still make sense of it,” he said. “That seems like it’s still in its infancy.”