It began with a simple problem: how do I figure out who is accessing my web site and what they are viewing? Actually, it began even before that, with the grand challenge of trying to crawl and index an exponentially growing number of websites in the early 21st century.

But that challenge was unique to Google and a few other players, while analyzing web site access was more ubiquitous among the burgeoning social media sites that were coming of age around the same time.

Low barriers to entry

The solution to both of these data quandaries came in the form of new ways of storing and processing massive amounts of data that broke away from traditional databases and focused on distributed data processing techniques. These techniques also didn’t rely on supercomputers that were only affordable by nation states, but instead could be built from often-unreliable commodity computers and disk drives.

"Almost all literature and video storytelling relies on conflict of some sort, and this one is known as man vs. technology."

Today, these technologies have already become a critical part of low-friction startups that can be built to solve new types of problems. More established enterprises and organizations are also investing heavily in this next wave of pervasive data analytics and expect that investment to continue to grow over the next decade.

Broad application

The possibilities of big data are tantalizing and palpable. Businesses can start to analyze their buildings and correlate AC systems with weather patterns, reducing costs and enhancing planning for heat islands and other effects. Wearable health monitors can provide longitudinal data that feeds into medical planning and epidemiological estimates. Cities can detect road wear patterns and proactively improve responses, improving the lives of people and reducing vehicle wear. Web sites and mobile apps can better contextualize the information that helps you work and be entertained.

Perhaps even more intriguing is the emerging possibility of advancing intelligent systems design and training using big data. Recent improvements in algorithms like simulated neural networks have been combined with big data sets to show human-like—or better—performance on a number of tasks where advances had been stalled for a number of years and incremental improvements were only being made using extremely complicated software architectures.

Automatic image understanding is one such area, but large-scale language translation has also shown advances. In each case, big data collections were a prerequisite, but cheap hardware and big data software infrastructure were important components to the results.

The flip side

We are all familiar with the narrative theme of technology gone awry, from Mary Shelley’s Frankenstein to Terminator. Almost all literature and video storytelling relies on conflict of some sort, and this one is known as man vs. technology, though perhaps we need to update that to people vs. technology in a more modern world.

Big data carries risks, too. We already routinely see data breaches due to hackers exploiting our connected world to gather large-scale personal information. Here we see an area where security, privacy and even physical safety collide with big data. If we can’t secure data and safeguard privacy, everything from transit systems to our physical safety is at risk. For instance, stalkers and thieves are already mining social media for clues about our locations and habits. And inverting this, big data technologies are also being used to detect breaches to find bad guys.

Engine of change

One of the most interesting risks also carries with it the potential for enhanced reward. That is, some economists see economic productivity largely stabilizing—if not stagnating.

Industrial revolutions driven by steam engines, electrification, telephony and even connected computing led to radical reshaping of our economy in the past and leaps in the productivity of workers. But there is no clear candidate for those kinds of changes in the near future. Big data feeding into more intelligent systems may be the driver for the next economic wave, though revolutions are always messier than anyone expected.