Call it the big data paradox. On one hand, new software technologies can solve problems once considered far too complex or unwieldy. Analytics, for instance, are being employed to reduce traffic congestion and pollution in megacities in Asia and Africa.

On the other hand, we need new hardware, software, and other technology to handle the staggering amount of data being produced. Simply put, we generate more data than we can handle the old-fashioned way.

Quest to improve health care

A single human genome requires approximately 200GB of storage. Sequencing a million human genomes—the White House’s goal—would require approximately 200 petabytes, or about as much data as Facebook uploads in a year.

"Big data can give us greater insight and understanding of everything around us to create a better society. But it’s also going to require a tremendous amount of effort behind the scenes."

“It’s a matter of physics,” says Talli Somekh, CEO of AppSoma, a startup focused on computational biology. “Some of the data sets are so large you just can’t move them to the cloud.”

Or take video. If we could better analyze video streams, architects could design buildings to correspond to interior traffic patterns. Accidents could be explained. Our understanding of the real world would expand exponentially.

Video, however, also requires an incredible backbone. A single minute of UltraHD requires about 5.3 GB of storage. The volume and velocity of information coming from thousands of camera could easily overwhelm us.

Matching the pace

Achieving these aims won’t be easy or practical with current data centers and technologies. In many companies, less than 50 percent of the computing cycles get used for productive work because data simply can’t be delivered fast enough.

They also use a substantial amount of power. The NRDC has calculated that U.S. data centers consumed 91 billion kilowatt hours in 2013, or two-times the amount of power used by households in New York City. Unchecked, the figure will rise to 140 billion by 2020.

Many data centers still rely on traditional hard drives, which, like record players, are mechanical devices. Solid-state technology, which can reduce power by 80 percent while increasing performance by 50 percent, software for better utilizing assets and even new cooling technologies will all be needed.

Big data is one of the magical concepts of our era. It can give us greater insight and understanding of everything around us to create a better society. But it’s also going to require a tremendous amount of effort behind the scenes.