Imagine the following scenario: You sit down at a restaurant for lunch, wondering what you are going to order. You take a picture of the menu with your phone, starting a whirlwind of activity.

The software combs through the data it’s collected during the day about you, such as your breakfast, exercise and calorie expenditure, blood pressure and blood sugar levels, etc. It combines this data with long-term information such as previous user reviews from this restaurant, your weight loss goals, food preferences and perhaps even your individual genetic properties.

Not just science fiction

In milliseconds, it makes its top three suggestions, from which you choose your meal. In the meantime, data is collected anonymously in the aggregate about your choices, as well as those of other patrons. The restaurant manager can use it to adjust the menu. Researchers can use it to better understand the relationship between nutrition, fitness and health. Your friends can use it to obtain personalized food recommendations, and you can use it to track your progress toward your health goals.

"To fully exploit the opportunity promised by big data, we must find ways to bridge the gap between data growth and processing capability."

This scenario exemplifies how business intelligence and big data analytics can be used to seamlessly affect decisions from the prosaic (your choice of lunch) to the strategic (the FDA’s nutrition guidelines). It represents but one of many opportunities envisioned for the large-scale analytics of diverse data.

To fully exploit the opportunity promised by big data, we must find ways to bridge the gap between data growth and processing capability.

This requires innovations in both hardware and software technologies. On the hardware side, it would take radical technology shifts to match resource growth with data growth. Barring an unpredictable disruptive technology, a more feasible path to closing the gap is innovation in software. Big data software can be considered still in its infancy, with plenty of opportunities for growth, such as:

  • Reducing the amount of data to be processed with bet­ter compression, early detection of irrelevant data and more effective sampling techniques

  • Algorithmic efforts

  • Better utilization and sharing of available hardware resources

  • Analysis that produces much higher-level analytics and an­swers than is standard today

Keeping a human touch

Beyond that, though, technical solutions are useless without specialized human skills. In a recent study, 88 percent of companies surveyed have already reported a talent shortage in the area of big data analytics. This challenge needs to be met with better workforce education, both in academia and industry.

Other aspects of the big data shift will require societal response. Perhaps the biggest one is the concern about eroding privacy and data leaks, with a potential for very significant personal, business or military damage. The concentration of big data in the hands of governments also evokes concerns about the risk to democracy and civil rights.

The challenge is then to find ways to collect, share and benefit from big data technologies while still preserving the privacy, trust and rights of the individuals whose data is collected.