Weighing the Opportunities and Challenges of Big Data
Business Solutions Big data is transitioning from the classrooms of computer science and machine learning, into numerous real-world scenarios in business, government and military, science, politics, medicine, climate and personal analytics.
Imagine the following scenario: You sit down at a restaurant for lunch, wondering what you are going to order. You take a picture of the menu with your phone, starting a whirlwind of activity.
The software combs through the data it’s collected during the day about you, such as your breakfast, exercise and calorie expenditure, blood pressure and blood sugar levels, etc. It combines this data with long-term information such as previous user reviews from this restaurant, your weight loss goals, food preferences and perhaps even your individual genetic properties.
Not just science fiction
In milliseconds, it makes its top three suggestions, from which you choose your meal. In the meantime, data is collected anonymously in the aggregate about your choices, as well as those of other patrons. The restaurant manager can use it to adjust the menu. Researchers can use it to better understand the relationship between nutrition, fitness and health. Your friends can use it to obtain personalized food recommendations, and you can use it to track your progress toward your health goals.
"To fully exploit the opportunity promised by big data, we must find ways to bridge the gap between data growth and processing capability."
This scenario exemplifies how business intelligence and big data analytics can be used to seamlessly affect decisions from the prosaic (your choice of lunch) to the strategic (the FDA’s nutrition guidelines). It represents but one of many opportunities envisioned for the large-scale analytics of diverse data.
To fully exploit the opportunity promised by big data, we must find ways to bridge the gap between data growth and processing capability.
This requires innovations in both hardware and software technologies. On the hardware side, it would take radical technology shifts to match resource growth with data growth. Barring an unpredictable disruptive technology, a more feasible path to closing the gap is innovation in software. Big data software can be considered still in its infancy, with plenty of opportunities for growth, such as:
Reducing the amount of data to be processed with better compression, early detection of irrelevant data and more effective sampling techniques
Better utilization and sharing of available hardware resources
Analysis that produces much higher-level analytics and answers than is standard today
Keeping a human touch
Beyond that, though, technical solutions are useless without specialized human skills. In a recent study, 88 percent of companies surveyed have already reported a talent shortage in the area of big data analytics. This challenge needs to be met with better workforce education, both in academia and industry.
Other aspects of the big data shift will require societal response. Perhaps the biggest one is the concern about eroding privacy and data leaks, with a potential for very significant personal, business or military damage. The concentration of big data in the hands of governments also evokes concerns about the risk to democracy and civil rights.
The challenge is then to find ways to collect, share and benefit from big data technologies while still preserving the privacy, trust and rights of the individuals whose data is collected.