Corporate entities around the world have been the first movers in harnessing big data for their business needs, with consumer and social media spaces being the most active in this respect.
But the current applications of big data are just the tip of the iceberg. The true potential of big data is both larger in volume and variety, especially when it comes to social good.
The amount of structured and unstructured data is enormous. The ability to sift, collate and analyse pieces of information to gain maximum value from it is a challenge that organisations, governments and policymakers need to understand. Big data can generate new jobs, improve quality of life, optimise resources, advance economies as well as save lives.
By its very nature, big data is empowered to deal with and solve problems of scale. In the same way that big data warrants the collection, collation and integration of disparate forms of data, so do organisations, governments, and large service providers in telecom, energy, and healthcare need to work together to make this possible.
Sectors such as Infrastructure and power can benefit tremendously from the application of insights gained from big data. The analysis of population expansion trends, prediction of power and other infrastructural needs according to the trends being seen in a region, modelling to see, for example, the potential impact of developing specific industries in specific zones – big data analytics can help in policy-level decision-making. Big data can integrate historical information with real time analysis to predict and react to situations. An example is the efficient generation and distribution of power. Smart grids and energy networks can utilise data analytics to identify power consumption patterns and re-direct power at off-peak hours to areas that need it most.
Big data, when employed for social good, can create an impact in ways that were unimaginable just a few years ago. For example, by analysing magnetic resonance imaging (MRI) scans and information from other medical tests, the healthcare sector can create a health map of regions — mapping the prevalence of certain diseases, nutritional deficiencies or deviations and so on. Real-time analysis of medical big data can also play a vital part in the early discovery of a potential epidemic to trigger early-stage counter-measures.
TechAmerica’s report — Demystifying Big Data: A Practical Guide to Transforming the Business of Government — highlighted a case study from the University of Ontario Institute of Technology. An examination of retrospective data revealed that, starting 12 to 24 hours before any apparent symptoms, almost undetectable changes begin to appear in the vital signs of infants who have contracted nosocomial infection. Rather than being a single warning sign, it was a trend of subtle indications that were difficult to spot.
The result was creating an early warning system that gave caregivers the ability to proactively deal with potential complications.
Major challengesThe first challenge that big data throws up is the volume, velocity and variety of data that is generated. The term itself includes inputs from social networks, web server logs, traffic flow sensors, satellite imagery, broadcast audio streams, banking transactions, web pages, government documents, GPS trails, telemetry from automobiles, financial market data, and more. Only about 15 per cent of this information is structured. Unstructured information such as emails, videos, blogs, call centre conversations and social media, makes up the rest. Deriving the value out of this data is a huge task.
The other challenge, however, is one of shortage. There is an acute shortage of talent. Gifted data scientists who are able to understand the sort of data being generated are needed to create the appropriate software to analyse and get value out of it. Given the right stimulus, India is uniquely poised to be a global frontrunner in this field, thanks to a traditional strength in STEM (science, technology, engineering, mathematics) education.
In developing countries like India where social inclusion and increasing access to services and benefits are major challenges, big data analysis can fundamentally change the way citizens, corporates, social organisations and governments interact with each other.
The writer is the vice-president of Hitachi Data Systems