Everyone seems to be talking about Big Data. Organisations across sectors are buying into the premise of data-driven decision-making and growth. Studies show that the overall adoption of data science and analytics at large Indian firms has been as high as 64 per cent. Analytics, data science, and Big Data in India are set to become a $20-billion industry by 2025. The reason is clear — data holds transformative potential, and organisations across the spectrum are ready to invest in this precious asset.

Despite this revolution, organisations still face challenges when trying to acquire the most fundamental of datasets — those on consumers. While Big Data insights — generated through social networks, the Internet of Things (IoT), large business databases, etc — continue to grow in number, they often fall short when organisations need simple behavioural insights on consumers, their choices and usage patterns. A growing concern as data becomes indispensable to decision-making is whether the right information is being used to drive decisions that directly impact the well-being of consumers.

Rural consumption

The problem today is not about harnessing more data, but rather capturing data that is actionable. Amidst the excitement surrounding Big Data, one needs to reaffirm the merit of primary consumer data that directly reflects the choices, demands, and consumption patterns across socio-economic and demographic scales.

Data on consumption is integral not just to private players, but to an entire spectrum of organisations, including governments and philanthropies, who require evidence-based local insights to formulate policies and interventions. Organisations across the board are deprived of this demand-side consumer data. When it comes to rural consumers, data becomes even harder to find. This gap is a clear reflection of the lack of representation of rural choices and demands in existing data sources.

Without social and demographic information on consumers, it is impossible to contextualise the vast amounts of tech-enabled datapoints that are produced literally each second. For example, tracking digitisation and financial inclusion by harnessing mobile payment data without any social context on the nature of livelihoods, income, or educational level would be akin to telling only half a story. Without information on how farmers use fertilisers, seeds, or irrigation techniques, any sensor or satellite data on agriculture is devoid of crucial human and behavioural elements.

It is faulty, even dangerous, to make decisions that directly impact rural consumers without primary socio-economic and demographic data from the demand-side. In order to enhance lives and meet demands, it is crucial to reinforce consumer data, particularly that of rural consumers. Big Data might not necessarily be the magic solution for impactful decision-making. However, social and demographic indicators that may offer 360-degree insights on consumers can be.

Data shortcomings

The low levels of primary data usage might be a result of its own shortcomings. Household data on social and demographic indicators is limited to over 10 large-scale surveys, such as the NFHS and the NSSO, and several smaller isolated and fragmented surveys. While these are excellent sources of data, collected with immense precision and statistical rigor, the scope to derive cross-cutting insights across indicators and geographies is low given the overall lack of integration.

Further, surveys are expensive affairs beset with long turn-around times. The DHS is rolled out every five years. The Census is released at the end of every 10 years. Data, like other commodities, comes with an expiry date. To enable effective decision-making, it must be consistent with the fast and ever-changing nature of the social landscape. Indicators as dynamic and fast-moving as health and consumption require data collection that moves and reflects changes in real-time. Unless data collection intervals can be reduced and insights can be generated with speed, primary data, despite its transformative capabilities, will be of little use to decision-makers.

Innovation is key

There is a pressing need to reorient how we think about primary demand-side data. Can we reevaluate the arduous, labour-intensive practices that surveys in India have traditionally taken? Can we start considering how technology can streamline physical data collection, replacing it with faster, more automated processes? Innovation might be where the answers lie.

Re-inventing and consolidating primary data practices is key to making it relevant and indispensable to decision-making. The conversation must move from solely speaking about Big Data, drones, satellite imagery, GIS, etc to how these can be integrated with knowledge that represents people and their choices. Research think-tanks, universities and government entities together must take the lead in driving forward this innovation and seismic shift in the possibilites of quality data.

Shekhar is Director, Sambodhi Research. Patherya is Programme Manager, Sambodhi Analytics

Related Topics
comment COMMENT NOW