Home / Opinion / The promise of big data analytics

The digitization of our daily lives generates a flood of data: from our phones, credit cards, tax records, car movements, television usage.Those who have the computing abilities to trawl through this river in spate can tell your online bookseller what your reading habits are or they can help city managers plan traffic flows better.

The fictional detective Sherlock Holmes was surely not thinking of big data but he captured the essence of the opportunity when he said: “Having gathered these facts, Watson, I smoked several pipes over them, trying to separate those which were crucial from others which were merely incidental".

Analysis of big data is ubiquitous today: first in companies but now increasingly in government as well. It is for this reason that we welcome the recent moves by the Indian government to use data analytics to help it design policies based on empirical evidence rather than fond hopes.

The starting point should be the Aadhaar database. There are already over 700 million registered users.

This newspaper reported earlier in the week that the government is looking at ways to bring data analytics, supercomputing and the Aadhaar database together so as to design better public policies.

“When you have that much amount of information, you can do a lot of data analysis and figure out trends. We can thus see what are the policies needed to be defined. Therefore proactive policymaking based on previous trends is possible provided we have data," said Rajat Moona, director general of the Centre for Development of Advanced Computing in Pune.

Singapore’s Land Transport Authority regularly crunches 20 million fare transactions per day to better understand the transport needs of its citizens. Based on the extensive usage data, it has been able to build more cost effective routes for cars, buses and trains, schedule the best timings and lowest fares. The remarkable aspect is that while doing so, it has also been able to reduce the cost of its revenue collection.

Similarly, according to a McKinsey & Co. report, the Bundesagentur für Arbeit (the federal labour agency in Germany), with a €54 billion annual budget and 120,000 full-time employees, not only improved its customer services (by reducing the unemployed from 4.4 million to 3.2 million) but also cut around €10 billion of costs between 2003-2010 using big data strategies.

However, such revolutionary ideas are largely alien to India’s policymaking scenario. The ad hocism in policy choices reflects not just in India’s persistently low rank in the World Bank’s ease of doing business index but also in the country’s grossly inefficient public sector—be it social welfare schemes or the administration of law and order.

Take the recent example of the extent of leakages in India’s public distribution system (PDS). There are several competing numbers doing the rounds. The only commonality being that they are all unacceptably high. Beyond the mere numbers, there is little unanimity among researchers about the ability of PDS in improving nutritional outcomes. A big reason for this is the fact that most conclusions are based on minute sample surveys instead of the full data.

But in a country like India, big data can create its own problems. Herbert Simon, who received the Nobel Prize for economics in 1978, had once said, “A wealth of information creates a poverty of attention and a need to allocate that attention efficiently among the overabundance of information sources that might consume it."

Yet, the potential use of data analytics in evidence-based policymaking is immense. The mobile phone call data records, which proved critical in reaching out to the victims in the aftermath of Haiti’s earthquake in 2010, could be used for better, even predictive, policing. Similarly, analysis of the farm related queries in Kisan Call Centres can cull out tremendous insights about the exact needs of the farmers. The government can, in turn, customize its farm support measures across fertilizers, seeds and irrigation in a far more efficient manner.

Of course, the use of big data also raises questions of right to privacy and the freedom of choice. But most Orwellian nightmares are exaggerations that can be addressed by well-designed privacy legislation.

Should India push aggressively for more evidence based policymaking? Tell us at

Subscribe to Mint Newsletters
* Enter a valid email
* Thank you for subscribing to our newsletter.
Recommended For You
Edit Profile
Get alerts on WhatsApp
Set Preferences My ReadsFeedbackRedeem a Gift CardLogout