Big Data can rejuvenate India’s statistical system

Look, compare, click: A growing number of urban consumers now prefer to buy essentials online rather than trudge to the local market. Can online prices be used by policymakers to understand inflation pressures in the economy?

In a recent paper published in the Economic and Political Weekly, Ritwik Banerjee and Chetan Subramanian of the Indian Institute of Management, Bangalore (IIM-B) have teamed up with Nished Singhal of data science company NeenOpal to construct an index of online prices, using data from online retailer Big Basket. Their index closely tracks the official consumer price index that is the formal nominal anchor of Indian monetary policy. In other words, online prices can be used by policymakers to understand inflation trends well before the data from official sample surveys rolls in.

India once had one of the best statistical systems in the world. That is history. The past few years have seen heated debates on whether official statistics reveal the true state of economic growth, industrial production and job creation. Policymakers have been flying with broken radars. One former Reserve Bank of India (RBI) governor politely described the official industrial production numbers as analytically bewildering. Other critics have been much harsher.

Big Data analytics is now gradually creeping into Indian policy debates. Economists at the finance ministry have used preliminary data from the goods and services tax network (GSTN) to understand the patterns of trade between states. They have also used data from the railway system to understand migration patterns in a country on the move. Economists Soumya Kanti Ghosh of the State Bank of India and Pulak Ghosh of IIM-B have used payroll data from the Employees’ Provident Fund Organisation to capture the state of job creation in formal enterprises. Researchers at the private sector think tank IDFC Institute have used satellite images to show that India is far more urbanized than the census suggests. Praveen Chakravarty and Vivek Dehejia have bolstered their case about growing regional inequality through the use of night-light images. The RBI has announced that it will be setting up an in-house lab for Big Data analytics.

Much of this work is still in the nature of academic exercises. Big Data has not yet entered active policymaking, especially in the generation of high-frequency indicators that can give both policymakers as well as the financial markets a more regular snapshot of the Indian economy. The case for its use is a compelling one. It is relatively easy to collect, is available real time, and is less prone to human sampling errors. However, how closely it tracks developments in the economy needs to be more rigorously established. For example, is the monthly GSTN data a good lead indicator of activity in the real economy? More statistical analysis is needed.

None of this means that the existing system of data collection through sample surveys should be allowed to wither away. The need to get it back on track is still important. This newspaper has earlier commented on the need to rebuild the Indian statistical system. The first half of the previous century saw a flurry of activity—from the founding of the Indian Statistical Institute in 1931 to the setting up of the National Sample Survey as the first multi-purpose survey of this scale in the world in 1950—that led to an official statistical system that was better than not just its peers in emerging markets but also arguably on a par with the best in the world. 

In a speech he gave as president of the Indian Science Congress in 1950, P.C. Mahalanobis, the doyen of Indian statisticians, wryly remembered the opposition of members of the apex body to the setting up of a separate statistics branch. One sceptic remarked: “If statistics is to have a section, you may as well have a section for astrology.” It is ironic that these discussions were held just a few years before India built one of the best statistical systems in the world—and one that is now, once again, being equated with astrology because of recent failures.

Big Data can play an important role in rejuvenating the Indian statistical system. We live in a data-rich world, and policymakers have good reason to harvest this data to get a better understanding of the state of the economy. It is good that a small beginning is being made.

How can Big Data analysis improve Indian policy debates? Tell us at views@livemint.com