Need a credible plan to fix India’s statistical system

Since then, India’s statistical system has lost sheen, and several of India’s key statistical products are mired in controversy today. Over the past year, I have analysed hundreds of documents and conducted interviews with diverse stakeholders to make sense of the rise and wane of India’s statistical system. Half of them were data producers. The rest were active data consumers who have spent long years studying official statistics.

The detailed results of that research are available in a working paper published by the Carnegie Endowment for International Peace (rb.gy/6k3bi). This column is about its key takeaways, focusing on what the Indian state should do to fix the gaping holes in our statistical system.

A majority of stakeholders feel that India’s statistical system is facing a crisis, and one that has been long in the making. In the early post-independence period, academic researchers and official statisticians worked together to produce a number of statistical innovations that helped policymakers make sense of a complex economy. Since the mid-1970s, both statistical innovation and investments slowed down even as the worlds of research and official statistics moved apart.

Growing insularity, the lack of investments in computing resources and the declining influence of the Planning Commission (which had earlier been a pillar of support for statisticians) eroded the statistical system’s effectiveness. By the turn of the 20th century, India’s statistical crisis had become too severe to be ignored. In early 2000, the government appointed a commission led by C. Rangarajan to review and suggest ways to improve the statistical system.

Some of the commission’s recommendations were implemented, but only in a half-hearted manner. The modest reforms initiated in the wake of the Rangarajan commission’s recommendations failed to resolve the deep-rooted crisis the system faced. The development of the country’s statistical system remained stunted, impacting the credibility of data releases.

While the digitization of public data-sets over the past two decades has led to explosive growth in the quantity of data, there has been very little investment in improving data quality. As a result, the quality of administrative data varies vastly across states and departments. Even the quality of survey data varies significantly across agencies and regions. Growing political pressures on the statistical system lately have made matters worse, leading to delays and disruptions in statistical activities.

While national statistical products get most attention, matters are far worse in state capitals. For many decades, state governments neglected statistical issues. As a result, state-level policymakers came to depend on national surveys for state-level figures. In recent years, some state governments have initiated attempts to set things right. Yet, it is too early to say how far these efforts will bear fruit, and whether they can overcome decades of stasis.

There are no easy fixes for India’s statistical crisis. Even among the well-wishers of the statistical system, opinions differ on which part of the system has greater holes, and what needs to be done to fix them. To arrive at a shared understanding of what ails the statistical system and to devise the right set of prescriptions for it, the government needs to set up a Statistical Reforms Commission.

A lot has changed in the world of statistics since the Rangarajan Commission submitted its report in 2001. The time has come for a fresh review and a comprehensive plan for rebooting the statistical system.

The new commission should have a three-fold mandate. First, it should conduct a cost-benefit analysis of the existing mechanisms of collecting and organizing public data-sets. For instance, it should examine how much key ministries spend on ad-hoc surveys, and whether the government and public will be better served if fewer high-quality surveys are conducted under the supervision of an independent statistical authority.

Second, it should suggest legal and organizational changes that can ensure effective statistical coordination at all levels of government without compromising the autonomy of statistical bodies.

Third, it should prepare a national statistical strategy document to deliver timely and credible statistics in a cost-effective manner. The document should guide the actions of the future statistical authorities once they are set up, and should be reviewed at least once every decade.

The commission could be headed by an eminent statistician or a technocrat who is non-partisan, and is perceived as such. Beyond official statisticians, economic policymakers and jurists, it should also include representatives from India’s growing community of non-official data users. As one of the leading data analytics hubs of the world, India has a pool of extremely talented data scientists in its private sector. Some of that talent should also be tapped to reimagine India’s statistical system.

This is the first of a two-part series on India’s statistical crisis.

Pramit Bhattacharya is a Chennai-based journalist. His Twitter handle is pramit_b