Home/ Opinion / Columns/  A whole-of-government effort must unlock public data

The past few years have seen growing demands for a common gateway to access public data-sets in India. Efforts to build one date back to the 1990s, when two major data warehouse projects were initiated by the Reserve Bank of India (RBI) and the Union ministry of planning.

The RBI initiative focused on building a unified data portal by bringing together databases held in different RBI departments. By the early 2000s, RBI was able to build a centralized data warehouse, becoming one of the earliest central banks in the world to do so. A wide section of economic researchers and market analysts were able to tap into it after it was opened to the public.

The planning ministry’s initiative, spearheaded by the department of statistics, aimed to integrate all official databases of the Union government. As part of its mandate to implement the National Policy on Dissemination of Statistical Data, the department of statistics (which later morphed into the ministry of statistics and programme implementation or Mospi) announced the setting up of a centralized data warehouse in 1999. It said that its data warehouse would “collect data from various source agencies, integrate the data into logical subject areas, store the data in a manner that is accessible and understandable to non-technical decision makers and deliver data/information to decision-makers through report-writing and query tools".

Unlike the RBI project, the statistics department’s initiative floundered before being abandoned. The failure of Mospi’s initiative led individual ministries to build their own data portals. In the early 2010s, there was a renewed effort to integrate these databases, and make them more accessible to the public. This time, the ministry of science and technology took the initiative to spearhead a new data dissemination policy.

The National Data Sharing and Accessibility Policy (NDSAP) laid down the protocols for sharing official data-sets online, and set the stage for a new data warehouse. The ‘data.gov.in’ portal was created in collaboration with the department of information technology (which later morphed into the ministry of electronics and information technology or Meity) to act as a one-stop data repository.

This data warehouse initiative was partly inspired by the ‘data.gov.us’ example, and launched with the support of India’s open data community. But it failed to meet user needs, turning into a data graveyard. As part of the NDSAP, all departments were supposed to appoint data officers who would compile lists of non-restricted data-sets and open them up for public use. Very few departments bothered to implement these guidelines. The initiative failed to break departmental data silos. Neither Meity nor the department of science and technology had any authority to set statistical standards for the source agencies. Nor did they have the bandwidth to follow up on data quality concerns.

The chequered history of data warehousing in India suggests that it can work well only if it has buy-in from all major stakeholders, including state governments. It would also require IT professionals to work closely with statisticians. For the data portal to realize its full potential, it would need to be accompanied by a data quality assurance mechanism. The Rangarajan Commission had suggested a regular mechanism for statistical audits in 2001. Such statistical checks are needed to assign each data-set a quality rating.

In the absence of a quality certification mechanism, each data user today has to make several judgment calls while using public data-sets based on her own assessment of their quality, coverage, and accuracy.

To address data users’ need for reliable high-frequency data-sets, a whole-of-government approach is required to clean up and organize the official data ecosystem.

However, turf issues might triumph over public interest in this case. Both Meity and Mospi have announced separate plans to set up data warehouses. In 2019, Mospi resurrected its old data warehouse plan, and rechristened it as the National Integrated Information Platform (NIIP). Not to be outdone, Meity announced its own plans to create an ‘India Datasets’ platform in its draft data governance policy released last year.

Meity intends to set up an India Data Management Office (IDMO) that will manage the ‘India Datasets’ portal. IDMO is expected to set data standards across departments and integrate these data-sets in the common data portal. Meity may score over Mospi in speed and execution, and its new data platform may be an improved version of the ‘data.gov.in’ site. But it will still fail to address data users’ demand for quality checks.

The joker in the pack is the Niti Aayog. As a coordinating agency, it could have brought Meity and Mospi together to work on a common goal. Instead, it decided to set up its own data platform, riding on the back of private-sector data consultants. With more modest ambitions, it has been able to get its project off the ground more quickly. The NDAP (National Data Analytics Platform) portal was launched last year to provide easy access to a few core data-sets. It is not clear how far this initiative will be expanded in the coming years.

A key weakness of India’s statistical system has been its fragmented nature, with different parts of the system relying on varying definitions, classifications and quality standards. It is ironic that efforts to integrate the varied parts of the official statistical system are also fragmented.

Pramit Bhattacharya is a Chennai-based journalist. His Twitter handle is pramit_b

Catch all the Business News, Market News, Breaking News Events and Latest News Updates on Live Mint. Download The Mint News App to get Daily Market Updates.
More Less
Updated: 27 Mar 2023, 11:31 PM IST
Recommended For You
Get alerts on WhatsApp
Set Preferences My Reads Watchlist Feedback Redeem a Gift Card Logout